importer

Jsonnet Custom-Importers

• Extend the jsonnet import and importstr functions with the help of custom importers.
• Choose either a specific importer or use the NewMultiImporter() function to enable all together. The prefix of an import path will route to the right importer. As a fallback the default go-jsonnet FileImporter will be used.
• A custom importer can be set for a jsonnet VM like:

import (
    ...
	importer "github.com/peterbueschel/jsonnet-custom-importers"
)
...
    
    m := importer.NewMultiImporter()
    vm := jsonnet.MakeVM()
    vm.Importer(m)

Custom Importers

The main idea is to add a kind of intrinsic functionality into the import path string by adding extra content. The custom importers can parse these extra content and act on them.

The pattern for the import path is based on URLs

• url:

[scheme:][//[userinfo@]host][/]path[?query][#fragment]

• import pattern:

import '[<importer-prefix>://][<glob-pattern>|<filepath>][?<query-parameters>]'

import 'example.jsonnet'

import 'glob.stem+://**/*.jsonnet'

import 'glob.stem+://**/*.jsonnet?exclude=**/*ignore.*'

List of available custom importers and the supported prefixa

MultiImporter

• This importer includes all custom importers and as fallback the default go-jsonnet FileImporter. The MultiImporter tries to find the right custom importer with the help of the <importer prefix>. If it found one, the import string will be forwarded to this custom importer, which in turn takes care of the string.
• Optionally, custom importers can be chosen via:

  m := NewMultiImporter(NewGlobImporter(), NewFallbackFileImporter())

GlobImporter

• Is a custom importer, which:
  • Imports multiple files at once via glob patterns handled by the doublestar library.
  • Supports Continuous imports: If inside the resolved files other glob-patterns will be found, the GlobImporter will also take these glob-imports and resolves the underlying files.
  • Can Exclude imports: use exclude=<glob pattern> as query parameter to exclude files from further handlings.
  • Supports extra JPaths: extra search paths for additional libraries. (⚠️ matches of library paths have a lower priority then matches in the current work dir)
  • Sorts the resolved files: in lexicographical and hierarchical order. Example: [a0 b02 a0/b/c a1 d/x c b2 a1/b b10] becomes [a0 a0/b/c a1 a1/b b02 b10 b2 c d/x]
• Activate the glob-import via the prefix glob.<?> or glob.<?>+ to get the content of the resolved files as object. The content of each file will be available under its resolved path, filename, stem (filename with file extension) or dirname. (see also table in section "Prefix glob.<?> And glob.<?>+")
• Use the prefix glob+ to merge the returned imports. (similar to the jsonnet +: functionality)

Examples

(More examples can be found in the testings file.)

Folder structure:

models
├── blackbox_exporter.json
├── blackbox_exporter.libsonnet
├── node_exporter.json
├── node_exporter.libsonnet
├── development
│   └── grafana.libsonnet
├── production
│   ├── victor_ops.libsonnet
│   └── grafana.libsonnet
└── wavefront.libsonnet

import 'glob://models/**/*.libsonnet';

 {
   'models/blackbox_exporter.libsonnet': import 'models/blackbox_exporter.libsonnet',
   'models/node_exporter.libsonnet': import 'models/node_exporter.libsonnet',
   'models/wavefront.libsonnet': import 'models/wavefront.libsonnet',
   'models/development/grafana.libsonnet': import 'models/development/grafana.libsonnet',
   'models/production/grafana.libsonnet': import 'models/production/grafana.libsonnet',
   'models/production/victor_ops.libsonnet': import 'models/production/victor_ops.libsonnet',
 }

import 'glob.stem://models/**/*.libsonnet'

  {
    blackbox_exporter: import 'models/blackbox_exporter.libsonnet',
    node_exporter: import 'models/node_exporter.libsonnet',
    wavefront: import 'models/wavefront.libsonnet',
    grafana: import 'models/production/grafana.libsonnet',
    victor_ops: import 'models/production/victor_ops.libsonnet',
  }

import 'glob.stem+://models/**/*.libsonnet'

 {
   blackbox_exporter: import 'models/blackbox_exporter.libsonnet',
   node_exporter: import 'models/node_exporter.libsonnet',
   wavefront: import 'models/wavefront.libsonnet',
   grafana: (import 'models/development/grafana.libsonnet') + (import 'models/production/grafana.libsonnet'),
   victor_ops: import 'models/production/victor_ops.libsonnet',
 }

import 'glob+://models/**/*.libsonnet'

(import 'models/blackbox_exporter.libsonnet') +
(import 'models/node_exporter.libsonnet') +
(import 'models/wavefront.libsonnet') +
(import 'models/sub-folder-2/grafana.libsonnet')

Options

Logging

Enable/add a zap.Logger via <Importer>.Logger() per importer or just use the this method on the MultiImporter instance to enable this of all underlying custom importers:

import (
  ...
  "go.uber.org/zap"
)
...
l := zap.Must(zap.NewDevelopment()) // or use zap.NewProduction() to avoid debug messages
m := NewMultiImporter()
m.Logger(l)
...

(update since v0.0.3-alpha) Another option is to use the special config import inside an jsonnet file:

// set the logLevel
local importers = import 'config://set?logLevel=debug';

// enable it for example via:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Aliases

Add an alias for an importer prefix, like:

 g := NewGlobImporter()
 if err := g.SetAliasPrefix("glob", "glob.stem+"); err != nil {
   return err
 }
 m := NewMultiImporter(g)

The SetAliasPrefix() can be used multiple times, whereby only the last setting for an alias-prefix pair will be used.

Import Graph

The MultiImporter can detect import cycles and creates an import graph in dot format once it found a cycle.

In addition such import graphs can also be enable independently via the special config import inside an jsonnet file:

// set the import graph file name
local importers = import 'config://set?importGraph=import_graph.gv';

// enable the file creation:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Example image from testdata/inFileConfigs/importGraph.jsonnet:

the image was created via dot -Tsvg -O graph.gv command (ref. graphviz cli tool)

Ignore Import Cycles

To disable the tests and therefore any error handling for import cycles, you can use the following config in your jsonnet code:

// set the import graph file name
local importers = import 'config://set?ignoreImportCycles';

// enable the file creation:
local myother_imports = importers + (import 'somethingElse.jsonnet');
...

Or directly in your go code via:

 m := NewMultiImporter(g)
 m.IgnoreImportCycles()

Dependencies

• https://github.com/google/go-jsonnet the reason for everything :-)
• https://github.com/bmatcuk/doublestar support for double star (**) glob patterns
• https://github.com/uber-go/zap for structured logging
• https://github.com/dominikbraun/graph to check for import cycles and generate import graphs

Other Projects

• another glob–importer https://qbec.io/reference/jsonnet-glob-importer/
  • example usage https://github.com/splunk/qbec/blob/main/vm/internal/importers/glob_test.go#L32

Follow-Up Tasks

• importstr: add support for importstr
• Ignore paths: add support in the GlobImporter for ignore paths
• Alias: add a prefix to a custom importer via <Importer>.AddPrefix(string)
• HTTP support: loads single files per url
• Git support: loads files in branches from repositories