api

type AbsPath ¶

type AbsPath string

type FilesetPackFilter ¶

type FilesetPackFilter struct {
	// contains filtered or unexported fields
}

func MustParseFilesetPackFilter(s string) FilesetPackFilter

func ParseFilesetPackFilter(s string) (_ FilesetPackFilter, err error)

func (ff FilesetPackFilter) Apply(ff2 FilesetPackFilter) FilesetPackFilter

func (ff FilesetPackFilter) Dev() (keep bool, reject bool)

func (ff FilesetPackFilter) Gid() (keep bool, setTo int)

func (ff FilesetPackFilter) IsComplete() bool

func (ff FilesetPackFilter) Mtime() (keep bool, setTo time.Time)

func (ff FilesetPackFilter) MtimeUnix() (keep bool, setTo int64)

func (ff FilesetPackFilter) Setid() (keep bool, reject bool)

func (ff FilesetPackFilter) Sticky() (keep bool)

func (x FilesetPackFilter) String() (v string)

func (ff FilesetPackFilter) Uid() (keep bool, setTo int)

type FilesetUnpackFilter ¶

type FilesetUnpackFilter struct {
	// contains filtered or unexported fields
}

func MustParseFilesetUnpackFilter(s string) FilesetUnpackFilter

func ParseFilesetUnpackFilter(s string) (_ FilesetUnpackFilter, err error)

func (ff FilesetUnpackFilter) Altering() bool

func (ff FilesetUnpackFilter) Apply(ff2 FilesetUnpackFilter) FilesetUnpackFilter

func (ff FilesetUnpackFilter) Dev() (follow bool, reject bool)

func (ff FilesetUnpackFilter) Gid() (follow, setMine bool, setTo int)

func (ff FilesetUnpackFilter) IsComplete() bool

func (ff FilesetUnpackFilter) Mtime() (follow, setNow bool, setTo time.Time)

func (ff FilesetUnpackFilter) MtimeUnix() (follow, now bool, setTo int64)

func (ff FilesetUnpackFilter) Setid() (follow bool, reject bool)

func (ff FilesetUnpackFilter) Sticky() (follow bool)

func (x FilesetUnpackFilter) String() (v string)

func (ff FilesetUnpackFilter) Uid() (follow, setMine bool, setTo int)

type Formula ¶

type Formula struct {
	Inputs  map[AbsPath]WareID
	Action  FormulaAction
	Outputs map[AbsPath]FormulaOutputSpec
}

func (f Formula) Clone() (f2 Formula)

func (frm Formula) SetupHash() FormulaSetupHash

type FormulaAction ¶

type FormulaAction struct {
	// An array of strings to hand as args to exec -- creates a single process.
	Exec []string `refmt:",omitempty"`

	// Noop may be set as an alternative to Exec; this allows manipulations of
	// files that can be done from pure path of inputs and outputs alone.
	Noop bool `refmt:",omitempty"`

	// How much power to give the process.  Default is quite low.
	Policy FormulaPolicy `refmt:",omitempty"`

	// The working directory to set when invoking the executable.
	// If not set, will be defaulted to "/task".
	Cwd AbsPath `refmt:",omitempty"`

	// Environment variables.
	Env map[string]string `refmt:",omitempty"`

	// User info -- uid, gid, etc.
	Userinfo *FormulaUserinfo `refmt:",omitempty"`

	// Cradle -- enabled by default, enum value for disable.
	Cradle string `refmt:",omitempty"`

	// Hostname to set inside the container (if the executor supports this -- not all do).
	Hostname string `refmt:",omitempty"`
}

FormulaAction defines the action to perform to "evaluate" the formula -- after the input filesets have been assembled, these commands will be run in a contained sandbox on with those filesets, and when the commands terminate, the output filesets will be saved.

The definition of the Action includes at minimum what commands to run, but also includes the option of specifying other execution parameters: things like environment variables, working directory, hostname... and (though hopefully you rarely get hung up and need to change these) also things like UID, GID, username, homedir, and soforth. All of these additional parameters have "sensible defaults" if unset.

The Action also includes the ability to set "Policy" level -- these define simple privilege levels. (The default policy is of extremely low privileges.)

type FormulaOutputSpec ¶

type FormulaOutputSpec struct {
	PackType PackType          `refmt:"packtype"`
	Filter   FilesetPackFilter `refmt:"filters",omitempty`
}

type FormulaPolicy ¶

type FormulaPolicy string

FormulaPolicy constants enumerate the privilege levels a contained process can be started with. (They're a shorthand for linux 'capabilities', with some sensible safe sets pre-selected.)

Policies are meant as a rough, relatively approachable, user-facing shorthand for privilege levels. In practice they typically map onto linux 'capabilities', but this is considered an implementation detail, not guaranteed, and may be executor engine specific (for example, the 'chroot' executor cannot provide fine-grained capabilities at all).

const (
	/*
		Operate with a low privilege, as if you were a regular user on a
		regular system.  No special permissions will be granted
		(and in systems with capabilities support, special permissions
		will not be available even if processes do manage to
		change uid, e.g. through suid binaries; most capabilities
		are dropped).

		This is the safest mode to run as.  And, naturally, the default.

		Note that you may still (separately) set the Userinfo to values like
		uid=0 and gid=0, even while at 'routine' policy privileges.
		This is fine; an executor engine that supports capabilities dropping
		will still result in operations that the "root" user would normally
		be able to perform (like chown any file) will still result in
		permission denied.
	*/
	FormulaPolicy_Routine FormulaPolicy = "routine"

	/*
		Operate with escalated but still relatively safe privilege.
		Dangerous capabilities (e.g. "muck with devices") are dropped,
		but the most commonly used of root's powers (like chown any file)
		are available.

		This may be slightly safer than enabling full 'sysad' mode,
		but you should still prefer to use any of the lower power levels
		if possible.

		This mode is the most similar to what you would experience with
		docker defaults.

		This mode should not be assumed secure when combined with host mounts.
		(For example, one can trivially make an executable file in the
		host mount, set it to owner=0, set it setuid, and thus have a
		beachhead ready for a later phase in an attack.)
	*/
	FormulaPolicy_Governor FormulaPolicy = "governor"

	/*
		Operate with *ALL CAPABILITIES*.

		This is absolutely not secure against untrusted code -- it is
		completely equivalent in power to root on your host.  Please
		try to use any of the lower power levels first.

		Among the things a system administrator may do is rebooting
		the machine and updating the kernel.  Seriously, *only* use
		with trusted code.
	*/
	FormulaPolicy_Sysad FormulaPolicy = "sysad"
)

type FormulaRunRecord ¶

type FormulaRunRecord struct {
	Guid      string             `refmt:"guid"`      // random number, presumed globally unique.
	Time      int64              `refmt:"time"`      // time at start of build.
	FormulaID FormulaSetupHash   `refmt:"formulaID"` // HID of formula ran.
	ExitCode  int                `refmt:"exitCode"`  // exit code of the contained process.
	Results   map[AbsPath]WareID `refmt:"results"`   // wares produced by the run!

	Hostname string            `refmt:",omitempty"` // Optional: hostname.  not a trusted field, but useful for debugging.
	Metadata map[string]string `refmt:",omitempty"` // Optional: escape valve.  you can attach freetext here.
}

type FormulaSetupHash ¶

type FormulaSetupHash string

FormulaSetupHash is an opaque string derived from a cryptographic hash of the deterministic serialization of a Formula. Which is a fancy way of saying it's a fantastic primary key for memoizing computations.

type FormulaUserinfo ¶

type FormulaUserinfo struct {
	Uid      *int    `refmt:",omitempty"`
	Gid      *int    `refmt:",omitempty"`
	Username string  `refmt:",omitempty"`
	Homedir  AbsPath `refmt:",omitempty"`
}

type ImportRef ¶

type ImportRef interface {
	String() string
	// contains filtered or unexported methods
}

ImportRef is a sum type, containing either a catalog reference ("catalog:{moduleName}:{releaseName}:{itemName}") or parent reference ("parent:{slotRef}"; only valid in submodules) or an ingest reference ("ingest:{ingestKind}[:{addntl}]"; only valid on main module).

Ingest references are interesting and should be used sparingly; they're for where new data comes into the Timeless ecosystem -- and that also means ingest references are also where the Timeless Stack abilities to automatically recursively audit where that data came from has reached its end.

Ingest references may explicitly reference wares (ex. "ingest:literal:tar:f00bAr"), or lean on other extensions to bring data into the system (ex. "ingest:git:.:HEAD"). Again, use sparingly: anything beyond "ingest:literal" and your module pipeline has become virtually impossible for anyone to evaluate without whatever additional un-contained un-tracked context your ingest refers to.

Ingest references should be passed on directly as an export of a module. Failure to do so is not *exactly* illegal, but it would make any replay of this module impossible without un-tracked context, and as such most of the tools in the Timeless Stack will issue either warnings or outright errors if the ingested data isn't also in the module exports.

func ParseImportRef(x string) (ImportRef, error)

type ImportRef_Catalog ¶

type ImportRef_Catalog ItemRef

func (x ImportRef_Catalog) String() string

type ImportRef_Ingest ¶

type ImportRef_Ingest struct {
	IngestKind string
	Args       string
}

func (x ImportRef_Ingest) String() string

type ImportRef_Parent ¶

type ImportRef_Parent SlotRef

func (x ImportRef_Parent) String() string

type ItemName ¶

type ItemName string

type ItemRef ¶

type ItemRef struct {
	ModuleName
	ReleaseName
	ItemName
}

func ParseItemRef(x string) (v ItemRef, err error)

func (x ItemRef) String() string

type Lineage ¶

type Lineage struct {
	// Name of self.
	Name ModuleName

	// Ordered list of release entries.
	// Order not particularly important, though UIs generally display in this order.
	// Most recent entries are should be placed at the top (e.g. index zero).
	//
	// Each entry must have a unique ReleaseName in the scope of this Lineage.
	Releases []Release
}

Lineage contains the metadata for all releases for a particular module. Treat it as an append-only record: new releases append to the module's lineage.

type Module ¶

type Module struct {
	Imports map[SlotName]ImportRef
	Steps   map[StepName]StepUnion
	Exports map[ItemName]SlotRef `refmt:",omitempty"`
}

type ModuleName ¶

type ModuleName string

func (x ModuleName) Validate() error

[[[...]subsubdomain.]subdomain.]domain[/path[/morepath[...]]]

type Operation ¶

type Operation struct {
	Inputs  map[AbsPath]SlotRef
	Action  FormulaAction
	Outputs map[SlotName]AbsPath `refmt:",omitempty"`
}

Operation is one of the concrete types of StepUnion which composes a Module; it describes a containerizable computation, all of its input filesystem paths bound to slot references, and all of the paths that should be collected as outputs and assigned to another slot for further use.

When all of the input slot references in an Operation are known, it can be bound, becoming a Formula -- which is structurally similar, but now with all specific, concrete WareID hashes instead of SlotRef.

type OperationRecord ¶

type OperationRecord struct {
	FormulaRunRecord
	Results map[SlotName]WareID
}

OperationRecord is mostly an alias of FormulaRunRecord, but with Results indexed by SlotName from the Operation rather than path in the Formula.

We usually serialize FormulaRunRecord, because it's more convergent when content-addressed; OperationRecord contains immaterial details (e.g. the SlotName). OperationRecord is sometimes more convenient to use internally.

type PackType ¶

type PackType string

A PackType string identifies what kind of packing format is used when packing a ware. It's the first part of a WareID tuple.

Typically, the desired PackType is an argument when using packing tools; whereas the PackType is communicated by the WareID when using unpack tools.

PackTypes are a simple [a-zA-Z0-9] string. Colons in particular are not allowable (since a PackType string is the first part of a WareID).

type Release ¶

type Release struct {
	Name     ReleaseName
	Items    map[ItemName]WareID
	Metadata map[string]string
	Hazards  map[string]string
}

Release describes a single atomic release of wares. Each release must have a name, and contains a set of items, where each item refers to a WareID.

Releases are used to group something chronologically; items in a release are used to distinguish between multiple artifacts in a release.

In the context of building software, a Release usually has semantics lining up with "a bunch of software built from a particular source checkout". And thus, typically, there is also an Item in the release called "src"; and often enough, this will be a "git" wareID. Other Item names likely to appear might be "linux-amd64", for example. All of this is convention, however; releases could just as well be used to track various versions of a photo album.

It is recommended that a series of Release entries in a Lineage should stick to the same set of ItemName over time, because consumers of catalog information generally expect this, and changing Item names may produce work for other people.

type ReleaseName ¶

type ReleaseName string

type SlotName ¶

type SlotName string

type SlotRef ¶

type SlotRef struct {
	StepName // zero for module import reference
	SlotName
}

func ParseSlotRef(x string) (SlotRef, error)

func (x SlotRef) String() string

type StepName ¶

type StepName string

type StepUnion ¶

type StepUnion interface {
	// contains filtered or unexported methods
}

type SubmoduleRef ¶

type SubmoduleRef string // .-sep.  really is a []StepName, but we wanted something easily used as a map key.

func (ref SubmoduleRef) Child(child StepName) SubmoduleRef

func (ref SubmoduleRef) Contextualize(parent SubmoduleRef) SubmoduleRef

func (ref SubmoduleRef) Decontextualize() SubmoduleRef

func (ref SubmoduleRef) First() StepName

type SubmoduleSlotRef ¶

type SubmoduleSlotRef struct {
	SubmoduleRef
	SlotRef
}

func (ref SubmoduleSlotRef) Contextualize(parent SubmoduleRef) SubmoduleSlotRef

func (ref SubmoduleSlotRef) Decontextualize() SubmoduleSlotRef

func (x SubmoduleSlotRef) String() string

type SubmoduleSlotRefList ¶

type SubmoduleSlotRefList []SubmoduleSlotRef

func (s SubmoduleSlotRefList) Len() int

func (s SubmoduleSlotRefList) Less(i, j int) bool

func (s SubmoduleSlotRefList) Swap(i, j int)

type SubmoduleStepRef ¶

type SubmoduleStepRef struct {
	SubmoduleRef
	StepName
}

func (ref SubmoduleStepRef) Contextualize(parent SubmoduleRef) SubmoduleStepRef

func (ref SubmoduleStepRef) Decontextualize() SubmoduleStepRef

func (x SubmoduleStepRef) String() string

type WareID ¶

type WareID struct {
	Type PackType
	Hash string
}

WareID is a content-addressable, cryptographic hashes that uniquely identifies a "ware" -- a packed Fileset. (Fileset and Ware are distinct concepts because a fileset is not packed in any particular way and thus has no innate hash; a Ware is packed and hashed.)

Ware IDs are serialized as a string in two parts, separated by a colon -- for example like "git:f23ae1829" or "tar:WJL8or32vD". The first part communicates which kind of packing system computed the hash, and the second part is the hash itself.

func ParseWareID(x string) (WareID, error)

func (x WareID) String() string

type WareSourcing ¶

type WareSourcing struct {
	ByPackType map[PackType][]WarehouseLocation                `refmt:",omitempty"`
	ByModule   map[ModuleName]map[PackType][]WarehouseLocation `refmt:",omitempty"`
	ByWare     map[WareID][]WarehouseLocation                  `refmt:",omitempty"`
}

WareSourcing contains suggestions on WarehouseLocations which may be able to provide Wares.

This information may be indexed in several different ways: most specifically (and inflexibly, and verbosely) by specific WareID; or by module name; or by pack type in general. (Non-content-addressible WarehouseLocations only semantically make sense when indexed by specific WareID; since the other forms of indexing will recommend the WarehouseLocation for more than one specific WareID, it stands to reason that they the WarehouseLocation ought to specify a system which can store more than one Ware!)

WareSourcing is meant to be reasonable to provide to *more than one* Operation (each of which may also have more than one input, of course) -- the various mechanisms of indexing allow such generalized suggestions.

func (ws *WareSourcing) Append(ws2 WareSourcing)

func (ws *WareSourcing) AppendByModule(modName ModuleName, packtype PackType, locations ...WarehouseLocation)

func (ws *WareSourcing) AppendByPackType(packtype PackType, locations ...WarehouseLocation)

func (ws *WareSourcing) AppendByWare(wareID WareID, locations ...WarehouseLocation)

func (ws WareSourcing) PivotToInputs(frm Formula) WareSourcing

func (ws WareSourcing) PivotToModuleWare(wareID WareID, assumingModName ModuleName) WareSourcing

func (ws WareSourcing) PivotToWareID(wareID WareID) (v []WarehouseLocation)

func (ws WareSourcing) PivotToWareIDs(wareIDs map[WareID]struct{}) WareSourcing

type WareStaging ¶

type WareStaging struct {
	ByPackType map[PackType]WarehouseLocation
}

WareStaging contains instructions on where to store wares that are output from an Operation.

WareStaging only takes a single warehouse location per packtype. It is intended that if you want to replicate the ware storage to multiple locations, you should do this later, *not* while saving the output from the Operation. An Operation may fail if the WarehouseLocation provided by the WareStaging info is not writable.

It is semantically unreasonable to provide a non-content-addressable WarehouseLocation in WareStaging info: WareStaging info is meant to be reasonable to provide to *more than one* Operation (each of which may also have more than one output, of course) -- therefore it is only sensible to provide a WarehouseLocation which is capable of storing more than one Ware! (You may still run Repeatr with non-CA WarehouseLocation configurations for specific outputs; it's only the higher level pipelining tools which become opinionated about this.)

type WarehouseLocation ¶

type WarehouseLocation string