zed

package module
v1.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 29, 2024 License: BSD-3-Clause Imports: 21 Imported by: 21

README ¶

Zed Tests GoPkg

Zed offers a new approach to data that makes it easier to manipulate and manage your data.

With Zed's new super-structured data model, messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON's uncanny ability to represent eclectic data.

Trying out Zed is easy: just install the command-line tool zq.

zq is a lot like jq but is built from the ground up as a search and analytics engine based on the Zed data model. Since Zed data is a proper superset of JSON, zq also works natively with JSON.

While zq and the Zed data formats are production quality, the Zed project's Zed data lake is a bit earlier in development.

For a non-technical user, Zed is as easy to use as web search while for a technical user, Zed exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand ZSON data format and Zed language.

Why?

We think data is hard and it should be much, much easier.

While schemas are a great way to model and organize your data, they often get in the way when you are just trying to store or transmit your semi-structured data.

Also, why should you have to set up one system for search and another completely different system for historical analytics? And the same unified search/analytics system that works at cloud scale should run easily as a lightweight command-line tool on your laptop.

And rather than having to set up complex ETL pipelines with brittle transformation logic, managing your data lake should be as easy as git.

Finally, we believe a lightweight data store that provides easy search and analytics would be a great place to store data sets for data science and data engineering experiments running in Python and providing easy integration with your favorite Python libraries.

How?

Zed solves all these problems with a new foundational data format called ZSON, which is a superset of JSON and the relational models. ZSON is syntax-compatible with JSON but it has a comprehensive type system that you can use as little or as much as you like. Zed types can be used as schemas.

The Zed language offers a gentle learning curve, which spans the gamut from simple keyword search to powerful data-transformation operators like lateral sub-queries and shaping.

Zed also has a cloud-based object design that was modeled after the git design pattern. Commits to the lake are transactional and consistent.

Quick Start

Check out the installation page for a quick and easy install.

Detailed documentation for the entire Zed system and language is available on the Zed docs site.

Zui

The Zui app is an Electron-based desktop app to explore, query, and shape data in your Zed lake.

We originally developed Zui for security-oriented use cases (having tight integration with Zeek, Suricata, and Wireshark), but we are actively extending Zui with UX for handling generic data sets to support data science, data engineering, and ETL use cases.

Contributing

See the contributing guide on how you can help improve Zed!

Join the Community

Join our public Slack workspace for announcements, Q&A, and to trade tips!

Acknowledgment

We modeled this README after Philip O'Toole's brilliantly succinct description of rqlite.

Documentation ¶

Overview ¶

Package zng implements a data typing system based on the zeek type system. All zeek types are defined here and implement the Type interface while instances of values implement the Value interface. All values conform to exactly one type. The package provides a fast-path for comparing a value to a byte slice without having to create a zeek value from the byte slice. To exploit this, all values include a Comparison method that returns a Predicate function that takes a byte slice and a Type and returns a boolean indicating whether the the byte slice with the indicated Type matches the value. The package also provides mechanism for coercing values in well-defined and natural ways.

Index ¶

Constants ¶

View Source 
const (
	MaxEnumSymbols  = 100_000
	MaxRecordFields = 100_000
	MaxUnionTypes   = 100_000
)
View Source 
const (
	IDUint8       = 0
	IDUint16      = 1
	IDUint32      = 2
	IDUint64      = 3
	IDUint128     = 4
	IDUint256     = 5
	IDInt8        = 6
	IDInt16       = 7
	IDInt32       = 8
	IDInt64       = 9
	IDInt128      = 10
	IDInt256      = 11
	IDDuration    = 12
	IDTime        = 13
	IDFloat16     = 14
	IDFloat32     = 15
	IDFloat64     = 16
	IDFloat128    = 17
	IDFloat256    = 18
	IDDecimal32   = 19
	IDDecimal64   = 20
	IDDecimal128  = 21
	IDDecimal256  = 22
	IDBool        = 23
	IDBytes       = 24
	IDString      = 25
	IDIP          = 26
	IDNet         = 27
	IDType        = 28
	IDNull        = 29
	IDTypeComplex = 30
)
View Source 
const (
	TypeValueRecord  = 30
	TypeValueArray   = 31
	TypeValueSet     = 32
	TypeValueMap     = 33
	TypeValueUnion   = 34
	TypeValueEnum    = 35
	TypeValueError   = 36
	TypeValueNameDef = 37
	TypeValueNameRef = 38
	TypeValueMax     = TypeValueNameRef
)

Variables ¶

View Source 
var (
	ErrNotArray  = errors.New("cannot index a non-array")
	ErrIndex     = errors.New("array index out of bounds")
	ErrUnionTag  = errors.New("invalid union tag")
	ErrEnumIndex = errors.New("enum index out of bounds")
)
View Source 
var (
	TypeUint8    = &TypeOfUint8{}
	TypeUint16   = &TypeOfUint16{}
	TypeUint32   = &TypeOfUint32{}
	TypeUint64   = &TypeOfUint64{}
	TypeInt8     = &TypeOfInt8{}
	TypeInt16    = &TypeOfInt16{}
	TypeInt32    = &TypeOfInt32{}
	TypeInt64    = &TypeOfInt64{}
	TypeDuration = &TypeOfDuration{}
	TypeTime     = &TypeOfTime{}
	TypeFloat16  = &TypeOfFloat16{}
	TypeFloat32  = &TypeOfFloat32{}
	TypeFloat64  = &TypeOfFloat64{}
	// XXX add TypeDecimal
	TypeBool   = &TypeOfBool{}
	TypeBytes  = &TypeOfBytes{}
	TypeString = &TypeOfString{}
	TypeIP     = &TypeOfIP{}
	TypeNet    = &TypeOfNet{}
	TypeType   = &TypeOfType{}
	TypeNull   = &TypeOfNull{}
)
View Source 
var (
	ErrMissingField = errors.New("record missing a field")
	ErrNotContainer = errors.New("expected container type, got primitive")
)
View Source 
var (
	NullUint8    = Value{/* contains filtered or unexported fields */}
	NullUint16   = Value{/* contains filtered or unexported fields */}
	NullUint32   = Value{/* contains filtered or unexported fields */}
	NullUint64   = Value{/* contains filtered or unexported fields */}
	NullInt8     = Value{/* contains filtered or unexported fields */}
	NullInt16    = Value{/* contains filtered or unexported fields */}
	NullInt32    = Value{/* contains filtered or unexported fields */}
	NullInt64    = Value{/* contains filtered or unexported fields */}
	NullDuration = Value{/* contains filtered or unexported fields */}
	NullTime     = Value{/* contains filtered or unexported fields */}
	NullFloat16  = Value{/* contains filtered or unexported fields */}
	NullFloat32  = Value{/* contains filtered or unexported fields */}
	NullFloat64  = Value{/* contains filtered or unexported fields */}
	NullBool     = Value{/* contains filtered or unexported fields */}
	NullBytes    = Value{/* contains filtered or unexported fields */}
	NullString   = Value{/* contains filtered or unexported fields */}
	NullIP       = Value{/* contains filtered or unexported fields */}
	NullNet      = Value{/* contains filtered or unexported fields */}
	NullType     = Value{/* contains filtered or unexported fields */}
	Null         = Value{/* contains filtered or unexported fields */}

	False = NewBool(false)
	True  = NewBool(true)
)
View Source 
var ErrMissing = errors.New("missing")

ErrMissing is a Go error that implies a missing value in the runtime logic whereas Missing is a Zed error value that represents a missing value embedded in the dataflow computation.

View Source 
var Missing = zcode.Bytes("missing")

Missing is value that represents an error condition arising from a referenced entity not present, e.g., a reference to a non-existent record field, a map lookup for a key not present, an array index that is out of range, etc. The Missing error can be propagated through functions and expressions and each operator has clearly defined semantics with respect to the Missing value. For example, "true AND MISSING" is MISSING.

View Source 
var Quiet = zcode.Bytes("quiet")
View Source 
var SkipContainer = errors.New("skip this container")

SkipContainer is used as a return value from Visitors to indicate that the container passed in the call should not be visited. It is not returned as an error by any function.

Functions ¶

func AppendBool ¶ 

func AppendBool(zb zcode.Bytes, b bool) zcode.Bytes

func AppendDuration ¶ 

func AppendDuration(bytes zcode.Bytes, d nano.Duration) zcode.Bytes

func AppendFloat16 ¶ added in v1.5.0 

func AppendFloat16(zb zcode.Bytes, f float32) zcode.Bytes

func AppendFloat32 ¶ 

func AppendFloat32(zb zcode.Bytes, f float32) zcode.Bytes

func AppendFloat64 ¶ 

func AppendFloat64(zb zcode.Bytes, d float64) zcode.Bytes

func AppendIP ¶ 

func AppendIP(zb zcode.Bytes, a netip.Addr) zcode.Bytes

func AppendInt ¶ 

func AppendInt(bytes zcode.Bytes, i int64) zcode.Bytes

func AppendNet ¶ 

func AppendNet(zb zcode.Bytes, p netip.Prefix) zcode.Bytes

func AppendTime ¶ 

func AppendTime(bytes zcode.Bytes, t nano.Ts) zcode.Bytes

func AppendTypeValue ¶ added in v1.0.0 

func AppendTypeValue(b zcode.Bytes, t Type) zcode.Bytes

func AppendUint ¶ 

func AppendUint(bytes zcode.Bytes, i uint64) zcode.Bytes

func BuildUnion ¶ 

func BuildUnion(b *zcode.Builder, tag int, val zcode.Bytes)

BuildUnion appends to b a union described by tag and val.

func CompareTypes ¶ added in v1.0.0 

func CompareTypes(a, b Type) int

func DecodeBool ¶ 

func DecodeBool(zv zcode.Bytes) bool

func DecodeBytes ¶ 

func DecodeBytes(zv zcode.Bytes) []byte

func DecodeDuration ¶ 

func DecodeDuration(zv zcode.Bytes) nano.Duration

func DecodeFloat ¶ 

func DecodeFloat(zb zcode.Bytes) float64

func DecodeFloat16 ¶ added in v1.5.0 

func DecodeFloat16(zb zcode.Bytes) float32

func DecodeFloat32 ¶ 

func DecodeFloat32(zb zcode.Bytes) float32

func DecodeFloat64 ¶ 

func DecodeFloat64(zv zcode.Bytes) float64

func DecodeIP ¶ 

func DecodeIP(zv zcode.Bytes) netip.Addr

func DecodeInt ¶ 

func DecodeInt(zv zcode.Bytes) int64

func DecodeLength ¶ added in v1.0.0 

func DecodeLength(tv zcode.Bytes) (int, zcode.Bytes)

func DecodeName ¶ added in v1.0.0 

func DecodeName(tv zcode.Bytes) (string, zcode.Bytes)

func DecodeNet ¶ 

func DecodeNet(zv zcode.Bytes) netip.Prefix

func DecodeString ¶ 

func DecodeString(zv zcode.Bytes) string

func DecodeTime ¶ 

func DecodeTime(zv zcode.Bytes) nano.Ts

func DecodeUint ¶ 

func DecodeUint(zv zcode.Bytes) uint64

func EncodeBool ¶ 

func EncodeBool(b bool) zcode.Bytes

func EncodeBytes ¶ 

func EncodeBytes(b []byte) zcode.Bytes

func EncodeDuration ¶ 

func EncodeDuration(d nano.Duration) zcode.Bytes

func EncodeFloat16 ¶ added in v1.5.0 

func EncodeFloat16(d float32) zcode.Bytes

func EncodeFloat32 ¶ 

func EncodeFloat32(d float32) zcode.Bytes

func EncodeFloat64 ¶ 

func EncodeFloat64(d float64) zcode.Bytes

func EncodeIP ¶ 

func EncodeIP(a netip.Addr) zcode.Bytes

func EncodeInt ¶ 

func EncodeInt(i int64) zcode.Bytes

func EncodeNet ¶ 

func EncodeNet(p netip.Prefix) zcode.Bytes

func EncodeString ¶ 

func EncodeString(s string) zcode.Bytes

func EncodeTime ¶ 

func EncodeTime(t nano.Ts) zcode.Bytes

func EncodeTypeValue ¶ 

func EncodeTypeValue(t Type) zcode.Bytes

func EncodeUint ¶ 

func EncodeUint(i uint64) zcode.Bytes

func IsContainerType ¶ 

func IsContainerType(typ Type) bool

func IsFloat ¶ 

func IsFloat(id int) bool

True iff the type id is encoded as a float encoding. XXX add IDDecimal here when we implement coercible math with it.

func IsInteger ¶ 

func IsInteger(id int) bool

True iff the type id is encoded as a zng signed or unsigened integer zcode.Bytes.

func IsNumber ¶ 

func IsNumber(id int) bool

True iff the type id is encoded as a zng signed or unsigned integer zcode.Bytes, float16 zcode.Bytes, float32 zcode.Bytes, or float64 zcode.Bytes.

func IsPrimitiveType ¶ 

func IsPrimitiveType(typ Type) bool

func IsRecordType ¶ 

func IsRecordType(typ Type) bool

func IsSigned ¶ 

func IsSigned(id int) bool

True iff the type id is encoded as a number encoding and is signed.

func IsUnionType ¶ 

func IsUnionType(typ Type) bool

func IsUnsigned ¶ added in v1.9.0 

func IsUnsigned(id int) bool

True iff the type id is encoded as a number encoding and is unsigned.

func LeadingOnes ¶ added in v1.4.0 

func LeadingOnes(b []byte) int

LeadingOnes returns the number of leading one bits in b.

func NormalizeMap ¶ 

func NormalizeMap(zv zcode.Bytes) zcode.Bytes

NormalizeMap interprets zv as a map body and returns an equivalent map body that is normalized according to the ZNG specification (i.e., the tag-counted value of each entry's key is lexicographically greater than that of the preceding entry).

func NormalizeSet ¶ 

func NormalizeSet(zv zcode.Bytes) zcode.Bytes

NormalizeSet interprets zv as a set body and returns an equivalent set body that is normalized according to the ZNG specification (i.e., each element's tag-counted value is lexicographically greater than that of the preceding element).

func PrimitiveName ¶ added in v1.0.0 

func PrimitiveName(typ Type) string

func TypeID ¶ 

func TypeID(typ Type) int

func Walk ¶ 

func Walk(typ Type, body zcode.Bytes, visit Visitor) error

Types ¶

type Allocator ¶ added in v1.0.0 

type Allocator interface{}

type Context ¶ 

type Context struct {
	// contains filtered or unexported fields
}

A Context implements the "type context" in the Zed model. For a given set of related Values, each Value has a type from a shared Context. The Context manages the transitive closure of Types so that each unique type corresponds to exactly one Type pointer allowing type equivlance to be determined by pointer comparison. (Type pointers from distinct Contexts obviously do not have this property.) A Context also provides an efficient means to translate type values (represented as serialized ZNG) to Types. This provides an efficient means to translate Type pointers from one context to another.

func NewContext ¶ 

func NewContext() *Context

func (*Context) AddFields ¶ added in v1.5.0 

func (c *Context) AddFields(r Value, newFields []Field, vals []Value) (Value, error)

AddFields returns a new Record with fields equal to the given record along with new rightmost fields as indicated with the given values. If any of the newly provided fieldss already exists in the specified value, an error is returned.

func (*Context) DecodeTypeValue ¶ added in v1.0.0 

func (c *Context) DecodeTypeValue(tv zcode.Bytes) (Type, zcode.Bytes)

func (*Context) LookupByValue ¶ 

func (c *Context) LookupByValue(tv zcode.Bytes) (Type, error)

LookupByValue returns the Type indicated by a binary-serialized type value. This provides a means to translate a type-context-independent serialized encoding for an arbitrary type into the reciever Context.

func (*Context) LookupType ¶ 

func (c *Context) LookupType(id int) (Type, error)

func (*Context) LookupTypeArray ¶ 

func (c *Context) LookupTypeArray(inner Type) *TypeArray

func (*Context) LookupTypeDef ¶ 

func (c *Context) LookupTypeDef(name string) *TypeNamed

LookupTypeDef returns the named type last bound to name by LookupTypeNamed. It returns nil if name is unbound.

func (*Context) LookupTypeEnum ¶ 

func (c *Context) LookupTypeEnum(symbols []string) *TypeEnum

func (*Context) LookupTypeError ¶ added in v1.0.0 

func (c *Context) LookupTypeError(inner Type) *TypeError

func (*Context) LookupTypeMap ¶ 

func (c *Context) LookupTypeMap(keyType, valType Type) *TypeMap

func (*Context) LookupTypeNamed ¶ added in v1.0.0 

func (c *Context) LookupTypeNamed(name string, inner Type) (*TypeNamed, error)

LookupTypeNamed returns the named type for name and inner. It also binds name to that named type. LookupTypeNamed returns an error if name is not a valid UTF-8 string or is a primitive type name.

func (*Context) LookupTypeRecord ¶ 

func (c *Context) LookupTypeRecord(fields []Field) (*TypeRecord, error)

LookupTypeRecord returns a TypeRecord within this context that binds with the indicated fields. Subsequent calls with the same fields will return the same record pointer. If the type doesn't exist, it's created, stored, and returned. The closure of types within the fields must all be from this type context. If you want to use fields from a different type context, use TranslateTypeRecord.

func (*Context) LookupTypeSet ¶ 

func (c *Context) LookupTypeSet(inner Type) *TypeSet

func (*Context) LookupTypeUnion ¶ 

func (c *Context) LookupTypeUnion(types []Type) *TypeUnion

func (*Context) LookupTypeValue ¶ 

func (c *Context) LookupTypeValue(typ Type) Value

func (*Context) Missing ¶ added in v1.0.0 

func (c *Context) Missing() Value

func (*Context) MustLookupTypeRecord ¶ 

func (c *Context) MustLookupTypeRecord(fields []Field) *TypeRecord

func (*Context) NewError ¶ added in v1.0.0 

func (c *Context) NewError(err error) Value

func (*Context) NewErrorf ¶ added in v1.0.0 

func (c *Context) NewErrorf(format string, args ...interface{}) Value

func (*Context) Quiet ¶ added in v1.0.0 

func (c *Context) Quiet() Value

func (*Context) Reset ¶ 

func (c *Context) Reset()

func (*Context) StringTypeError ¶ added in v1.0.0 

func (c *Context) StringTypeError() *TypeError

func (*Context) TranslateType ¶ 

func (c *Context) TranslateType(ext Type) (Type, error)

TranslateType takes a type from another context and creates and returns that type in this context.

func (*Context) WrapError ¶ added in v1.0.0 

func (c *Context) WrapError(msg string, val Value) Value

type DuplicateFieldError ¶ added in v1.0.0 

type DuplicateFieldError struct {
	Name string
}

func (*DuplicateFieldError) Error ¶ added in v1.0.0 

func (d *DuplicateFieldError) Error() string

type Field ¶ added in v1.5.0 

type Field struct {
	Name string
	Type Type
}

Field defines the name and type of a field for TypeRecord.

func NewField ¶ added in v1.5.0 

func NewField(name string, typ Type) Field

type Kind ¶ added in v1.0.0 

type Kind int
const (
	PrimitiveKind Kind = iota
	RecordKind
	ArrayKind
	SetKind
	MapKind
	UnionKind
	EnumKind
	ErrorKind
)

func (Kind) String ¶ added in v1.0.0 

func (k Kind) String() string

type Mapper ¶ 

type Mapper struct {
	// contains filtered or unexported fields
}

func NewMapper ¶ 

func NewMapper(out *Context) *Mapper

func (*Mapper) Enter ¶ 

func (m *Mapper) Enter(ext Type) (Type, error)

func (*Mapper) EnterType ¶ 

func (m *Mapper) EnterType(id int, typ Type)

func (*Mapper) Lookup ¶ 

func (m *Mapper) Lookup(id int) Type

Lookup tranlates Zed types by type ID from one context to another. The first context is implied by the argument to Lookup() and the output type context is explicitly determined by the argument to NewMapper(). If a binding has not yet been entered, nil is returned and Enter() should be called to create the binding. There is a race here when two threads attempt to update the same ID, but it is safe because the outputContext will return the same the pointer so the second update does not change anything.

type MapperLookupCache ¶ added in v1.1.0 

type MapperLookupCache struct {
	// contains filtered or unexported fields
}

MapperLookupCache wraps a Mapper with an unsynchronized cache for its Lookup method. Cache hits incur none of the synchronization overhead of Mapper.Lookup.

func (*MapperLookupCache) Lookup ¶ added in v1.1.0 

func (m *MapperLookupCache) Lookup(id int) Type

func (*MapperLookupCache) Reset ¶ added in v1.1.0 

func (m *MapperLookupCache) Reset(mapper *Mapper)

type RecordBuilder ¶ added in v1.4.0 

type RecordBuilder struct {
	// contains filtered or unexported fields
}

func NewRecordBuilder ¶ added in v1.4.0 

func NewRecordBuilder(zctx *Context, fields field.List) (*RecordBuilder, error)

NewRecordBuilder constructs the zcode.Bytes representation for records built from an array of input field selectors expressed as field.Path. Append should be called to enter field values in the left to right order of the provided fields and Encode is called to retrieve the nested zcode.Bytes value. Reset should be called before encoding the next record.

func (*RecordBuilder) Append ¶ added in v1.4.0 

func (r *RecordBuilder) Append(leaf []byte)

func (*RecordBuilder) Encode ¶ added in v1.4.0 

func (r *RecordBuilder) Encode() (zcode.Bytes, error)

func (*RecordBuilder) Reset ¶ added in v1.4.0 

func (r *RecordBuilder) Reset()

func (*RecordBuilder) Type ¶ added in v1.5.0 

func (r *RecordBuilder) Type(types []Type) *TypeRecord

A RecordBuilder understands the shape of the field.List from which it was created (i.e., which fields are inside nested records) but not the types. Type takes types for the individual fields and constructs a TypeRecord reflecting the fully typed structure.

type Type ¶ 

type Type interface {
	// ID returns a unique (per Context) identifier that
	// represents this type.  For a named type, this identifier
	// represents the underlying type and not the named type itself.
	// Callers that care about the underlying type of a Value for
	// example should prefer to use this instead of using a Go
	// type assertion on a Type instance.
	ID() int
	Kind() Kind
}

A Type is an interface presented by a zeek type. Types can be used to infer type compatibility and create new values of the underlying type.

func InnerType ¶ 

func InnerType(typ Type) Type

InnerType returns the element type for the underlying set or array type or nil if the underlying type is not a set or array.

func LookupPrimitive ¶ 

func LookupPrimitive(name string) Type

func LookupPrimitiveByID ¶ 

func LookupPrimitiveByID(id int) (Type, error)

func TypeUnder ¶ added in v1.0.0 

func TypeUnder(typ Type) Type

func UniqueTypes ¶ added in v1.0.0 

func UniqueTypes(types []Type) []Type

UniqueTypes returns the set of unique Types in types in sorted order. types will be sorted and deduplicated in place.

type TypeArray ¶ 

type TypeArray struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeArray ¶ 

func NewTypeArray(id int, typ Type) *TypeArray

func (*TypeArray) ID ¶ 

func (t *TypeArray) ID() int

func (*TypeArray) Kind ¶ added in v1.0.0 

func (t *TypeArray) Kind() Kind

type TypeEnum ¶ 

type TypeEnum struct {
	Symbols []string
	// contains filtered or unexported fields
}

func NewTypeEnum ¶ 

func NewTypeEnum(id int, symbols []string) *TypeEnum

func (*TypeEnum) ID ¶ 

func (t *TypeEnum) ID() int

func (*TypeEnum) Kind ¶ added in v1.0.0 

func (t *TypeEnum) Kind() Kind

func (*TypeEnum) Lookup ¶ 

func (t *TypeEnum) Lookup(symbol string) int

func (*TypeEnum) Symbol ¶ 

func (t *TypeEnum) Symbol(index int) (string, error)

type TypeError ¶ 

type TypeError struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeError ¶ added in v1.0.0 

func NewTypeError(id int, typ Type) *TypeError

func (*TypeError) ID ¶ added in v1.0.0 

func (t *TypeError) ID() int

func (*TypeError) IsMissing ¶ added in v1.0.0 

func (t *TypeError) IsMissing(zv zcode.Bytes) bool

func (*TypeError) IsQuiet ¶ added in v1.0.0 

func (t *TypeError) IsQuiet(zv zcode.Bytes) bool

func (*TypeError) Kind ¶ added in v1.0.0 

func (t *TypeError) Kind() Kind

type TypeMap ¶ 

type TypeMap struct {
	KeyType Type
	ValType Type
	// contains filtered or unexported fields
}

func NewTypeMap ¶ 

func NewTypeMap(id int, keyType, valType Type) *TypeMap

func (*TypeMap) ID ¶ 

func (t *TypeMap) ID() int

func (*TypeMap) Kind ¶ added in v1.0.0 

func (t *TypeMap) Kind() Kind

type TypeNamed ¶ added in v1.0.0 

type TypeNamed struct {
	Name string
	Type Type
	// contains filtered or unexported fields
}

func NewTypeNamed ¶ added in v1.0.0 

func NewTypeNamed(id int, name string, typ Type) *TypeNamed

func (*TypeNamed) ID ¶ added in v1.0.0 

func (t *TypeNamed) ID() int

func (*TypeNamed) Kind ¶ added in v1.0.0 

func (t *TypeNamed) Kind() Kind

func (*TypeNamed) NamedID ¶ added in v1.0.0 

func (t *TypeNamed) NamedID() int

type TypeOfBool ¶ 

type TypeOfBool struct{}

func (*TypeOfBool) ID ¶ 

func (t *TypeOfBool) ID() int

func (*TypeOfBool) Kind ¶ added in v1.0.0 

func (t *TypeOfBool) Kind() Kind

type TypeOfBytes ¶ 

type TypeOfBytes struct{}

func (*TypeOfBytes) Format ¶ 

func (t *TypeOfBytes) Format(zv zcode.Bytes) string

func (*TypeOfBytes) ID ¶ 

func (t *TypeOfBytes) ID() int

func (*TypeOfBytes) Kind ¶ added in v1.0.0 

func (t *TypeOfBytes) Kind() Kind

type TypeOfDuration ¶ 

type TypeOfDuration struct{}

func (*TypeOfDuration) ID ¶ 

func (t *TypeOfDuration) ID() int

func (*TypeOfDuration) Kind ¶ added in v1.0.0 

func (t *TypeOfDuration) Kind() Kind

type TypeOfFloat16 ¶ added in v1.5.0 

type TypeOfFloat16 struct{}

func (*TypeOfFloat16) ID ¶ added in v1.5.0 

func (t *TypeOfFloat16) ID() int

func (*TypeOfFloat16) Kind ¶ added in v1.5.0 

func (t *TypeOfFloat16) Kind() Kind

type TypeOfFloat32 ¶ 

type TypeOfFloat32 struct{}

func (*TypeOfFloat32) ID ¶ 

func (t *TypeOfFloat32) ID() int

func (*TypeOfFloat32) Kind ¶ added in v1.0.0 

func (t *TypeOfFloat32) Kind() Kind

type TypeOfFloat64 ¶ 

type TypeOfFloat64 struct{}

func (*TypeOfFloat64) ID ¶ 

func (t *TypeOfFloat64) ID() int

func (*TypeOfFloat64) Kind ¶ added in v1.0.0 

func (t *TypeOfFloat64) Kind() Kind

type TypeOfIP ¶ 

type TypeOfIP struct{}

func (*TypeOfIP) ID ¶ 

func (t *TypeOfIP) ID() int

func (*TypeOfIP) Kind ¶ added in v1.0.0 

func (t *TypeOfIP) Kind() Kind

type TypeOfInt16 ¶ 

type TypeOfInt16 struct{}

func (*TypeOfInt16) ID ¶ 

func (t *TypeOfInt16) ID() int

func (*TypeOfInt16) Kind ¶ added in v1.0.0 

func (t *TypeOfInt16) Kind() Kind

type TypeOfInt32 ¶ 

type TypeOfInt32 struct{}

func (*TypeOfInt32) ID ¶ 

func (t *TypeOfInt32) ID() int

func (*TypeOfInt32) Kind ¶ added in v1.0.0 

func (t *TypeOfInt32) Kind() Kind

type TypeOfInt64 ¶ 

type TypeOfInt64 struct{}

func (*TypeOfInt64) ID ¶ 

func (t *TypeOfInt64) ID() int

func (*TypeOfInt64) Kind ¶ added in v1.0.0 

func (t *TypeOfInt64) Kind() Kind

type TypeOfInt8 ¶ 

type TypeOfInt8 struct{}

func (*TypeOfInt8) ID ¶ 

func (t *TypeOfInt8) ID() int

func (*TypeOfInt8) Kind ¶ added in v1.0.0 

func (t *TypeOfInt8) Kind() Kind

type TypeOfNet ¶ 

type TypeOfNet struct{}

func (*TypeOfNet) ID ¶ 

func (t *TypeOfNet) ID() int

func (*TypeOfNet) Kind ¶ added in v1.0.0 

func (t *TypeOfNet) Kind() Kind

type TypeOfNull ¶ 

type TypeOfNull struct{}

func (*TypeOfNull) ID ¶ 

func (t *TypeOfNull) ID() int

func (*TypeOfNull) Kind ¶ added in v1.0.0 

func (t *TypeOfNull) Kind() Kind

type TypeOfString ¶ 

type TypeOfString struct{}

func (*TypeOfString) ID ¶ 

func (t *TypeOfString) ID() int

func (*TypeOfString) Kind ¶ added in v1.0.0 

func (t *TypeOfString) Kind() Kind

type TypeOfTime ¶ 

type TypeOfTime struct{}

func (*TypeOfTime) ID ¶ 

func (t *TypeOfTime) ID() int

func (*TypeOfTime) Kind ¶ added in v1.0.0 

func (t *TypeOfTime) Kind() Kind

type TypeOfType ¶ 

type TypeOfType struct{}

func (*TypeOfType) ID ¶ 

func (t *TypeOfType) ID() int

func (*TypeOfType) Kind ¶ added in v1.0.0 

func (t *TypeOfType) Kind() Kind

type TypeOfUint16 ¶ 

type TypeOfUint16 struct{}

func (*TypeOfUint16) ID ¶ 

func (t *TypeOfUint16) ID() int

func (*TypeOfUint16) Kind ¶ added in v1.0.0 

func (t *TypeOfUint16) Kind() Kind

type TypeOfUint32 ¶ 

type TypeOfUint32 struct{}

func (*TypeOfUint32) ID ¶ 

func (t *TypeOfUint32) ID() int

func (*TypeOfUint32) Kind ¶ added in v1.0.0 

func (t *TypeOfUint32) Kind() Kind

type TypeOfUint64 ¶ 

type TypeOfUint64 struct{}

func (*TypeOfUint64) ID ¶ 

func (t *TypeOfUint64) ID() int

func (*TypeOfUint64) Kind ¶ added in v1.0.0 

func (t *TypeOfUint64) Kind() Kind

type TypeOfUint8 ¶ 

type TypeOfUint8 struct{}

func (*TypeOfUint8) ID ¶ 

func (t *TypeOfUint8) ID() int

func (*TypeOfUint8) Kind ¶ added in v1.0.0 

func (t *TypeOfUint8) Kind() Kind

type TypeRecord ¶ 

type TypeRecord struct {
	Fields []Field
	LUT    map[string]int
	// contains filtered or unexported fields
}

func NewTypeRecord ¶ 

func NewTypeRecord(id int, fields []Field) *TypeRecord

func TypeRecordOf ¶ 

func TypeRecordOf(typ Type) *TypeRecord

func (*TypeRecord) HasField ¶ 

func (t *TypeRecord) HasField(field string) bool

func (*TypeRecord) ID ¶ 

func (t *TypeRecord) ID() int

func (*TypeRecord) IndexOfField ¶ added in v1.9.0 

func (t *TypeRecord) IndexOfField(field string) (int, bool)

func (*TypeRecord) Kind ¶ added in v1.0.0 

func (t *TypeRecord) Kind() Kind

func (*TypeRecord) TypeOfField ¶ 

func (t *TypeRecord) TypeOfField(field string) (Type, bool)

type TypeSet ¶ 

type TypeSet struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeSet ¶ 

func NewTypeSet(id int, typ Type) *TypeSet

func (*TypeSet) ID ¶ 

func (t *TypeSet) ID() int

func (*TypeSet) Kind ¶ added in v1.0.0 

func (t *TypeSet) Kind() Kind

type TypeUnion ¶ 

type TypeUnion struct {
	Types []Type
	LUT   map[Type]int
	// contains filtered or unexported fields
}

func NewTypeUnion ¶ 

func NewTypeUnion(id int, types []Type) *TypeUnion

func (*TypeUnion) ID ¶ 

func (t *TypeUnion) ID() int

func (*TypeUnion) Kind ¶ added in v1.0.0 

func (t *TypeUnion) Kind() Kind

func (*TypeUnion) TagOf ¶ added in v1.3.0 

func (t *TypeUnion) TagOf(typ Type) int

TagOf returns the tag for typ in the union. If no type exists -1 is returned.

func (*TypeUnion) Type ¶ 

func (t *TypeUnion) Type(tag int) (Type, error)

Type returns the type corresponding to tag.

func (*TypeUnion) Untag ¶ added in v1.3.0 

func (t *TypeUnion) Untag(bytes zcode.Bytes) (Type, zcode.Bytes)

Untag takes bytes of the reciever's type and returns the underlying value as its type and bytes by removing the tag and determining that tag's type from the union. Untag panics if the tag is invalid.

type TypeVectorTable ¶ 

type TypeVectorTable struct {
	// contains filtered or unexported fields
}

func NewTypeVectorTable ¶ 

func NewTypeVectorTable() *TypeVectorTable

func (*TypeVectorTable) Length ¶ 

func (t *TypeVectorTable) Length() int

func (*TypeVectorTable) Lookup ¶ 

func (t *TypeVectorTable) Lookup(types []Type) int

func (*TypeVectorTable) LookupByValues ¶ 

func (t *TypeVectorTable) LookupByValues(vals []Value) int

func (*TypeVectorTable) Types ¶ 

func (t *TypeVectorTable) Types(id int) []Type

type Value ¶ 

type Value struct {
	// contains filtered or unexported fields
}

func NewBool ¶ 

func NewBool(b bool) Value

func NewBytes ¶ 

func NewBytes(b []byte) Value

func NewDuration ¶ 

func NewDuration(d nano.Duration) Value

func NewFloat ¶ added in v1.9.0 

func NewFloat(t Type, f float64) Value

func NewFloat16 ¶ added in v1.5.0 

func NewFloat16(f float32) Value

func NewFloat32 ¶ 

func NewFloat32(f float32) Value

func NewFloat64 ¶ 

func NewFloat64(f float64) Value

func NewIP ¶ 

func NewIP(a netip.Addr) Value

func NewInt ¶ added in v1.9.0 

func NewInt(t Type, i int64) Value

func NewInt16 ¶ added in v1.5.0 

func NewInt16(i int16) Value

func NewInt32 ¶ added in v1.5.0 

func NewInt32(i int32) Value

func NewInt64 ¶ added in v1.5.0 

func NewInt64(i int64) Value

func NewInt8 ¶ added in v1.5.0 

func NewInt8(i int8) Value

func NewNet ¶ 

func NewNet(p netip.Prefix) Value

func NewString ¶ 

func NewString(s string) Value

func NewTime ¶ 

func NewTime(ts nano.Ts) Value

func NewTypeValue ¶ 

func NewTypeValue(t Type) Value

func NewUint ¶ added in v1.9.0 

func NewUint(t Type, u uint64) Value

func NewUint16 ¶ added in v1.5.0 

func NewUint16(u uint16) Value

func NewUint32 ¶ added in v1.5.0 

func NewUint32(u uint32) Value

func NewUint64 ¶ 

func NewUint64(u uint64) Value

func NewUint8 ¶ added in v1.5.0 

func NewUint8(u uint8) Value

func NewValue ¶ added in v0.32.0 

func NewValue(t Type, b zcode.Bytes) Value

func (Value) ArrayIndex ¶ 

func (v Value) ArrayIndex(idx int64) (Value, error)

If the passed-in element is an array, attempt to get the idx'th element, and return its type and raw representation. Returns an error if the passed-in element is not an array or if idx is outside the array bounds.

func (*Value) AsBool ¶ added in v1.0.0 

func (v *Value) AsBool() bool

AsBool returns v's underlying value. It returns false if v is nil or v's underlying type is not TypeBool.

func (*Value) AsInt ¶ added in v1.0.0 

func (v *Value) AsInt() int64

func (*Value) AsString ¶ added in v1.0.0 

func (v *Value) AsString() string

func (*Value) AsTime ¶ added in v1.0.0 

func (v *Value) AsTime() nano.Ts

func (Value) Bool ¶ added in v1.9.0 

func (v Value) Bool() bool

Bool returns v's underlying value. It panics if v's underlying type is not TypeBool.

func (Value) Bytes ¶ 

func (v Value) Bytes() zcode.Bytes

Bytes returns v's ZNG representation.

func (Value) ContainerLength ¶ 

func (v Value) ContainerLength() (int, error)

func (Value) Copy ¶ 

func (v Value) Copy() Value

Copy returns a copy of v that shares no storage.

func (*Value) CopyFrom ¶ added in v1.0.0 

func (v *Value) CopyFrom(from Value)

CopyFrom copies from into v, reusing v's storage if possible.

func (*Value) Deref ¶ added in v0.32.0 

func (v *Value) Deref(field string) *Value

func (*Value) DerefByColumn ¶ added in v1.0.0 

func (v *Value) DerefByColumn(col int) *Value

func (*Value) DerefPath ¶ added in v1.0.0 

func (v *Value) DerefPath(path field.Path) *Value

func (Value) Elements ¶ 

func (v Value) Elements() ([]Value, error)

Elements returns an array of Values for the given container type. Returns an error if the element is not an array or set.

func (Value) Encode ¶ 

func (v Value) Encode(dst zcode.Bytes) zcode.Bytes

Encode appends the ZNG representation of this value to the passed in argument and returns the resulting zcode.Bytes (which may or may not be the same underlying buffer, as with append(), depending on its capacity)

func (Value) Equal ¶ 

func (v Value) Equal(p Value) bool

Equal reports whether p and v have the same type and the same ZNG representation.

func (Value) Fields ¶ added in v1.5.0 

func (r Value) Fields() []Field

func (Value) Float ¶ added in v1.9.0 

func (v Value) Float() float64

Float returns v's underlying value. It panics if v's underlying type is not TypeFloat16, TypeFloat32, or TypeFloat64.

func (Value) HasField ¶ added in v0.32.0 

func (r Value) HasField(field string) bool

func (Value) IndexOfField ¶ added in v1.9.0 

func (v Value) IndexOfField(field string) (int, bool)

func (Value) Int ¶ added in v1.9.0 

func (v Value) Int() int64

Int returns v's underlying value. It panics if v's underlying type is not TypeInt8, TypeInt16, TypeInt32, TypeInt64, TypeDuration, or TypeTime.

func (Value) IsContainer ¶ 

func (v Value) IsContainer() bool

func (Value) IsError ¶ 

func (v Value) IsError() bool

func (*Value) IsMissing ¶ 

func (v *Value) IsMissing() bool

func (Value) IsNull ¶ added in v1.0.0 

func (v Value) IsNull() bool

IsNull returns true if and only if v is a null value of any type.

func (Value) IsQuiet ¶ added in v1.0.0 

func (v Value) IsQuiet() bool

func (Value) IsString ¶ added in v1.0.0 

func (v Value) IsString() bool

func (Value) Iter ¶ 

func (v Value) Iter() zcode.Iter

func (*Value) MissingAsNull ¶ added in v1.0.0 

func (v *Value) MissingAsNull() Value

func (Value) Ptr ¶ added in v1.13.0 

func (v Value) Ptr() *Value

func (Value) String ¶ 

func (v Value) String() string

String implements fmt.Stringer.String. It should only be used for logs, debugging, etc. Any caller that requires a specific output format should use FormatAs() instead.

func (Value) Type ¶ 

func (v Value) Type() Type

func (Value) Uint ¶ added in v1.9.0 

func (v Value) Uint() uint64

Uint returns v's underlying value. It panics if v's underlying type is not TypeUint8, TypeUint16, TypeUint32, or TypeUint64.

func (Value) Under ¶ added in v1.3.0 

func (v Value) Under() Value

Under resolves named types and untags unions repeatedly, returning a value guaranteed to have neither a named type nor a union type.

func (Value) Validate ¶ added in v1.4.0 

func (v Value) Validate() (err error)

Validate checks that v.Bytes is structurally consistent with v.Type. It does not check that the actual leaf values when parsed are type compatible with the leaf types.

func (Value) Walk ¶ added in v0.32.0 

func (r Value) Walk(rv Visitor) error

Walk traverses a value in depth-first order, calling a Visitor on the way.

type Visitor ¶ 

type Visitor func(typ Type, body zcode.Bytes) error

A Visitor is called for each value in a record encountered by Walk. If the visitor returns an error, the walk stops and that error will be returned to the caller of Walk(). The sole exception is when the visitor returns the special value SkipContainer.

Source Files ¶

View all Source files

Directories ¶

Path Synopsis
api
client
client/auth0
queryio
cli
auto
cacheflags
commitflags
inputflags
lakeflags
logflags
outputflags
poolflags
queryflags
runtimeflags
zq
cmd
gentoken
mockzui
zc
zed
zed/auth
zed/branch
zed/compact
zed/create
zed/delete
zed/dev
zed/dev/compile
zed/dev/dig
zed/dev/dig/frames
zed/dev/dig/slice
zed/dev/vector
zed/dev/vector/agg
zed/dev/vector/copy
zed/dev/vector/project
zed/dev/vector/query
zed/dev/vector/search
zed/dev/vng
zed/drop
zed/init
zed/internal/lakemanage
zed/load
zed/log
zed/ls
zed/manage
zed/merge
zed/query
zed/rename
zed/revert
zed/root
zed/serve
zed/use
zed/vacate
zed/vacuum
zed/vector
zq
ast
Package ast declares the types used to represent syntax trees for Zed queries.
 Package ast declares the types used to represent syntax trees for Zed queries.
ast/dag
ast/zed
data
kernel
optimizer
optimizer/demand
parser
semantic
api
branches
commits
data
journal
pools
seekindex
Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
 Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
pkg
anymath
bufwriter
Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
 Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
byteconv
Package byteconv implements conversions from byte slice representations of various data types.
 Package byteconv implements conversions from byte slice representations of various data types.
charm
Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
 Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
colw
Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
 Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
display
field
fs
grok
httpd
jsonlexer
nano
peeker
plural
promtest
reglob
repl
Package repl is a simple read-eval-print loop.
 Package repl is a simple read-eval-print loop.
s3io
skim
storage
storage/cache
Package cache contains facilities for caching immutable files, typically for a cloud object store.
 Package cache contains facilities for caching immutable files, typically for a cloud object store.
storage/mock
Package mock is a generated GoMock package.
 Package mock is a generated GoMock package.
stringsearch
terminal
terminal/color
units
unpack
Package unpack provides a means to unmarshal Go values that have embedded interface values.
 Package unpack provides a means to unmarshal Go values that have embedded interface values.
exec
sam/expr
sam/expr/agg
sam/expr/coerce
sam/expr/extent
sam/expr/function
sam/expr/result
sam/op
sam/op/combine
A combine proc merges multiple upstream inputs into one output.
 A combine proc merges multiple upstream inputs into one output.
sam/op/explode
sam/op/exprswitch
sam/op/fork
sam/op/fuse
sam/op/groupby
sam/op/head
sam/op/join
sam/op/load
sam/op/merge
sam/op/meta
sam/op/pass
sam/op/shape
sam/op/sort
sam/op/spill
sam/op/switcher
sam/op/tail
sam/op/top
sam/op/traverse
sam/op/uniq
sam/op/yield
vam
vam/expr
vam/op
vcache
auth
logger
srverr
Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
 Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
Package vng implements the reading and writing of VNG serialization objects.
 Package vng implements the reading and writing of VNG serialization objects.
Package zcode implements serialization and deserialzation for ZNG values.
 Package zcode implements serialization and deserialzation for ZNG values.
zio
anyio
arrowio
csvio
emitter
jsonio
lakeio
lineio
parquetio
tableio
textio
vngio
zeekio
zjsonio
zngio
Package zngio provides an API for reading and writing zng values and directives in binary zng format.
 Package zngio provides an API for reading and writing zng values and directives in binary zng format.
zsonio
Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
 Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.
 Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.