README
¶
protoparts
Facilities for splitting apart binary Protocol Buffer messages into individual fields, and combining the parts again into whole (or partial) Protocol Buffer messages.
The motivation is to enable Protocol Buffers to be used as a storage format, where fields need to be addressable on-disk individually to enable projection, selection, and so on.
Documentation
¶
Index ¶
- type Part
- type Parts
- func (ps Parts) Exclude(prefixes ...Path) Parts
- func (ps Parts) Join() []byte
- func (ps Parts) Len() int
- func (ps Parts) Less(x, y int) bool
- func (ps Parts) Paths() []Path
- func (ps Parts) ProtoMessage() protoreflect.Message
- func (ps Parts) ProtoValue(p Path) (protoreflect.Value, bool)
- func (ps Parts) Select(prefixes ...Path) Parts
- func (ps Parts) Swap(x, y int)
- func (ps Parts) Value(p Path) []byte
- type Path
- func (p Path) Append(y Path) Path
- func (p Path) Compare(q Path) int
- func (p Path) Equal(q Path) bool
- func (p Path) HasPrefix(prefix Path) bool
- func (p Path) IsValid(md protoreflect.MessageDescriptor) bool
- func (p Path) Last() PathTerm
- func (p Path) Parent() Path
- func (p Path) String() string
- func (p Path) SymbolicString(md protoreflect.MessageDescriptor) (string, error)
- type PathTerm
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Part ¶
type Part struct {
Path Path
Type protowire.Type
// KeyType contains the wire type of the key, if this Part represents an entry within a map. -1 if not.
KeyType protowire.Type
// Bytes contains the Protobuf-encoded field value
Bytes []byte
// Md contains the message descriptor that the Path can be found in
Md protoreflect.MessageDescriptor
}
A Part represents a field within a binary Protocol Buffer message.
func (Part) Encode ¶
Encode adds the necessary prefix to Bytes to make a valid Protocol Buffer for just this field.
type Parts ¶
type Parts []Part
Parts are the combination of all the (potentially nested) fields within a serialised Protocol Buffer message.
Fields are ordered, and the order is important when encoding because protobuf can only decode messages when, for example, repeated elements are encoded in-order, the fields of nested messages are serialised together in a 'run' (but not necessarily in an order within that run), etc. This means there can be many valid encoding orders for a single message, but not all conceivable orders are valid. To always ensure a valid serialisation will result, Parts can be sorted to produce a canonical order. However, this may unnecessarily change an already-valid field order, so it is unnecessary to sort if the order is known to be good – for example if the Parts were produced by Split.
func Marshal ¶
Marshal is a convenient way to construct Parts from a Protocol Buffer message struct, without the need for the caller to do their own marshaling before calling Split().
func Merge ¶
Merge merges p1 and p2, with items in p2 taking precedence over p1. The resulting Parts are sorted.
func Split ¶
func Split(pb []byte, md protoreflect.MessageDescriptor) (Parts, error)
Split explodes a serialised Protocol Buffer into parts of its constituent fields and those within nested messages. The resulting parts are returned in the order they appear in, so recombining the parts (unless they are reordered either manually or by sorting them) yields a byte-wise identical message.
func (Parts) Exclude ¶
Exclude returns all the parts that are not prefixed with any of the passed paths. The order of the resulting parts will be the same as their order before exclusion.
func (Parts) Join ¶
Join stitches the parts back together as a serialised Protocol Buffer message.
Warning: If the order of fields is not valid, Join() may produce malformed output which cannot be parsed as a Protocol Buffer. If the order of parts is not known to be valid, sort them first.
func (Parts) ProtoMessage ¶
func (ps Parts) ProtoMessage() protoreflect.Message
ProtoMessage returns the unmarshalled protoreflect.Message for the entire Parts assembly. If the message cannot be unmarshalled, returns nil.
func (Parts) ProtoValue ¶
func (ps Parts) ProtoValue(p Path) (protoreflect.Value, bool)
ProtoValue returns (protoreflect.Value, true) for the given Path, or (protoreflect.Value{}, false) if it could not be found or unmarshalled.
func (Parts) Select ¶
Select returns all the parts that are prefixed with any of the passed paths. The order of selected parts will be the same as their order before selection.
func (Parts) Value ¶
Value returns the Protocol Buffer for the given Path, or nil if it could not be found which is possible if:
- the path is invalid within the Protocol Buffer
- the field was not set within the original Protocol Buffer
- the field is contained within a sub-message, and no message descriptor was provided
type Path ¶
type Path []PathTerm
A Path is a hierarchical path to a (potentially deeply-nested) value within a serialised Protocol Buffer, identified by the tag numbers, and potentially indices and keys, of its ancestors.
A nil Path is different from an empty Path: a nil path addresses nothing, but an empty path addresses the root of a message.
func DecodePath ¶
DecodePath turns an encoded Path of the variety returned by PathTerm.String() into a Path. If the path cannot be decoded returns nil.
If the string is to be constructed by hand, its format is a slash-separated concatenation of the strings of its terms (see DecodePathTerm).
func DecodeSymbolicPath ¶
func DecodeSymbolicPath(s string, md protoreflect.MessageDescriptor) Path
DecodeSymbolicPath turns a symbolically encoded Path (as returned by Path.SymbolicString()) back into a Path. If the path cannot be decoded returns nil.
If the string is to be constructed by hand, its format is a slash-separated concatenation of the symbolic strings of\ its terms (see DecodeSymbolicPathTerm).
func (Path) Compare ¶
Compare returns an integer defining the ordering relationship between the two paths. Much as with bytes.Compare, the result will be 0 if p==q, -1 if p<q, and +1 if p>q.
func (Path) HasPrefix ¶
HasPrefix returns whether this Path is a descendant of the prefix. True if the prefix is equal to the Path.
func (Path) IsValid ¶
func (p Path) IsValid(md protoreflect.MessageDescriptor) bool
IsValid returns whether this path is valid for the passed message descriptor. This does not necessarily mean that lookups are guaranteed to work; this can check the fields used are valid, but if there are list indices it cannot know what its cardinality will be.
func (Path) Last ¶
Last returns the last element of the path. Eg: 1/2/3 -> 3. If the path is empty, this will panic.
func (Path) Parent ¶
Parent returns a Path with the last element of this one removed. Eg: 1/2/3 -> 1/2
func (Path) String ¶
String encodes the Path into a slash-separated string, eg: "1/2/3[4]/5[0x0a]/6[7]". This string can be decoded using DecodePath.
The format of this string is suitable for long-term storage and is suitable for equality comparisons.
func (Path) SymbolicString ¶
func (p Path) SymbolicString(md protoreflect.MessageDescriptor) (string, error)
SymbolicString returns a human-readable representation of the Path with field names in place of tag numbers, eg: "foo/bar/baz[4]/foobar[0x0a]/foobaz[7]". This string can be decoded using DecodeSymbolicPath.
Warning: Fields can be renamed freely in Protocol Buffers without affecting their serialisation. So the output of this function is NOT guaranteed to be stable, should NOT be stored and is only intended for display to humans. For storage, the output of String() should be used since it does not rely on the stability of field names within a message descriptor to reconstruct.
type PathTerm ¶
type PathTerm struct {
// Tag is the tag number of the field.
Tag protowire.Number
// Index is an index into a repeated sequence. -1 indicates the absence of an index.
Index int
// Key is the key for the value in a map
Key []byte
}
A PathTerm is a single item in a hierarchical Path. It encodes a tag number, and an index denoting which item in a repeated sequence the value belongs to.
func DecodePathTerm ¶
DecodePathTerm turns an string of the variety returned by PathTerm.String() into a PathTerm. If the term cannot be decoded returns a zero PathTerm which can be identified if its Tag == 0 (no valid Protocol Buffer can have a tag <1).
If the string is to be constructed by hand, its format is tag[idx][0xkey], where:
- tag is the base-10 representation of the field's tag number
- idx, which is optional, is the base-10 representation of an ordinal within a repeated field, to select an individual element from a list
- key, which is optional, is a hex-encoded binary protocol buffer representation prefixed with 0x, of an key within a map, to select an individual value from a map
For example: 1 to select the field with number 1, 2[0x0a] to select the key 0a from the map with field number 2, or 4[5] to select the fifth item from a list with field number 4.
func DecodeSymbolicPathTerm ¶
func DecodeSymbolicPathTerm(s string, md protoreflect.MessageDescriptor) PathTerm
DecodeSymbolicPathTerm turns a symbolically encoded PathTerm as returned by PathTerm.SymbolicString() back into a PathTerm. If the term cannot be decoded returns a zero PathTerm which can be identified if its Tag == 0 (no valid Protocol Buffer can have a tag <1).
If the string is to be constructed by hand, its format is field[idx][0xkey], where:
- field is the name of the field
- idx, which is optional, is the base-10 representation of an ordinal within a repeated field, to select an individual element from a list
- key, which is optional, is a hex-encoded binary protocol buffer representation prefixed with 0x, of an key within a map, to select an individual value from a map
For example: foo to select the field with name foo, bar[0x0a] to select the key 0a from the map with field name bar, or baz[5] to select the fifth item from a list with field name baz.
func (PathTerm) Compare ¶
Compare returns an integer defining the ordering between the two terms. Much as with bytes.Compare, the result is 0 if t==u, -1 if t<u, and +1 if t>u.
func (PathTerm) SymbolicString ¶
func (t PathTerm) SymbolicString(fd protoreflect.FieldDescriptor) string
Source Files
Directories