Documentation
¶
Overview ¶
Package starlarkproto exposes protobuf messages as Starlark types.
Internally a message is stored as a tree of Starlark values, with some type checking done when manipulating fields, based on proto message descriptors loaded dynamically from a serialized FileDescriptorSet.
For example, reading or assigning to a field not defined in a message will cause a runtime error. Similarly, trying to assign a value of a wrong type to a non-repeated field will fail.
Instantiating messages and default field values ¶
Each proto message in a loaded package is exposed via a constructor function that takes optional keyword arguments and produces a new object of *Message type.
All unassigned fields are implicitly set to their default zero values on first access, including message-typed fields, lists and maps. It means, for example, if a message 'a' has a singular field 'b', that has a field 'c', it is always fine to write 'a.b.c' to read or set 'c' value, without explicitly checking that 'b' is set.
To clear a field, assign None to it (regardless of its type).
References and aliasing ¶
Messages are passed around everywhere by reference. In particular it is possible to have multiple fields pointing to the exact same message, e.g.
m1 = M() m2 = M() a = A() m1.f = a m2.f = a a.i = 123 # changes both m1.f.i and m2.f.i
Note that 'm1.f = a' assignment checks the type of 'a', it should either match type of 'f' identically (no duck typing), or be a dict or None (which will be converted to messages, see below).
Working with repeated fields and maps ¶
Values of repeated fields are represented by special sequence types that behave as strongly-typed lists. Think of them as list[T] types or as hypothetical 'message List<T>' messages.
When assigning a non-None value R to a repeated field F of type list[T], the following rules apply (sequentially):
- If R is not a sequence => error.
- If R has type list[T'], then a. If T == T', then F becomes an alias of R. b. If T != T' => error.
- A new list[T] is instantiated from R and assigned to F.
Notice that rule 2 is exactly like the aliasing rule for messages. This is where "think of them as 'message List<T>'" point of view comes into play.
As a concrete example, consider this:
m1 = M() m1.int64s = [1, 2] # a list is implicitly converted (copied) to list[T] m2 = M() m2.int64s = m1.int64s # points to the exact same list[T] object now m2.int64s.append(3) # updates both m1 and m2 m1.int32s = m1.int64s # error! list[int64] is not list[int32] m1.int32s = list(m1.int64s) # works now (by making a copy of a list copy)
Maps behave in the similar way. Think of them as strongly-typed map<K,V> values or as 'message List<Pair<K,V>>' messages. They can be implicitly instantiated from iterable mappings (e.g. dicts).
Auto-conversion of dicts and None's into Messages ¶
When assigning to a message-typed value (be it a field, an element of a list[T] or a value of map<K,V>) dicts are implicitly converted into messages as if via 'T(**d)' call. Similarly, None's are converted into empty messages, as if via 'T()' call.
Differences from starlarkproto (beside using different guts):
- Message types are instantiated through proto.new_loader().
- Text marshaller appends\removes trailing '\n' somewhat differently.
- Text marshaller marshals empty messages as '<>' (without line breaks).
- Bytes fields are represented as str, not as []uint8{...}.
- proto.to_jsonpb doesn't have 'emit_defaults' kwarg (it is always False).
- Better support for proto2 messages.
Index ¶
- func ProtoLib() starlark.StringDict
- func SetDefaultLoader(th *starlark.Thread, l *Loader)
- func SetMessageCache(th *starlark.Thread, mc MessageCache)
- func ToJSONPB(msg *Message, useProtoNames bool) ([]byte, error)
- func ToTextPB(msg *Message) ([]byte, error)
- func ToWirePB(msg *Message) ([]byte, error)
- type DescriptorSet
- func (ds *DescriptorSet) Attr(name string) (starlark.Value, error)
- func (ds *DescriptorSet) AttrNames() []string
- func (ds *DescriptorSet) Freeze()
- func (ds *DescriptorSet) Hash() (uint32, error)
- func (ds *DescriptorSet) String() string
- func (ds *DescriptorSet) Truth() starlark.Bool
- func (ds *DescriptorSet) Type() string
- type Loader
- func (l *Loader) AddDescriptorSet(ds *DescriptorSet) error
- func (l *Loader) Attr(name string) (starlark.Value, error)
- func (l *Loader) AttrNames() []string
- func (l *Loader) Freeze()
- func (l *Loader) Hash() (uint32, error)
- func (l *Loader) MessageType(desc protoreflect.MessageDescriptor) *MessageType
- func (l *Loader) Module(path string) (*starlarkstruct.Module, error)
- func (l *Loader) String() string
- func (l *Loader) Truth() starlark.Bool
- func (l *Loader) Type() string
- func (l *Loader) Types() *protoregistry.Types
- type Message
- func (m *Message) Attr(name string) (starlark.Value, error)
- func (m *Message) AttrNames() []string
- func (m *Message) CompareSameType(op syntax.Token, y starlark.Value, depth int) (bool, error)
- func (m *Message) Freeze()
- func (m *Message) FromDict(d starlark.IterableMapping) error
- func (m *Message) HasProtoField(name string) bool
- func (m *Message) Hash() (uint32, error)
- func (m *Message) IsFrozen() bool
- func (m *Message) MessageType() *MessageType
- func (m *Message) SetField(name string, val starlark.Value) error
- func (m *Message) String() string
- func (m *Message) ToProto() proto.Message
- func (m *Message) Truth() starlark.Bool
- func (m *Message) Type() string
- type MessageCache
- type MessageType
- func (t *MessageType) Attr(name string) (starlark.Value, error)
- func (t *MessageType) AttrNames() []string
- func (t *MessageType) Converter() typed.Converter
- func (t *MessageType) Descriptor() protoreflect.MessageDescriptor
- func (t *MessageType) Loader() *Loader
- func (t *MessageType) Message() *Message
- func (t *MessageType) MessageFromProto(p proto.Message) *Message
- func (t *MessageType) String() string
- func (t *MessageType) Type() string
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ProtoLib ¶
func ProtoLib() starlark.StringDict
ProtoLib returns a dict with single struct named "proto" that holds public Starlark API for working with proto messages.
Exported functions:
def new_descriptor_set(name=None, blob=None, deps=None):
"""Returns a new DescriptorSet.
Args:
name: name of this set for debug and error messages, default is '???'.
blob: raw serialized FileDescriptorSet, if any.
deps: an iterable of DescriptorSet's with dependencies, if any.
Returns:
New DescriptorSet.
"""
def new_loader(*descriptor_sets):
"""Returns a new proto loader."""
def default_loader():
"""Returns a loader used by default when registering descriptor sets."""
def message_type(msg):
"""Returns proto.MessageType of the given message."""
def to_textpb(msg):
"""Serializes a protobuf message to text proto.
Args:
msg: a *Message to serialize.
Returns:
A str representing msg in text format.
"""
def to_jsonpb(msg, use_proto_names = None):
"""Serializes a protobuf message to JSONPB string.
Args:
msg: a *Message to serialize.
use_proto_names: boolean, whether to use snake_case in field names
instead of camelCase. The default is False.
Returns:
A str representing msg in JSONPB format.
"""
def to_wirepb(msg):
"""Serializes a protobuf message to a string using binary wire encoding.
Args:
msg: a *Message to serialize.
Returns:
A str representing msg in binary wire format.
"""
def from_textpb(ctor, body):
"""Deserializes a protobuf message given in text proto form.
Unknown fields are not allowed.
Args:
ctor: a message constructor function.
body: a string with serialized message.
discard_unknown: boolean, whether to discard unrecognized fields. The
default is False.
Returns:
Deserialized frozen message constructed via `ctor`.
"""
def from_jsonpb(ctor, body):
"""Deserializes a protobuf message given as JBONPB string.
Unknown fields are silently skipped.
Args:
ctor: a message constructor function.
body: a string with serialized message.
discard_unknown: boolean, whether to discard unrecognized fields. The
default is True.
Returns:
Deserialized frozen message constructed via `ctor`.
"""
def from_wirepb(ctor, body):
"""Deserializes a protobuf message given its wire serialization.
Unknown fields are silently skipped.
Args:
ctor: a message constructor function.
body: a string with serialized message.
discard_unknown: boolean, whether to discard unrecognized fields. The
default is True.
Returns:
Deserialized frozen message constructed via `ctor`.
"""
def struct_to_textpb(s):
"""Converts a struct to a text proto string.
Args:
s: a struct object. May not contain dicts.
Returns:
A str containing a text format protocol buffer message.
"""
def clone(msg):
"""Returns a deep copy of a given proto message.
Args:
msg: a proto message to make a copy of.
Returns:
A deep copy of the message
"""
def has(msg, field):
"""Checks if a proto message has the given optional field set.
Args:
msg: a message to check.
field: a string name of the field to check.
Returns:
True if the message has the field set.
"""
func SetDefaultLoader ¶
SetDefaultLoader installs the given loader as default in the thread.
It can be obtained via DefaultLoader or proto.default_loader() from Starlark. Note that Starlark code has no way of changing the default loader. It's responsibility of the hosting environment to prepare the default loader (just like it prepares starlark.Thread itself).
func SetMessageCache ¶
func SetMessageCache(th *starlark.Thread, mc MessageCache)
SetMessageCache installs the given messages cache into the thread.
It will be used by all deserialization operations performed by the Starlark code in this thread.
Types ¶
type DescriptorSet ¶
type DescriptorSet struct {
// contains filtered or unexported fields
}
DescriptorSet contains FileDescriptorProto of zero or more *.proto files, along with pointers to DescriptorSets with their imports.
A descriptor set can be registered in a Loader. Doing so transitively registers all its dependencies. See Loader.AddDescriptorSet for more details.
Implements starlark.Value and starlark.HasAttrs interfaces. Usage from Starlark side may look like this:
load(".../wellknown_descpb.star", "wellknown_descpb")
myprotos_descpb = proto.new_descriptor_set(
name = "myprotos",
deps = [wellknown_descpb],
blob = io.read_file("myprotos.descpb"),
)
myprotos_descpb.register()
By default register() registers the descriptor set in the default loader, i.e. ds.register() is same as ds.register(loader=proto.default_loader()). Also note that ds.register(loader=l) is a sugar for l.add_descriptor_set(ds), so ds.register() is same as proto.default_loader().add_descriptor_set(ds), just much shorter.
func NewDescriptorSet ¶
func NewDescriptorSet(name string, fdps []*descriptorpb.FileDescriptorProto, deps []*DescriptorSet) (*DescriptorSet, error)
NewDescriptorSet evaluates given file descriptors and their dependencies and produces new DescriptorSet if there are no unresolved imports and no duplicated files.
'fdps' should be ordered topologically (i.e. if file A imports file B, then B should precede A in 'fdps' or be somewhere among 'deps'). This is always the case when generating sets via 'protoc --descriptor_set_out=...'.
Note that dependencies can only be specified when creating the descriptor set and can't be changed later. Cycles thus are impossible.
func (*DescriptorSet) Attr ¶
func (ds *DescriptorSet) Attr(name string) (starlark.Value, error)
Attr returns an attribute given its name (or nil if not present).
func (*DescriptorSet) AttrNames ¶
func (ds *DescriptorSet) AttrNames() []string
AtrrNames lists available attributes.
func (*DescriptorSet) Freeze ¶
func (ds *DescriptorSet) Freeze()
Freeze does nothing since DescriptorSet is already immutable.
func (*DescriptorSet) Hash ¶
func (ds *DescriptorSet) Hash() (uint32, error)
Hash returns unique value associated with this set.
func (*DescriptorSet) String ¶
func (ds *DescriptorSet) String() string
String returns str(...) representation of the set, for debug messages.
func (*DescriptorSet) Truth ¶
func (ds *DescriptorSet) Truth() starlark.Bool
Truth returns the truth value of an object.
func (*DescriptorSet) Type ¶
func (ds *DescriptorSet) Type() string
Type returns a short string describing the value's type.
type Loader ¶
type Loader struct {
// contains filtered or unexported fields
}
Loader can instantiate Starlark values that correspond to proto messages.
Holds a pool of descriptors that describe all available proto types. Use AddDescriptorSet to seed it. Once seeded, use Module to get a Starlark module with symbols defined in some registered `*.proto` file.
Loader is also a Starlark value itself, with the following methods:
- add_descriptor_set(ds) - see AddDescriptorSet.
- module(path) - see Module.
Can be used concurrently. Non-freezable.
func DefaultLoader ¶
DefaultLoader returns a loader installed in the thread via SetDefaultLoader.
Returns nil if there's no default loader.
func NewLoader ¶
func NewLoader() *Loader
NewLoader instantiates a new loader with empty proto registry.
func (*Loader) AddDescriptorSet ¶
func (l *Loader) AddDescriptorSet(ds *DescriptorSet) error
AddDescriptorSet makes all *.proto files defined in the given descriptor set and all its dependencies available for use from Starlark.
AddDescriptorSet is idempotent in a sense that calling AddDescriptorSet(ds) multiple times with the exact same 'ds' is not an error. But trying to register a proto file through multiple different descriptor sets is an error.
func (*Loader) MessageType ¶
func (l *Loader) MessageType(desc protoreflect.MessageDescriptor) *MessageType
MessageType creates new (or returns existing) MessageType.
The return value can be used to instantiate Starlark values via Message() or MessageFromProto(m).
func (*Loader) Module ¶
func (l *Loader) Module(path string) (*starlarkstruct.Module, error)
Module returns a module with top-level definitions from some *.proto file.
The descriptor of this proto file should be registered already via AddDescriptorSet. 'path' here is matched to what's in the descriptor, which is a path to *.proto EXACTLY as it was given to 'protoc'.
The name of the module matches the proto package name (per 'package ...' statement in the proto file).
func (*Loader) Types ¶
func (l *Loader) Types() *protoregistry.Types
Types returns a registry for looking up or iterating over descriptor types.
type Message ¶
type Message struct {
// contains filtered or unexported fields
}
Message is a Starlark value that implements a struct-like type structured like a protobuf message.
Implements starlark.Value, starlark.HasAttrs, starlark.HasSetField and starlark.Comparable interfaces.
Can be instantiated through Loader as loader.MessageType(...).Message() or loader.MessageType(...).MessageFromProto(p).
TODO(vadimsh): Currently not safe for a cross-goroutine use without external locking, even when frozen, due to lazy initialization of default fields on first access.
func FromJSONPB ¶
func FromJSONPB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)
FromJSONPB deserializes a protobuf message given as JBONPB string.
Unlike the equivalent Starlark proto.from_jsonpb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.
func FromTextPB ¶
func FromTextPB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)
FromTextPB deserializes a protobuf message given in text proto form.
Unlike the equivalent Starlark proto.from_textpb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.
func FromWirePB ¶
func FromWirePB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)
FromWirePB deserializes a protobuf message given as a wire-encoded blob.
Unlike the equivalent Starlark proto.from_wirepb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.
func (*Message) CompareSameType ¶
CompareSameType does 'm <op> y' comparison.
func (*Message) FromDict ¶
func (m *Message) FromDict(d starlark.IterableMapping) error
FromDict populates fields of this message based on values in an iterable mapping (usually a starlark.Dict).
Doesn't reset the message. Basically does this:
for k in d: setattr(msg, k, d[k])
Returns an error on type mismatch.
func (*Message) HasProtoField ¶
HasProtoField returns true if the message has the given field initialized.
func (*Message) MessageType ¶
func (m *Message) MessageType() *MessageType
MessageType returns type information about this message.
type MessageCache ¶
type MessageCache interface {
// Fetch returns a previously stored message or (nil, nil) if missing.
Fetch(th *starlark.Thread, cache, body string, typ *MessageType) (*Message, error)
// Store stores a deserialized message.
Store(th *starlark.Thread, cache, body string, msg *Message) error
}
MessageCache knows how to cache deserialized messages.
Semantically it is a map `(cache, typ, body) => frozen Message`, where `cache` identifies a serialization format, `typ` identifies the type being deserialized and `body` is the raw message body to be deserialized.
Works with frozen messages.
Use SetMessageCache to install a cache into a starlark thread.
type MessageType ¶
type MessageType struct {
*starlark.Builtin // the callable, initialize in Loader
// contains filtered or unexported fields
}
MessageType represents a proto message type and acts as its constructor: it is a Starlark callable that produces instances of Message.
Implements starlark.HasAttrs interface. Attributes represent constructors for nested messages and int values of enums. Note that starlark.HasSetField is not implemented, making values of MessageType immutable.
Stringifying an instance of MessageType (e.g. with `str(...)` in Starlark) returns its full proto type name.
Given a MessageDescriptor, MessageType can be instantiated through Loader as loader.MessageType(...).
func (*MessageType) Attr ¶
func (t *MessageType) Attr(name string) (starlark.Value, error)
Attr returns either a nested message or an enum value.
func (*MessageType) AttrNames ¶
func (t *MessageType) AttrNames() []string
AttrNames return names of all nested messages and enum values.
func (*MessageType) Converter ¶
func (t *MessageType) Converter() typed.Converter
Converter returns an object that can convert Starlark dicts and Nones to values of this message type.
Can be used by typed.List and typed.Dict.
func (*MessageType) Descriptor ¶
func (t *MessageType) Descriptor() protoreflect.MessageDescriptor
Descriptor returns protobuf type information for this message type.
func (*MessageType) Loader ¶
func (t *MessageType) Loader() *Loader
Loader returns the loader this message type was loaded from.
func (*MessageType) Message ¶
func (t *MessageType) Message() *Message
Message instantiates a new empty message of this type.
func (*MessageType) MessageFromProto ¶
func (t *MessageType) MessageFromProto(p proto.Message) *Message
MessageFromProto instantiates a new message of this type and populates it based on values in the given proto.Message that should have a matching type.
Here "matching type" means p.ProtoReflect().Descriptor() *is* t.Descriptor(). Panics otherwise.
func (*MessageType) String ¶
func (t *MessageType) String() string
String returns the full proto type name.
func (*MessageType) Type ¶
func (t *MessageType) Type() string
Type returns "proto.MessageType", it's the type of the message type itself.
It is sort of like a meta-type, i.e. like "type" type in Python.
Source Files
Directories