luci: go.chromium.org/luci/starlark/starlarkproto Index | Files | Directories

package starlarkproto

import "go.chromium.org/luci/starlark/starlarkproto"

Package starlarkproto exposes protobuf messages as Starlark types.

Uses a slightly modified vendored copy of "google.golang.org/protobuf" internally. It's expected that once "google.golang.org/protobuf" is officially released, starlarkproto will switch to using the released version.

Internally a message is stored as a tree of Starlark values, with some type checking done when manipulating fields, based on proto message descriptors loaded dynamically from a serialized FileDescriptorSet.

For example, reading or assigning to a field not defined in a message will cause a runtime error. Similarly, trying to assign a value of a wrong type to a non-repeated field will fail.

Instantiating messages and default field values

Each proto message in a loaded package is exposed via a constructor function that takes optional keyword arguments and produces a new object of *Message type.

All unassigned fields are implicitly set to their default zero values on first access, including message-typed fields, lists and maps. It means, for example, if a message 'a' has a singular field 'b', that has a field 'c', it is always fine to write 'a.b.c' to read or set 'c' value, without explicitly checking that 'b' is set.

To clear a field, assign None to it (regardless of its type).

References and aliasing

Messages are passed around everywhere by reference. In particular it is possible to have multiple fields pointing to the exact same message, e.g.

m1 = M()
m2 = M()
a = A()
m1.f = a
m2.f = a
a.i = 123  # changes both m1.f.i and m2.f.i

Note that 'm1.f = a' assignment checks the type of 'a', it should either match type of 'f' identically (no duck typing), or be a dict or None (which will be converted to messages, see below).

Working with repeated fields and maps

Values of repeated fields are represented by special sequence types that behave as strongly-typed lists. Think of them as list[T] types or as hypothetical 'message List<T>' messages.

When assigning a non-None value R to a repeated field F of type list[T], the following rules apply (sequentially):

1. If R is not a sequence => error.
2. If R has type list[T'], then
   a. If T == T', then F becomes an alias of R.
   b. If T != T' => error.
3. A new list[T] is instantiated from R and assigned to F.

Notice that rule 2 is exactly like the aliasing rule for messages. This is where "think of them as 'message List<T>'" point of view comes into play.

As a concrete example, consider this:

m1 = M()
m1.int64s = [1, 2]  # a list is implicitly converted (copied) to list[T]

m2 = M()
m2.int64s = m1.int64s  # points to the exact same list[T] object now
m2.int64s.append(3)    # updates both m1 and m2

m1.int32s = m1.int64s        # error! list[int64] is not list[int32]
m1.int32s = list(m1.int64s)  # works now (by making a copy of a list copy)

Maps behave in the similar way. Think of them as strongly-typed map<K,V> values or as 'message List<Pair<K,V>>' messages. They can be implicitly instantiated from iterable mappings (e.g. dicts).

Auto-conversion of dicts and None's into Messages

When assigning to a message-typed value (be it a field, an element of a list[T] or a value of map<K,V>) dicts are implicitly converted into messages as if via 'T(**d)' call. Similarly, None's are converted into empty messages, as if via 'T()' call.

Differences from starlarkproto (beside using different guts):

* Message types are instantiated through proto.new_loader().
* Text marshaller appends\removes trailing '\n' somewhat differently.
* Text marshaller marshals empty messages as '<>' (without line breaks).
* Bytes fields are represented as str, not as []uint8{...}.
* proto.to_jsonpb doesn't have 'emit_defaults' kwarg (it is always False).
* Better support for proto2 messages.

Index

Package Files

descriptor_set.go doc.go functions.go loader.go message.go type.go values.go values_list.go values_map.go values_singular.go

func ProtoLib Uses

func ProtoLib() starlark.StringDict

ProtoLib returns a dict with single struct named "proto" that holds public Starlark API for working with proto messages.

Exported functions:

def new_descriptor_set(name=None, blob=None, deps=None):
  """Returns a new DescriptorSet.

  Args:
    name: name of this set for debug and error messages, default is '???'.
    blob: raw serialized FileDescriptorSet, if any.
    deps: an iterable of DescriptorSet's with dependencies, if any.

  Returns:
    New DescriptorSet.
  """

def new_loader(*descriptor_sets):
  """Returns a new proto loader."""

def default_loader():
  """Returns a loader used by default when registering descriptor sets."""

def to_textpb(msg):
  """Serializes a protobuf message to text proto.

  Args:
    msg: a *Message to serialize.

  Returns:
    A str representing msg in text format.
  """

def to_jsonpb(msg):
  """Serializes a protobuf message to JSONPB string.

  Args:
    msg: a *Message to serialize.

  Returns:
    A str representing msg in JSONPB format.
  """

def from_textpb(ctor, body):
  """Deserializes a protobuf message given in text proto form.

  Unknown fields are not allowed.

  Args:
    ctor: a message constructor function.
    body: a string with serialized message.

  Returns:
    Deserialized message constructed via `ctor`.
  """

def from_jsonpb(ctor, body):
  """Deserializes a protobuf message given as JBONPB string.

  Unknown fields are silently skipped.

  Args:
    ctor: a message constructor function.
    body: a string with serialized message.

  Returns:
    Deserialized message constructed via `ctor`.
  """

def struct_to_textpb(s):
  """Converts a struct to a text proto string.

  Args:
    s: a struct object. May not contain dicts.

  Returns:
    A str containing a text format protocol buffer message.
  """

func SetDefaultLoader Uses

func SetDefaultLoader(th *starlark.Thread, l *Loader)

SetDefaultLoader installs the given loader as default in the thread.

It can be obtained via DefaultLoader or proto.default_loader() from Starlark. Note that Starlark code has no way of changing the default loader. It's responsibility of the hosting environment to prepare the default loader (just like it prepares starlark.Thread itself).

func ToJSONPB Uses

func ToJSONPB(msg *Message) ([]byte, error)

ToJSONPB serializes a protobuf message to JSONPB string.

func ToTextPB Uses

func ToTextPB(msg *Message) ([]byte, error)

ToTextPB serializes a protobuf message to text proto.

type DescriptorSet Uses

type DescriptorSet struct {
    // contains filtered or unexported fields
}

DescriptorSet contains FileDescriptorProto of zero or more *.proto files, along with pointers to DescriptorSets with their imports.

A descriptor set can be registered in a Loader. Doing so transitively registers all its dependencies. See Loader.AddDescriptorSet for more details.

Implements starlark.Value and starlark.HasAttrs interfaces. Usage from Starlark side may look like this:

load(".../wellknown_descpb.star", "wellknown_descpb")
myprotos_descpb = proto.new_descriptor_set(
    name = "myprotos",
    deps = [wellknown_descpb],
    blob = io.read_file("myprotos.descpb"),
)
myprotos_descpb.register()

By default register() registers the descriptor set in the default loader, i.e. ds.register() is same as ds.register(loader=proto.default_loader()). Also note that ds.register(loader=l) is a sugar for l.add_descriptor_set(ds), so ds.register() is same as proto.default_loader().add_descriptor_set(ds), just much shorter.

func NewDescriptorSet Uses

func NewDescriptorSet(name string, fdps []*descriptorpb.FileDescriptorProto, deps []*DescriptorSet) (*DescriptorSet, error)

NewDescriptorSet evaluates given file descriptors and their dependencies and produces new DescriptorSet if there are no unresolved imports and no duplicated files.

'fdps' should be ordered topologically (i.e. if file A imports file B, then B should precede A in 'fdps' or be somewhere among 'deps'). This is always the case when generating sets via 'protoc --descriptor_set_out=...'.

Note that dependencies can only be specified when creating the descriptor set and can't be changed later. Cycles thus are impossible.

func (*DescriptorSet) Attr Uses

func (ds *DescriptorSet) Attr(name string) (starlark.Value, error)

Attr returns an attribute given its name (or nil if not present).

func (*DescriptorSet) AttrNames Uses

func (ds *DescriptorSet) AttrNames() []string

AtrrNames lists available attributes.

func (*DescriptorSet) Freeze Uses

func (ds *DescriptorSet) Freeze()

Freeze does nothing since DescriptorSet is already immutable.

func (*DescriptorSet) Hash Uses

func (ds *DescriptorSet) Hash() (uint32, error)

Hash returns unique value associated with this set.

func (*DescriptorSet) String Uses

func (ds *DescriptorSet) String() string

String returns str(...) representation of the set, for debug messages.

func (*DescriptorSet) Truth Uses

func (ds *DescriptorSet) Truth() starlark.Bool

Truth returns the truth value of an object.

func (*DescriptorSet) Type Uses

func (ds *DescriptorSet) Type() string

Type returns a short string describing the value's type.

type Loader Uses

type Loader struct {
    // contains filtered or unexported fields
}

Loader can instantiate Starlark values that correspond to proto messages.

Holds a pool of descriptors that describe all available proto types. Use AddDescriptorSet to seed it. Once seeded, use Module to get a Starlark module with symbols defined in some registered `*.proto` file.

Loader is also a Starlark value itself, with the following methods:

* add_descriptor_set(ds) - see AddDescriptorSet.
* module(path) - see Module.

Can be used concurrently. Non-freezable.

func DefaultLoader Uses

func DefaultLoader(th *starlark.Thread) *Loader

DefaultLoader returns a loader installed in the thread via SetDefaultLoader.

Returns nil if there's no default loader.

func NewLoader Uses

func NewLoader() *Loader

NewLoader instantiates a new loader with empty proto registry.

func (*Loader) AddDescriptorSet Uses

func (l *Loader) AddDescriptorSet(ds *DescriptorSet) error

AddDescriptorSet makes all *.proto files defined in the given descriptor set and all its dependencies available for use from Starlark.

AddDescriptorSet is idempotent in a sense that calling AddDescriptorSet(ds) multiple times with the exact same 'ds' is not an error. But trying to register a proto file through multiple different descriptor sets is an error.

func (*Loader) Attr Uses

func (l *Loader) Attr(name string) (starlark.Value, error)

Attr returns an attribute given its name (or nil if not present).

func (*Loader) AttrNames Uses

func (l *Loader) AttrNames() []string

AtrrNames lists available attributes.

func (*Loader) Freeze Uses

func (l *Loader) Freeze()

Freeze is noop for now.

func (*Loader) Hash Uses

func (l *Loader) Hash() (uint32, error)

Hash returns an integer assigned to this loader when it was created.

func (*Loader) MessageType Uses

func (l *Loader) MessageType(desc protoreflect.MessageDescriptor) *MessageType

MessageType creates new (or returns existing) MessageType.

The return value can be used to instantiate Starlark values via Message() or MessageFromProto(m).

func (*Loader) Module Uses

func (l *Loader) Module(path string) (*starlarkstruct.Module, error)

Module returns a module with top-level definitions from some *.proto file.

The descriptor of this proto file should be registered already via AddDescriptorSet. 'path' here is matched to what's in the descriptor, which is a path to *.proto EXACTLY as it was given to 'protoc'.

The name of the module matches the proto package name (per 'package ...' statement in the proto file).

func (*Loader) String Uses

func (l *Loader) String() string

String returns str(...) representation of the loader.

func (*Loader) Truth Uses

func (l *Loader) Truth() starlark.Bool

Truth returns True.

func (*Loader) Type Uses

func (l *Loader) Type() string

Type returns "proto.Loader".

type Message Uses

type Message struct {
    // contains filtered or unexported fields
}

Message is a Starlark value that implements a struct-like type structured like a protobuf message.

Implements starlark.Value, starlark.HasAttrs, starlark.HasSetField and starlark.Comparable interfaces.

Can be instantiated through Loader as loader.MessageType(...).Message() or loader.MessageType(...).MessageFromProto(p).

TODO(vadimsh): Currently not safe for a cross-goroutine use without external locking, even when frozen, due to lazy initialization of default fields on first access.

func FromJSONPB Uses

func FromJSONPB(typ *MessageType, blob []byte) (*Message, error)

FromJSONPB deserializes a protobuf message given as JBONPB string.

func FromTextPB Uses

func FromTextPB(typ *MessageType, blob []byte) (*Message, error)

FromTextPB deserializes a protobuf message given in text proto form.

func (*Message) Attr Uses

func (m *Message) Attr(name string) (starlark.Value, error)

Attr is called when a field is read from Starlark code.

func (*Message) AttrNames Uses

func (m *Message) AttrNames() []string

AttrNames lists available attributes.

func (*Message) CompareSameType Uses

func (m *Message) CompareSameType(op syntax.Token, y starlark.Value, depth int) (bool, error)

CompareSameType does 'm <op> y' comparison.

func (*Message) Freeze Uses

func (m *Message) Freeze()

Freeze makes this message immutable.

func (*Message) FromDict Uses

func (m *Message) FromDict(d starlark.IterableMapping) error

FromDict populates fields of this message based on values in an iterable mapping (usually a starlark.Dict).

Doesn't reset the message. Basically does this:

for k in d:
  setattr(msg, k, d[k])

Returns an error on type mismatch.

func (*Message) Hash Uses

func (m *Message) Hash() (uint32, error)

Hash returns an error, indicating proto messages are not hashable.

func (*Message) MessageType Uses

func (m *Message) MessageType() *MessageType

MessageType returns type information about this message.

func (*Message) SetField Uses

func (m *Message) SetField(name string, val starlark.Value) error

SetField is called when a field is assigned to from Starlark code.

func (*Message) String Uses

func (m *Message) String() string

String returns compact text serialization of this message.

func (*Message) ToProto Uses

func (m *Message) ToProto() proto.Message

ToProto returns a new populated proto message of an appropriate type.

func (*Message) Truth Uses

func (m *Message) Truth() starlark.Bool

Truth always returns True.

func (*Message) Type Uses

func (m *Message) Type() string

Type returns full proto message name.

type MessageType Uses

type MessageType struct {
    *starlark.Builtin // the callable, initialize in Loader
    // contains filtered or unexported fields
}

MessageType represents a proto message type and acts as its constructor: it is a Starlark callable that produces instances of Message.

Implements starlark.HasAttrs interface. Attributes represent constructors for nested messages and int values of enums. Note that starlark.HasSetField is not implemented, making values of MessageType immutable.

Given a MessageDescriptor can be instantiated through Loader as loader.MessageType(...).

func (*MessageType) Attr Uses

func (t *MessageType) Attr(name string) (starlark.Value, error)

Attr returns either a nested message or an enum value.

func (*MessageType) AttrNames Uses

func (t *MessageType) AttrNames() []string

AttrNames return names of all nested messages and enum values.

func (*MessageType) Converter Uses

func (t *MessageType) Converter() typed.Converter

Converter returns an object that can convert Starlark dicts and Nones to values of this message type.

Can be used by typed.List and typed.Dict.

func (*MessageType) Descriptor Uses

func (t *MessageType) Descriptor() protoreflect.MessageDescriptor

Descriptor returns protobuf type information for this message type.

func (*MessageType) Message Uses

func (t *MessageType) Message() *Message

Message instantiates a new empty message of this type.

func (*MessageType) MessageFromProto Uses

func (t *MessageType) MessageFromProto(p proto.Message) *Message

MessageFromProto instantiates a new message of this type and populates it based on values in the given proto.Message that should have a matching type.

Here "matching type" means p.ProtoReflect().Descriptor() *is* t.Descriptor(). Panics otherwise.

func (*MessageType) Type Uses

func (t *MessageType) Type() string

Type returns "proto.MessageType", it's the type of the message type itself.

It is sort of like a meta-type, i.e. like "type" type in Python.

Directories

PathSynopsis
testprotos

Package starlarkproto imports 20 packages (graph) and is imported by 2 packages. Updated 2019-10-16. Refresh now. Tools for package owners.