pachyderm: github.com/pachyderm/pachyderm/src/server/pkg/serde Index | Files

package serde

import "github.com/pachyderm/pachyderm/src/server/pkg/serde"

Package serde contains Pachyderm-specific data structures for marshalling and unmarshalling Go structs and maps to structured text formats (currently just JSON and YAML).

Similar to https://github.com/ghodss/yaml, all implementations of the Format interface marshal and unmarshal data using the following pipeline:

Go struct/map (fully structured)
  <-> JSON document
  <-> map[string]interface{}
  <-> target format document

Despite the redundant round of marshalling and unmarshalling, there are two main advantages to this approach:

- YAML (and any future storage formats) can re-use existing `json:` struct
  tags
- The intermediate map[string]interface{} can be manipulated, making it
  possible to have flexible converstions between structs and documents. For
  examples, PPS pipelines may include a full TFJob spec, which is converted
  to a string and stored in the 'TFJob' field of Pachyderm's
  CreatePipelineRequest struct.

Index

Package Files

interface.go json_decoder.go json_encoder.go yaml_decoder.go yaml_encoder.go

func DecodeJSON Uses

func DecodeJSON(jsonData []byte, v interface{}) error

DecodeJSON is a convenience function that decodes json data using a JSONDecoder, but can be called inline

func DecodeYAML Uses

func DecodeYAML(yamlData []byte, v interface{}) error

DecodeYAML is a convenience function that decodes yaml data using a YAMLDecoder, but can be called inline

func EncodeJSON Uses

func EncodeJSON(v interface{}, options ...EncoderOption) ([]byte, error)

EncodeJSON is a convenience function that encodes json data using a JSONEncoder, but can be called inline

func EncodeYAML Uses

func EncodeYAML(v interface{}, options ...EncoderOption) ([]byte, error)

EncodeYAML is a convenience function that encodes yaml data using a YAMLEncoder, but can be called inline

func WithIndent Uses

func WithIndent(numSpaces int) func(d Encoder)

WithIndent is an EncoderOption that causes the returned encoder to use 'numSpaces' spaces as the indentation prefix for embedded messages. If applied to a JSON encoder, it also changes the encoder output to be multi-line (instead of inline).

func WithOrigName Uses

func WithOrigName(origName bool) func(d Encoder)

WithOrigName is an EncoderOption that, if set, encodes proto messages using the name set in the original .proto message definition, rather than the munged version of the generated struct's field name

type Decoder Uses

type Decoder interface {
    // Decode parses an underlying stream of text in the given format into 'v'
    // (a struct or Go map)
    Decode(interface{}) error

    // DecodeProto is similar to Decode, but instead of converting between
    // the canonicalized JSON and 'v' using 'encoding/json', it does so using
    // 'gogo/protobuf/jsonpb'.  This allows callers to take advantage of more
    // sophisticated timestamp parsing and such in the 'jsonpb' library.
    //
    // TODO(msteffen) same comment re: proto.Message as for
    // Encoder.EncodeProto()
    DecodeProto(proto.Message) error

    // DecodeTransform is similar to Decode, but users can manipulate the
    // intermediate map[string]interface generated by Format implementations by
    // passing a function. Note that 'Encode(v)' is equivalent to
    // 'EncodeTransform(v, nil)'
    DecodeTransform(interface{}, func(map[string]interface{}) error) error

    // DecodeProtoTransform is similar to DecodeTransform, but instead of
    // converting between the canonicalized JSON and 'v' using 'encoding/json', it
    // does so using 'gogo/protobuf/jsonpb'.  This allows callers to take
    // advantage of more sophisticated timestamp parsing and such in the 'jsonpb'
    // library.
    //
    // TODO(msteffen) same comment re: proto.Message as for
    // Encoder.EncodeProto()
    DecodeProtoTransform(proto.Message, func(map[string]interface{}) error) error
}

Decoder is the interface for decoding a particular serialization format. Currently, the only implementation of Decoder is in serde/yaml/decoder.go (the point of having a common interface is to make it possible to switch between decoding different serialization formats at runtime, but we have no need to switch between a YAML decoder and a JSON decoder, as our YAML decoder can decode JSON).

type Encoder Uses

type Encoder interface {
    // Marshall converts 'v' (a struct or Go map) to a structured-text document
    // and writes it to this Encoder's output stream
    Encode(v interface{}) error

    // EncodeProto is similar to Encode, but instead of converting between the
    // canonicalized JSON and 'v' using 'encoding/json', it does so using
    // 'gogo/protobuf/jsonpb'.  This allows callers to take advantage of more
    // sophisticated timestamp parsing and such in the 'jsonpb' library.
    //
    // TODO(msteffen): We can *almost* avoid the Encode/EncodeProto split by
    // checking if 'v' implements 'proto.Message', except for one case: the
    // kubernetes client library includes structs that are pseudo-protobufs.
    // Structs in the kubernetes go client implement the 'proto.Message()'
    // interface but are hand-generated and contain embedded structs, which
    // 'jsonpb' can't handle when parsing. If jsonpb is ever extended to be able
    // to parse JSON into embedded structs (even though the protobuf compiler
    // itself never generates such structs) then we could merge this into
    // Encode() and rely on:
    //
    //   if msg, ok := v.(proto.Message); ok {
    //     ... use jsonpb ...
    //   } else {
    //     ... use encoding/json ...
    //   }
    EncodeProto(proto.Message) error

    // EncodeTransform is similar to Encode, but users can manipulate the
    // intermediate map[string]interface generated by Format implementations by
    // passing a function. Note that 'Encode(v)' is equivalent to
    // 'EncodeTransform(v, nil)'
    EncodeTransform(interface{}, func(map[string]interface{}) error) error

    // EncodeProtoTransform is similar to EncodeTransform(), but instead of
    // converting between the canonicalized JSON and 'v' using 'encoding/json', it
    // does so using 'gogo/protobuf/jsonpb'.  This allows callers to take
    // advantage of more sophisticated timestamp parsing and such in the 'jsonpb'
    // library.
    //
    // TODO(msteffen) same comment re: proto.Message as for EncodeProto()
    EncodeProtoTransform(proto.Message, func(map[string]interface{}) error) error
}

Encoder is an interface for encoding data to an output stream (every implementation should provide the ability to construct an Encoder tied to an output stream, to which encoded text should be written)

func GetEncoder Uses

func GetEncoder(encoding string, w io.Writer, opts ...EncoderOption) (Encoder, error)

GetEncoder dynamically creates and returns an Encoder for the text format 'encoding' (currently, 'encoding' must be "yaml" or "json"). 'defaultEncoding' specifies the text format that should be used if 'encoding' is "". 'opts' are the list of options that should be applied to any result, if any are applicable. Typically EncoderOptions are encoder-specific (e.g. setting json indentation). If an option is passed to GetEncoder for e.g. a json encoder but a yaml encoder is requested, then the option will be ignored. This makes it possible to pass all options to GetEncoder, but defer the decision about what kind of encoding to use until runtime, like so:

enc, _ := GetEncoder(outputFlag, os.Stdout,

...options to use if json...,
...options to use if yaml...,

) enc.Encode(obj)

Note: There is currently no corresponding GetDecoder, because the only implementation of the Decoder interface is YAMLDecoder

type EncoderOption Uses

type EncoderOption func(Encoder)

EncoderOption modifies the behavior of new encoders and can be passed to GetEncoder.

type JSONDecoder Uses

type JSONDecoder struct {
    // contains filtered or unexported fields
}

JSONDecoder is an implementation of serde.Decoder that operates on JSON data.

func NewJSONDecoder Uses

func NewJSONDecoder(r io.Reader) *JSONDecoder

NewJSONDecoder returns a new JSONDecoder that reads from 'r'

func (*JSONDecoder) Decode Uses

func (d *JSONDecoder) Decode(v interface{}) error

Decode implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeProto Uses

func (d *JSONDecoder) DecodeProto(v proto.Message) error

DecodeProto implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeProtoTransform Uses

func (d *JSONDecoder) DecodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

DecodeProtoTransform implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeTransform Uses

func (d *JSONDecoder) DecodeTransform(v interface{}, f func(map[string]interface{}) error) error

DecodeTransform implements the corresponding method of serde.Decoder

type JSONEncoder Uses

type JSONEncoder struct {
    // contains filtered or unexported fields
}

JSONEncoder is an implementation of serde.Encoder that operates on JSON data

func NewJSONEncoder Uses

func NewJSONEncoder(w io.Writer, options ...EncoderOption) *JSONEncoder

NewJSONEncoder returns a new JSONEncoder that writes to 'w'

func (*JSONEncoder) Encode Uses

func (e *JSONEncoder) Encode(v interface{}) error

Encode implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeProto Uses

func (e *JSONEncoder) EncodeProto(v proto.Message) error

EncodeProto implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeProtoTransform Uses

func (e *JSONEncoder) EncodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

EncodeProtoTransform implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeTransform Uses

func (e *JSONEncoder) EncodeTransform(v interface{}, f func(map[string]interface{}) error) error

EncodeTransform implements the corresponding method of serde.Encoder

type YAMLDecoder Uses

type YAMLDecoder struct {
    // contains filtered or unexported fields
}

YAMLDecoder is an implementation of serde.Decoder that operates on yaml data

func NewYAMLDecoder Uses

func NewYAMLDecoder(r io.Reader) *YAMLDecoder

NewYAMLDecoder returns a new YAMLDecoder that reads from 'r'

func (*YAMLDecoder) Decode Uses

func (d *YAMLDecoder) Decode(v interface{}) error

Decode implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeProto Uses

func (d *YAMLDecoder) DecodeProto(v proto.Message) error

DecodeProto implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeProtoTransform Uses

func (d *YAMLDecoder) DecodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

DecodeProtoTransform implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeTransform Uses

func (d *YAMLDecoder) DecodeTransform(v interface{}, f func(map[string]interface{}) error) error

DecodeTransform implements the corresponding method of serde.Decoder

type YAMLEncoder Uses

type YAMLEncoder struct {
    // contains filtered or unexported fields
}

YAMLEncoder is an implementation of serde.Encoder that operates on YAML data

func NewYAMLEncoder Uses

func NewYAMLEncoder(w io.Writer, options ...EncoderOption) *YAMLEncoder

NewYAMLEncoder returns a new YAMLEncoder that writes to 'w'

func (*YAMLEncoder) Encode Uses

func (e *YAMLEncoder) Encode(v interface{}) error

Encode implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeProto Uses

func (e *YAMLEncoder) EncodeProto(v proto.Message) error

EncodeProto implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeProtoTransform Uses

func (e *YAMLEncoder) EncodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

EncodeProtoTransform implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeTransform Uses

func (e *YAMLEncoder) EncodeTransform(v interface{}, f func(map[string]interface{}) error) error

EncodeTransform implements the corresponding method of serde.Encoder

Package serde imports 8 packages (graph) and is imported by 6 packages. Updated 2020-03-30. Refresh now. Tools for package owners.