serde

Overview ¶

Package serde contains Pachyderm-specific data structures for marshalling and unmarshalling Go structs and maps to structured text formats (currently just JSON and YAML).

Similar to https://github.com/ghodss/yaml, all implementations of the Format interface marshal and unmarshal data using the following pipeline:

Go struct/map (fully structured)
  <-> JSON document
  <-> map[string]interface{}
  <-> target format document

Despite the redundant round of marshalling and unmarshalling, there are two main advantages to this approach:

• YAML (and any future storage formats) can re-use existing `json:` struct tags
• The intermediate map[string]interface{} can be manipulated, making it possible to have flexible converstions between structs and documents. For examples, PPS pipelines may include a full TFJob spec, which is converted to a string and stored in the 'TFJob' field of Pachyderm's CreatePipelineRequest struct.

json_encoder.go is similar to yaml_encoder.go; see its comment on general approach. Because the final output is json, the steps this uses are a little more redundant, but still the reverse of the steps in decoder.go (allowing for future-proof reversability):

1. Serialize structs to JSON using encoding/json or, in the case of protobufs, using Google's 'protojson' library.
2. Deserialize that text into a generic interface{}.
3. Serialize that interface using encoding/json.

Because the data is serialized to JSON twice, this approach seems more redundant in the context of json_encoder.go, but it's symmetrical with the encoding process. TODO(msteffen) src/server/identity/cmds/cmds.go uses this to serialize Pachyderm identity configs. That code could be written to modify the holder (map[string]interface{}) instead of using connectorConfig as an intermediate struct, which must be kept in sync.

yaml_encoder.go is the reverse of decoder.go. It applies the same steps in the opposite order:

1. Serialize structs to JSON using encoding/json or, in the case of protobufs, using Google's 'protojson' library
2. Deserialize that text into a generic interface{}
3. Serialize that interface using gopkg.in/yaml.v3

This allows for correct handling of timestamps and such. It also allows data that has already been deserialized into a generic interface{} to be written out. Finally, it allows for transformations to be applied to the interface{} in a structured way (e.g. if the struct being serialized in step 1 contains embedded json).