README
¶
Unstructured - for when you can't fit your data into a struct
Why?
Go is awesome at [de-]serialising structured data. We can do things like:
type MyType struct {
Key1 string `yaml:"key1"`
Key2 int `yaml:"key2"`
}
...
myYaml = `
key1: "Alice"
key2: 65537
`
var myVar MyType
err := yaml.Unmarshal([]byte(myYaml), &myVar)
This is fantastic so long as:
- You know the exact structure of your data at compile time.
- That structure can be mapped to a go struct type.
For an example of point (2) failing, consider the following yaml:
top-level-list:
- "this first element is a string - perhaps containing metadata"
- name: first real element
type: element-type-1
payload: [1,2,3,4]
- name: second real element
type: element-type-2
payload:
some: embedded structure
I do not know of a way to define a struct type in go which will accept this YAML when I attempt to deserialize it. If we're designing our systems end-to-end in go, we will almost certainly avoid producing YAML structures like this. However, sometimes we may have to interface with other systems written perhaps in more dynamically typed languages, where this sort of thing is more natural.
For an example of point (1) failing, consider the schema of a bosh
manifest. The type of the properties
block of an instance group is described as a "hash". In practice, these hashes
can be quite complex structures, depending on how configurable the jobs in that
particular instance group are. Unfortunately, the schema for each individual
properties block is defined by the authors of the jobs in the instance group.
If we are writing a tool in go that manages bosh manifests in general, then we
have no way to be more specific than the type map[string]interface{}.
What?
When working with data which cannot be described at compile time in the go type language, we have no choice but to either leave go and work in some other language, or to work without the safety net of our type system. This library attempts to make managing unstructured data in an untyped way less unpleasant than it might otherwise be.
How?
This library leans on the excellent gojsonpointer library to allow us to address deep into JSON and YAML structures and:
- retrieve data -- if it exists
- write data -- if the parent we're writing into exists
This allows us to handle the data above with code something like this:
myData, err := unstructured.ParseYAML(myYaml)
myPayloadData, err := myData.GetByPointer("/top-level-list/2/payload/some")
myPayloadString, err := myPayloadData.StringValue()
fmt.Println(myPayloadString)
myPayloadMap, err := myData.GetByPointer("/top-level-list/2/payload")
err := myPayloadMap.SetField("additional-key", []string{"some", "arbitrary", "data"})
Of course, in the real version, all the error values are checked.
We also provide a number of gomega matchers in case you want to inspect semi-structured data in your tests. You can see these used here.
Gotchas
Since we're deliberately working around go's type system, we have to perform a
lot of type-like checks ourselves, at runtime. For most of the methods in the
library, this means we might return an error, which you should check. However,
sometimes it's useful to be able to chain our methods, and this is only
possible if those methods return singletons. To facilitate this, we have some
"Unsafe" methods, which may panic if their preconditions are not met. For
example, d.F("parent-field").F("sub-field") will panic unless d is a map
with key "parent-field" which contains a map with key "sub-field". You can
check these things with methods like IsOb and HasKey, but you should
probably just use GetByPointer instead. The unsafe accessors are only really
useful for writing terse chained statements in tests, when you're almost
entirely certain that the path exists, and a panic would be a correctly failing
test anyway.
You can see examples of the usage of these unsafe accessors in the "alternative formulations" of the tests in this example.
Documentation
¶
Overview ¶
Package unstructured provides ways of manipulating unstructured data such as JSON or YAML
Index ¶
- Constants
- type Data
- func (j Data) BoolValue() (bool, error)
- func (j Data) F(key string) Data
- func (j Data) FindElem(match ElementMatcher) (Data, bool)
- func (j Data) GetByPointer(p string) (data Data, err error)
- func (j Data) HasKey(key string) bool
- func (j Data) HasPointer(p string) (bool, error)
- func (j Data) IsBool() bool
- func (j Data) IsList() bool
- func (j Data) IsNull() bool
- func (j Data) IsNum() bool
- func (j Data) IsOb() bool
- func (j Data) IsOfType(typ string) bool
- func (j Data) IsString() bool
- func (j Data) Keys() ([]string, error)
- func (j Data) ListValue() ([]Data, error)
- func (j Data) NumValue() (float64, error)
- func (j Data) ObValue() (map[string]interface{}, error)
- func (j Data) RawValue() interface{}
- func (j Data) SetElem(index int, value interface{}) error
- func (j Data) SetField(fieldName string, val interface{}) error
- func (j Data) StringValue() (string, error)
- func (j Data) UnsafeBoolValue() bool
- func (j Data) UnsafeGetField(key string) Data
- func (j Data) UnsafeListValue() (list []Data)
- func (j Data) UnsafeNumValue() float64
- func (j Data) UnsafeObValue() map[string]interface{}
- func (j Data) UnsafeStringValue() string
- type ElementMatcher
Constants ¶
const ( DataString = "string" DataNum = "number" DataOb = "object" DataList = "list" DataNull = "null" DataBool = "bool" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Data ¶
type Data struct {
// contains filtered or unexported fields
}
Data represents some unstructured data
func ParseJSON ¶
ParseJSON unmarshals json from an input string. Use this for generating a Data struct, whose contents you can examine using the following functions.
func ParseYAML ¶
ParseYAML unmarshals yaml from an input string. Use this for generating a Data struct, whose contents you can examine using the following functions.
func (Data) BoolValue ¶
BoolValue returns the golang bool representation of the bool represented by this Data struct. If the Data struct does not represent a bool, this method returns an error.
func (Data) FindElem ¶
func (j Data) FindElem(match ElementMatcher) (Data, bool)
FindElem finds an element in a list, using a provided matcher
func (Data) GetByPointer ¶
GetByPointer returns a Data struct containing the contents of the original data at the given pointer address `p`. For more information on json pointers, see https://tools.ietf.org/html/rfc6901
func (Data) HasKey ¶
HasKey returns true iff the object represented by this Data struct contains `key`
Note: this will panic if the data represented by this Data struct is not an object. If in doubt, check with `IsOb()`
func (Data) HasPointer ¶
HasPointer returns true iff the object represented by this Data struct contains the pointer `p`
For more information on json pointers, see https://tools.ietf.org/html/rfc6901
func (Data) IsOb ¶
IsOb returns true iff the data represented by this Data struct is an object or map.
func (Data) IsOfType ¶
IsOfType returns true iff the Data struct represents data of type `typ`. Valid values of `typ` are listed as constants above.
func (Data) IsString ¶
IsString returns true iff the data represented by this Data struct is a string.
func (Data) Keys ¶
Keys returns a list of the keys on this Data object.
If this is not a Data object, return an error
func (Data) ListValue ¶
ListValue returns a golang slice of Data structs representing the unstructured list represented by this Data struct. If the Data struct does not represent a list, this method returns an error.
func (Data) NumValue ¶
NumValue returns the golang float64 representation of the number represented by this Data struct. If the Data struct does not represent a number, this method returns an error.
func (Data) ObValue ¶
ObValue returns a golang map[string]interface{} represenation of the object represented by this Data struct. If the Data struct does not represent an object, this method returns an error
func (Data) RawValue ¶
func (j Data) RawValue() interface{}
RawValue returns the raw go value of the parsed data, without any type checking
func (Data) SetElem ¶
SetElem sets the element at a given index in this Data list to the given value. If this Data object does not represent a list, return an error
func (Data) SetField ¶
SetField updates the field `fieldName` of this Data object. If the field `fieldName` does not exist on this object, create it.
If this Data does not represent an object, return an error.
func (Data) StringValue ¶
StringValue returns the golang string representation of the string represented by this Data struct. If the Data struct does not represent a string, this method returns an error.
func (Data) UnsafeBoolValue ¶
UnsafeBoolValue returns the golang bool representation of the bool represented by this Data struct. If the Data struct does not represent a bool, this method panics. If in doubt, check with `IsBool()`
func (Data) UnsafeGetField ¶
UnsafeGetField returns a Data struct containing the contents of the original data at the given `key`. If this method name feels too long, use `F(key)`.
Note: this function panics if the given `key` does not exist. If in doubt, check with `HasKey()`.
func (Data) UnsafeListValue ¶
UnsafeListValue returns a golang slice of Data structs representing the unstructured list represented by this Data struct. If the Data struct does not represent a list, this method panics. If in doubt, check with `IsList()`
func (Data) UnsafeNumValue ¶
UnsafeNumValue returns the golang float64 representation of the number represented by this Data struct. If the Data struct does not represent a number, this method panics. If in doubt, check with `IsNum()`
func (Data) UnsafeObValue ¶
UnsafeObValue returns a golang map[string]interface{} represenation of the object represented by this Data struct. If the Data struct does not represent an object, this method panics. If in doubt, check with `IsOb()`
func (Data) UnsafeStringValue ¶
UnsafeStringValue returns the golang string representation of the string represented by this Data struct. If the Data struct does not represent a string, this method panics. If in doubt, check with `IsString()`
type ElementMatcher ¶
An ElementMatcher can be used with FindElem to find an element in an unstructured list.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package gunstructured provides gomega matchers for using unstructured with the ginkgo and gomega testing libraries:
|
Package gunstructured provides gomega matchers for using unstructured with the ginkgo and gomega testing libraries: |