sharding

type Algorithm ¶

type Algorithm interface {
	Label() string
	Shardable(resource.Resource) bool
	ComputeHash(context.Context, resource.Resource) (uint64, bool)
}

type Decorator ¶

type Decorator struct {
	// contains filtered or unexported fields
}

func NewDecorator(algorithms []Algorithm, rings []*RingConfig) *Decorator

func (a *Decorator) AssignShards(ctx context.Context, resource resource.Resource)

type RingConfig ¶

type RingConfig struct {

	// Ring size. It affects possible shard numbers assigned to all relevant resources, which are in range: [0:size).
	Size uint64 `protobuf:"varint,1,opt,name=size,proto3" json:"size,omitempty" firestore:"size"`
	// Field path in the resource where shard number is stored, for example "meta.shards.byProjectIdMod4096". Expected
	// underlying type is int64.
	FieldPaths []*RingConfig_FieldPath `protobuf:"bytes,2,rep,name=field_paths,json=fieldPaths,proto3" json:"field_paths,omitempty" firestore:"fieldPaths"`
	// contains filtered or unexported fields
}

RingConfig is an information about single shard range for one specific purpose on a set of resources. Example:

{
  size: 4096
  field_paths: [
   { path: "meta.shards.byProjectIdMod4096", label: "byProjectId" }
  ]
}

In the example above, we indicate that some set of resources (in this case, those that have project ID specified) are shardable and their respective shard IDs are stored in the field path with label "byProjectId". Also, size of the ring describing this shard range is equal to 4096. Labels are useful when there is more than one field path for given ring. This can happen, if we have two resources which must be within same ring, but have different paths. Or another case, when one resource instance belongs to two different rings of the same type -> for example, IAM/Group resources are bucketed into rings by their identifiers and, if one group belongs to another, their retrospective GroupMember resources are visible in two rings. In other words, resources that form a graph and resources "on the borders" should be visible in both groups. Notes: * Path pointed by FieldPath must be of type int64. Also map<..., int64> is also supported (meta.shards) can be a map. RingConfig in the controller config must be in sync with at least one RingConfig in server side config. Server side config can use multiple rings at once. * While controller uses just one RingConfig at the time, server can use many simultaneously. It is required that labelling across array of RingConfigs is consistent. * Common components for sharding offer ShardDecorator. It is a generic component that decorates each relevant resource with shard ID. It handles common logic. While developer is still responsible for providing algorithm to it, calculation of the actual shard ID (combination of hash + ring size), iterating over all ring configs to provide number for all the rings is done there. This component however requires list of algorithms where each algorithm must tell, via Label, which field path intends to write.

func (*RingConfig) Descriptor() ([]byte, []int)

func (m *RingConfig) GetDefaultFieldPath() string

func (m *RingConfig) GetFieldPathForLabel(label string) string

func (m *RingConfig) GetFieldPaths() []*RingConfig_FieldPath

func (m *RingConfig) GetSize() uint64

func (*RingConfig) GotenMessage()

func (obj *RingConfig) GotenValidate() error

func (m *RingConfig) Marshal() ([]byte, error)

func (m *RingConfig) MarshalJSON() ([]byte, error)

func (*RingConfig) ProtoMessage()

func (m *RingConfig) ProtoReflect() preflect.Message

func (m *RingConfig) Reset()

func (m *RingConfig) SetFieldPaths(fv []*RingConfig_FieldPath)

func (m *RingConfig) SetSize(fv uint64)

func (m *RingConfig) String() string

func (m *RingConfig) Unmarshal(b []byte) error

func (m *RingConfig) UnmarshalJSON(data []byte) error

type RingConfig_FieldPath ¶

type RingConfig_FieldPath struct {

	// Path in the resource where shard number is stored.
	Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty" firestore:"path"`
	// Label identifying field path.
	Label string `protobuf:"bytes,2,opt,name=label,proto3" json:"label,omitempty" firestore:"label"`
	// contains filtered or unexported fields
}

func (*RingConfig_FieldPath) Descriptor() ([]byte, []int)

func (m *RingConfig_FieldPath) GetLabel() string

func (m *RingConfig_FieldPath) GetPath() string

func (*RingConfig_FieldPath) GotenMessage()

func (obj *RingConfig_FieldPath) GotenValidate() error

func (m *RingConfig_FieldPath) Marshal() ([]byte, error)

func (m *RingConfig_FieldPath) MarshalJSON() ([]byte, error)

func (*RingConfig_FieldPath) ProtoMessage()

func (m *RingConfig_FieldPath) ProtoReflect() preflect.Message

func (m *RingConfig_FieldPath) Reset()

func (m *RingConfig_FieldPath) SetLabel(fv string)

func (m *RingConfig_FieldPath) SetPath(fv string)

func (m *RingConfig_FieldPath) String() string

func (m *RingConfig_FieldPath) Unmarshal(b []byte) error

func (m *RingConfig_FieldPath) UnmarshalJSON(data []byte) error