Documentation
¶
Overview ¶
Package partitioner implements utility routines for partitioning of a single YAML file.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// contains filtered or unexported fields
}
Config defines common configuration for yaml partitioning. Config must be immutable.
func NewConfig ¶
NewConfig creates a new common configuration for yaml partitioning.
Example:
var shardNames = []string{
"alpha",
"beta",
"gamma",
"delta",
"epsilon",
}
rndv, _ := hrw.New(xxhash.Sum64, shardNames...)
cfg, err := NewConfig(
WithConsistentHashing(rndv),
WithReplicasCount(1),
WithResultYamlIndent(2),
WithSplitPoint("groups.*.rules"),
WithThisShardID(-1),
) .
func (*Config) NodesCount ¶
NodesCount returns the number of nodes in the ConsistentHashing.
type ConsistentHashing ¶
type ConsistentHashing interface {
// NodeNames returns the list of node names in the ConsistentHashing
NodeNames() []string
// NodesCount returns the number of nodes in the ConsistentHashing
NodesCount() int
// Get gets the most suitable node name for a key
Get(key []byte) string
// GetN gets N most suitable node names for a key
GetN(key []byte, replicasCount int) map[string]struct{}
}
ConsistentHashing represent an abstract interface for consistent hashing.
type Option ¶
Option is a type for a function that accepts a pointer to an empty or minimal Config struct created in the Config constructor. Option represents Functional Options Pattern. See this article for details - https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis .
func WithConsistentHashing ¶
func WithConsistentHashing(h ConsistentHashing) Option
WithConsistentHashing sets the field with the object that implements ConsistentHashing interface REQUIRED .
func WithReplicasCount ¶
WithReplicasCount sets the number of replicas. Meaning how many shards will get the same data. This defaults to 1.
func WithResultYamlIndent ¶
WithResultYamlIndent sets the indentation used for Yaml encoding. This defaults to 2.
func WithSplitPoint ¶
WithSplitPoint sets the path to a yaml Node, which represents the so-called SplitPoint. YamlPartitioner dives into the YAML structure up to the given yaml Node and starts sharding items from there. Note: the SplitPoint yaml Node Kind must be either a SequenceNode (list) or a MappingNode (map). The SplitPoint must be in format "<key>", "<key>.*.<key>" REQUIRED .
func WithThisShardID ¶
WithThisShardID sets the shard id for which YamlPartitioner creates the resulting YAML(s). This defaults to -1, meaning that YamlPartitioner creates the resulting YAML for all shards.
func WithWorkingDirectory ¶
WithWorkingDirectory sets the directory where. This defaults to os.TempDir() .
type Partitioner ¶
type Partitioner struct {
// contains filtered or unexported fields
}
Partitioner represents the structure for partitioning a given input file.
func New ¶
func New(inputFile, commonPath string, opts ...Option) (*Partitioner, error)
New creates a new partitioner for the given inputFile. inputFile is an absolute path to a file that needs to be partitioned.
commonPath: Say inputFile is <commonPath>/<path-to-file.yml>, the result of partitioning will be written to <workDir>/<shardName>/<path-to-file.yml>.
func WithConfig ¶
func WithConfig(cfg *Config, inputFile, commonPath string) (*Partitioner, error)
WithConfig is being used to create a new partitioner with the previously created configuration.
func (*Partitioner) Report ¶
func (p *Partitioner) Report() string
Report returns a partitioning report.
func (*Partitioner) Reset ¶
func (p *Partitioner) Reset()
Reset sets the partitioner to its initial state.
func (*Partitioner) Run ¶
func (p *Partitioner) Run(ctx context.Context) error
Run performs the partitioning of a given input file accordingly to settings in the Partitioner Config.
func (*Partitioner) ShardItemsCount ¶
func (p *Partitioner) ShardItemsCount() map[string]int
ShardItemsCount returns how many items got each shard.
Source Files