praefect

package
v1.42.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 17, 2019 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package praefect provides data models and datastore persistence abstractions for tracking the state of repository replicas.

See original design discussion: https://gitlab.com/gitlab-org/gitaly/issues/1495

Package praefect is a Gitaly reverse proxy for transparently routing gRPC calls to a set of Gitaly services.

Index

Constants

View Source
const (
	// JobStatePending is the initial job state when it is not yet ready to run
	// and may indicate recovery from a failure prior to the ready-state
	JobStatePending = iota
	// JobStateReady indicates the job is now ready to proceed
	JobStateReady
	// JobStateInProgress indicates the job is being processed by a worker
	JobStateInProgress
	// JobStateComplete indicates the job is now complete
	JobStateComplete
	// JobStateCancelled indicates the job was cancelled. This can occur if the
	// job is no longer relevant (e.g. a node is moved out of a shard)
	JobStateCancelled
)

Variables

View Source
var ErrInvalidReplTarget = errors.New("target repository fails preconditions for replication")

ErrInvalidReplTarget indicates a target repository cannot be chosen because it fails preconditions for being replicatable

View Source
var ErrSecondariesMissing = errors.New("repository missing secondary replicas")

ErrSecondariesMissing indicates the repository does not have any backup replicas

Functions

This section is empty.

Types

type Coordinator

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator takes care of directing client requests to the appropriate downstream server. The coordinator is thread safe; concurrent calls to register nodes are safe.

func NewCoordinator added in v1.34.0

func NewCoordinator(l *logrus.Logger, storageLoc string, fileDescriptors ...*descriptor.FileDescriptorProto) *Coordinator

NewCoordinator returns a new Coordinator that utilizes the provided logger

func (*Coordinator) GetStorageNode added in v1.34.0

func (c *Coordinator) GetStorageNode(storage string) (Node, error)

GetStorageNode returns the registered node for the given storage location

func (*Coordinator) RegisterNode

func (c *Coordinator) RegisterNode(storageLoc, listenAddr string) error

RegisterNode will direct traffic to the supplied downstream connection when the storage location is encountered.

func (*Coordinator) RegisterProtos added in v1.36.0

func (c *Coordinator) RegisterProtos(protos ...*descriptor.FileDescriptorProto) error

RegisterProtos allows coordinator to register new protos on the fly

type Datastore added in v1.34.0

type Datastore interface {
	ReplJobsDatastore
	ReplicasDatastore
}

Datastore is a data persistence abstraction for all of Praefect's persistence needs

type JobState added in v1.36.0

type JobState int

JobState is an enum that indicates the state of a job

type MemoryDatastore added in v1.34.0

type MemoryDatastore struct {
	// contains filtered or unexported fields
}

MemoryDatastore is a simple datastore that isn't persisted to disk. It is only intended for early beta requirements and as a reference implementation for the eventual SQL implementation

func NewMemoryDatastore added in v1.34.0

func NewMemoryDatastore(cfg config.Config) *MemoryDatastore

NewMemoryDatastore returns an initialized in-memory datastore

func (*MemoryDatastore) CreateSecondaryReplJobs added in v1.36.0

func (md *MemoryDatastore) CreateSecondaryReplJobs(source Repository) ([]uint64, error)

CreateSecondaryReplJobs creates a replication job for each secondary that backs the specified repository. Upon success, the job IDs will be returned.

func (*MemoryDatastore) GetIncompleteJobs added in v1.36.0

func (md *MemoryDatastore) GetIncompleteJobs(storage string, count int) ([]ReplJob, error)

GetIncompleteJobs will return all incomplete replications jobs for the specified storage to the specified result limit.

func (*MemoryDatastore) GetSecondaries added in v1.34.0

func (md *MemoryDatastore) GetSecondaries(primary Repository) ([]string, error)

GetSecondaries will return the set of secondary storage locations for a given repository if they exist

func (*MemoryDatastore) SetSecondaries added in v1.34.0

func (md *MemoryDatastore) SetSecondaries(primary Repository, secondaries []string) error

SetSecondaries will replace the set of replicas for a repository

func (*MemoryDatastore) UpdateReplJob added in v1.36.0

func (md *MemoryDatastore) UpdateReplJob(jobID uint64, newState JobState) error

UpdateReplJob updates an existing replication job's state

type Node added in v1.34.0

type Node struct {
	Storage string
	// contains filtered or unexported fields
}

Node is a wrapper around the grpc client connection for a backend Gitaly node

type ReplJob added in v1.34.0

type ReplJob struct {
	ID     uint64     // autoincrement ID
	Target string     // which storage location to replicate to?
	Source Repository // source for replication
	State  JobState
}

ReplJob is an instance of a queued replication job. A replication job is meant for updating the repository so that it is synced with the primary copy. Scheduled indicates when a replication job should be performed.

type ReplJobsDatastore added in v1.34.0

type ReplJobsDatastore interface {
	// GetReplJobs fetches a list of chronologically ordered replication
	// jobs for the given storage replica. The returned list will be at most
	// count-length.
	GetIncompleteJobs(storage string, count int) ([]ReplJob, error)

	// CreateSecondaryJobs will create replication jobs for each secondary
	// replica of a repository known to the datastore. A set of replication job
	// ID's for the created jobs will be returned upon success.
	CreateSecondaryReplJobs(source Repository) ([]uint64, error)

	// UpdateReplJob updates the state of an existing replication job
	UpdateReplJob(jobID uint64, newState JobState) error
}

ReplJobsDatastore represents the behavior needed for fetching and updating replication jobs from the datastore

type ReplMgr added in v1.34.0

type ReplMgr struct {
	// contains filtered or unexported fields
}

ReplMgr is a replication manager for handling replication jobs

func NewReplMgr added in v1.34.0

func NewReplMgr(storage string, log *logrus.Logger, ds ReplJobsDatastore, c *Coordinator, opts ...ReplMgrOpt) ReplMgr

NewReplMgr initializes a replication manager with the provided dependencies and options

func (ReplMgr) ProcessBacklog added in v1.34.0

func (r ReplMgr) ProcessBacklog(ctx context.Context) error

ProcessBacklog will process queued jobs. It will block while processing jobs.

func (ReplMgr) ScheduleReplication added in v1.34.0

func (r ReplMgr) ScheduleReplication(ctx context.Context, repo Repository) error

ScheduleReplication will store a replication job in the datastore for later execution. It filters out projects that are not whitelisted. TODO: add a parameter to delay replication

type ReplMgrOpt added in v1.34.0

type ReplMgrOpt func(*ReplMgr)

ReplMgrOpt allows a replicator to be configured with additional options

func WithReplicator added in v1.34.0

func WithReplicator(r Replicator) ReplMgrOpt

WithReplicator overrides the default replicator

func WithWhitelist added in v1.34.0

func WithWhitelist(whitelistedRepos []string) ReplMgrOpt

WithWhitelist will configure a whitelist for repos to allow replication

type ReplicasDatastore added in v1.34.0

type ReplicasDatastore interface {
	// GetSecondaries will retrieve all secondary replica storage locations for
	// a primary replica
	GetSecondaries(primary Repository) ([]string, error)

	// SetSecondaries will set the secondary storage locations for a repository
	// in a primary replica.
	SetSecondaries(primary Repository, secondaries []string) error
}

ReplicasDatastore manages accessing and setting which secondary replicas backup a repository

type Replicator added in v1.34.0

type Replicator interface {
	Replicate(ctx context.Context, source Repository, target Node) error
}

Replicator performs the actual replication logic between two nodes

type Repository added in v1.34.0

type Repository struct {
	RelativePath string // relative path of repository
	Storage      string // storage location, e.g. default
}

Repository provides all necessary information to address a repository hosted in a specific Gitaly replica

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is a praefect server

func NewServer

func NewServer(c *Coordinator, repl ReplMgr, grpcOpts []grpc.ServerOption, l *logrus.Logger) *Server

NewServer returns an initialized praefect gPRC proxy server configured with the provided gRPC server options

func (*Server) Shutdown

func (srv *Server) Shutdown(ctx context.Context) error

Shutdown will attempt a graceful shutdown of the grpc server. If unable to gracefully shutdown within the context deadline, it will then forcefully shutdown the server and return a context cancellation error.

func (*Server) Start

func (srv *Server) Start(lis net.Listener) error

Start will start the praefect gRPC proxy server listening at the provided listener. Function will block until the server is stopped or an unrecoverable error occurs.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL