keysync

package module
v0.0.0-...-b8ca463 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2023 License: Apache-2.0 Imports: 34 Imported by: 0

README

Deprecated

As of 9/18/23 this project is now deprecated and no longer maintained; we recommend using HashiCorp Vault as a more robust and actively supported alternative.

Keysync

license report

Keysync is a production-ready program for accessing secrets in Keywhiz.

It is a replacement for the now-deprecated FUSE-based keywhiz-fs.

Getting Started

Building

Keysync must be built with Go 1.11+. You can build keysync from source:

$ git clone https://github.com/square/keysync
$ cd keysync
$ go build github.com/square/keysync/cmd/keysync

This will generate a binary called ./keysync

Dependencies

Keysync uses Go modules to manage dependencies. If you've cloned the repo into GOPATH, you should export GO111MODULE=on before running any go commands. All deps should be automatically fetched when using go build and go test. Add go mod tidy before committing.

Testing

Entire test suite:

go test ./...

Short, unit tests only:

go test -short ./...
Running locally

Keysync requires access to Keywhiz to work properly. Assuming you run Keywhiz locally on default port (4444), you can start keysync with:

./keysync --config keysync-config.yaml

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BackupFromConfig

func BackupFromConfig(cfg *Config) (backup.Backup, error)

func NewAPIServer

func NewAPIServer(syncer *Syncer, backup backup.Backup, port uint16, baseLogger *logrus.Entry, metrics *sqmetrics.SquareMetrics)

NewAPIServer is the constructor for an APIServer

Types

type APIServer

type APIServer struct {
	// contains filtered or unexported fields
}

APIServer holds state needed for responding to HTTP api requests

type BackupBundleClient

type BackupBundleClient struct {
	// contains filtered or unexported fields
}

BackupBundleClient is a secrets client that reads from a Keywhiz backup bundle.

func (BackupBundleClient) Logger

func (c BackupBundleClient) Logger() *logrus.Entry

Logger returns the logger for this client.

func (BackupBundleClient) RebuildClient

func (c BackupBundleClient) RebuildClient() error

RebuildClient for bundle clients is a no-op.

func (BackupBundleClient) Secret

func (c BackupBundleClient) Secret(name string) (secret *Secret, err error)

Secret returns secret with the given name from the bundle.

func (BackupBundleClient) SecretList

func (c BackupBundleClient) SecretList() (map[string]Secret, error)

SecretList returns all secrets in a bundle (unlike the real Keywhiz interface, it will return secrets' contents as well).

func (BackupBundleClient) SecretListWithContents

func (c BackupBundleClient) SecretListWithContents(secrets []string) (map[string]Secret, error)

SecretListWithContents returns the requested secrets from a bundle.

type Client

type Client interface {
	Secret(name string) (secret *Secret, err error)
	SecretList() (map[string]Secret, error)
	SecretListWithContents(secrets []string) (map[string]Secret, error)
	Logger() *logrus.Entry
	RebuildClient() error
}

Client represents an interface to a secrets storage backend.

func NewBackupBundleClient

func NewBackupBundleClient(path string, logger *logrus.Entry) (Client, error)

NewBackupBundleClient creates a new BackupBundleClient instance given a backup JSON file.

func NewClient

func NewClient(cfg *ClientConfig, caFile string, serverURL *url.URL, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (client Client, err error)

NewClient produces a ready-to-use client struct given client config and CA file with the list of trusted certificate authorities.

type ClientConfig

type ClientConfig struct {
	Key        string `yaml:"key"`       // Mandatory: Path to PEM key to use
	Cert       string `yaml:"cert"`      // Optional: PEM Certificate (If cert isn't in key file)
	User       string `yaml:"user"`      // Optional: User and Group are defaults for files without metadata
	DirName    string `yaml:"directory"` // Optional: What directory under SecretsDir this client is in. Defaults to the client name.
	Group      string `yaml:"group"`     // Optional: If unspecified, the global defaults are used.
	MaxRetries uint16
	Timeout    string
	MinBackoff string
	MaxBackoff string
}

The ClientConfig describes a single Keywhiz client. There are typically many of these per keysync instance.

type Config

type Config struct {
	ClientsDir    string            `yaml:"client_directory"`  // A directory of configuration files
	SecretsDir    string            `yaml:"secrets_directory"` // The directory secrets will be written to
	CaFile        string            `yaml:"ca_file"`           // The CA to trust (PEM) for Keywhiz communication
	YamlExt       string            `yaml:"yaml_ext"`          // The filename extension of the yaml config files
	PollInterval  string            `yaml:"poll_interval"`     // If specified, poll at the given interval, otherwise, exit after syncing
	ClientTimeout string            `yaml:"client_timeout"`    // If specified, timeout client connections after specified duration, otherwise use default.
	MinBackoff    string            `yaml:"min_backoff"`       // If specified, wait time before first retry, otherwise, use default.
	MaxBackoff    string            `yaml:"max_backoff"`       // If specified, max wait time before retries, otherwise, use default.
	MaxRetries    uint16            `yaml:"max_retries"`       // If specified, retry each HTTP call after non-200 response
	Server        string            `yaml:"server"`            // The server to connect to (host:port)
	Debug         bool              `yaml:"debug"`             // Enable debugging output
	DefaultUser   string            `yaml:"default_user"`      // Default user to own files
	DefaultGroup  string            `yaml:"default_group"`     // Default group to own files
	APIPort       uint16            `yaml:"api_port"`          // Port for API to listen on
	SentryDSN     string            `yaml:"sentry_dsn"`        // Sentry DSN
	SentryCaFile  string            `yaml:"sentry_ca_file"`    // The CA to trust (PEM) for Sentry communication
	FsType        output.Filesystem `yaml:"filesystem_type"`   // Enforce writing this type of filesystem. Use value from statfs.
	ChownFiles    bool              `yaml:"chown_files"`       // Do we chown files? Set to false when running without CAP_CHOWN.
	MetricsPrefix string            `yaml:"metrics_prefix"`    // Prefix metric names with this
	Monitor       MonitorConfig     `yaml:"monitor"`           // Config for monitoring/alerts
	BackupPath    string            `yaml:"backup_path"`       // If specified, back up secrets as an encrypted tarball to this location
	BackupKeyPath string            `yaml:"backup_key_path"`   // write wrapped key encrypting the backup to this location
	BackupPubkey  string            `yaml:"backup_pubkey"`     // Public key to wrap backup keys to, from keyunwrap --generate
}

Config is the main yaml configuration file passed to the keysync binary

func LoadConfig

func LoadConfig(configFile string) (*Config, error)

LoadConfig loads the "global" keysync configuration file. This would generally be called on startup.

func (*Config) LoadClients

func (config *Config) LoadClients() (map[string]ClientConfig, error)

LoadClients looks in directory for files with suffix, and tries to load them as Yaml files describing clients for Keysync to load We filter by the yaml extension so we can keep configs and keys in the same directory

type KeywhizHTTPClient

type KeywhizHTTPClient struct {
	// contains filtered or unexported fields
}

KeywhizHTTPClient is a client that reads from a Keywhiz server over HTTP (v2 API).

func (KeywhizHTTPClient) Logger

func (c KeywhizHTTPClient) Logger() *logrus.Entry

Logger returns the underlying logger for this client

func (KeywhizHTTPClient) RawSecret

func (c KeywhizHTTPClient) RawSecret(name string) ([]byte, error)

RawSecret returns raw JSON from requesting a secret.

func (KeywhizHTTPClient) RawSecretList

func (c KeywhizHTTPClient) RawSecretList() ([]byte, error)

RawSecretList returns raw JSON from requesting a listing of secrets.

func (KeywhizHTTPClient) RawSecretListWithContents

func (c KeywhizHTTPClient) RawSecretListWithContents(secrets []string) ([]byte, error)

RawSecretListWithContents returns raw JSON from requesting a listing of secrets with their contents.

func (*KeywhizHTTPClient) RebuildClient

func (c *KeywhizHTTPClient) RebuildClient() error

RebuildClient reloads certificates from disk. It should be called periodically to ensure up-to-date client certificates are used. This is important if you're using short-lived certificates that are routinely replaced.

func (KeywhizHTTPClient) Secret

func (c KeywhizHTTPClient) Secret(name string) (secret *Secret, err error)

Secret returns an unmarshalled Secret struct after requesting a secret.

func (KeywhizHTTPClient) SecretList

func (c KeywhizHTTPClient) SecretList() (map[string]Secret, error)

SecretList returns a map of unmarshalled Secret structs without their contents after requesting a listing of secrets. The map keys are the names of the secrets

func (KeywhizHTTPClient) SecretListWithContents

func (c KeywhizHTTPClient) SecretListWithContents(secrets []string) (map[string]Secret, error)

SecretList returns a map of unmarshalled Secret structs, including their contents, associated with the given list of secrets. The map keys are the names of the secrets. All secrets must be accessible to this client, or the entire request will fail.

func (KeywhizHTTPClient) ServerStatus

func (c KeywhizHTTPClient) ServerStatus() (data []byte, err error)

ServerStatus returns raw JSON from the server's _status endpoint

type MonitorConfig

type MonitorConfig struct {
	MinCertLifetime     time.Duration `yaml:"min_cert_lifetime"`     // If specified, warn if cert does not have given min lifetime.
	MinSecretsCount     int           `yaml:"min_secrets_count"`     // If specified, warn if client has less than minimum number of secrets
	AlertEmailServer    string        `yaml:"alert_email_server"`    // For alert emails: SMTP server host:port to use for sending email
	AlertEmailRecipient string        `yaml:"alert_email_recipient"` // For alert emails: Recipient of alert emails
	AlertEmailSender    string        `yaml:"alert_email_sender"`    // For alert emails: Sender (from) for alert emails
}

The MonitorConfig has extra settings for monitoring/alerts.

type Output

type Output interface {
	// Validate returns true if the secret is persisted already
	Validate(secret *Secret, state secretState) bool
	// Write a secret
	Write(secret *Secret) (*secretState, error)
	// Remove a secret
	Remove(name string) error
	// Remove all secrets and the containing directory (eg, when the client config is removed)
	// Returns a count of deleted files
	RemoveAll() (uint, error)
	// Cleanup unknown files (eg, ones deleted in Keywhiz while keysync was not running)
	// Returns a count of deleted files
	Cleanup(map[string]Secret) (uint, error)
}

Output is an interface that encapsulates what it means to store secrets

type OutputCollection

type OutputCollection interface {
	NewOutput(clientConfig ClientConfig, logger *logrus.Entry) (Output, error)
	// Cleanup unknown clients (eg, ones deleted while keysync was not running)
	// Returns a count of deleted clients
	Cleanup(map[string]struct{}, *logrus.Entry) (uint, []error)
}

OutputCollection handles a collection of outputs.

type OutputDir

type OutputDir struct {
	WriteDirectory    string
	DefaultOwnership  ownership.Ownership
	EnforceFilesystem output.Filesystem // What filesystem type do we expect to write to?
	ChownFiles        bool              // Do we chown the file? (Needs root or CAP_CHOWN).
	Logger            *logrus.Entry
}

OutputDir implements Output to files, which is the typical keysync usage to a tmpfs.

func (*OutputDir) Cleanup

func (out *OutputDir) Cleanup(secrets map[string]Secret) (uint, error)

func (*OutputDir) Remove

func (out *OutputDir) Remove(name string) error

func (*OutputDir) RemoveAll

func (out *OutputDir) RemoveAll() (uint, error)

func (*OutputDir) Validate

func (out *OutputDir) Validate(secret *Secret, state secretState) bool

Validate verifies the secret is written to disk with the correct content, permissions, and ownership

func (*OutputDir) Write

func (out *OutputDir) Write(secret *Secret) (*secretState, error)

Write puts a Secret into OutputDir

type OutputDirCollection

type OutputDirCollection struct {
	Config *Config
}

func (OutputDirCollection) Cleanup

func (c OutputDirCollection) Cleanup(known map[string]struct{}, logger *logrus.Entry) (uint, []error)

func (OutputDirCollection) NewOutput

func (c OutputDirCollection) NewOutput(clientConfig ClientConfig, logger *logrus.Entry) (Output, error)

type Secret

type Secret struct {
	Name             string
	Content          content   `json:"secret"`
	Length           uint64    `json:"secretLength"`
	Checksum         string    `json:"checksum"`
	CreatedAt        time.Time `json:"creationDate"`
	UpdatedAt        time.Time `json:"updateDate"`
	FilenameOverride *string   `json:"filename"`
	Mode             string
	Owner            string
	Group            string
}

Secret represents data returned after processing a server request.

json tags after fields indicate to json decoder the key name in JSON

func ParseSecret

func ParseSecret(data []byte) (s *Secret, err error)

ParseSecret deserializes raw JSON into a Secret struct.

func ParseSecretList

func ParseSecretList(data []byte) (secrets []Secret, err error)

ParseSecretList deserializes raw JSON into a list of Secret structs.

func (Secret) Filename

func (s Secret) Filename() (string, error)

Filename returns the expected filename of a secret. The filename metadata overrides the name, but it can't be path, so keysync can't delete or write arbitrary files outside its secrets directory.

func (Secret) ModeValue

func (s Secret) ModeValue() (os.FileMode, error)

ModeValue function helps by converting a textual mode to the expected value for fuse.

func (Secret) OwnershipValue

func (s Secret) OwnershipValue(fallback ownership.Ownership) (ret ownership.Ownership)

OwnershipValue returns the ownership for a given secret, falling back to the values given as an argument if they're not present in the secret

type SecretDeleted

type SecretDeleted struct{}

SecretDeleted is returned as an error when the server 404s.

func (SecretDeleted) Error

func (e SecretDeleted) Error() string

type StatusResponse

type StatusResponse struct {
	Ok      bool     `json:"ok"`
	Message string   `json:"message,omitempty"`
	Updated *Updated `json:"updated,omitempty"`
}

StatusResponse from API endpoints

type Syncer

type Syncer struct {
	// contains filtered or unexported fields
}

A Syncer manages a collection of clients, handling downloads and writing out updated secrets. Construct one using the NewSyncer and AddClient functions

func NewSyncer

func NewSyncer(config *Config, outputCollection OutputCollection, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (*Syncer, error)

NewSyncer instantiates the main stateful object in Keysync.

func NewSyncerFromFile

func NewSyncerFromFile(config *Config, clientConfig ClientConfig, bundle string, logger *logrus.Entry, metricsHandle *sqmetrics.SquareMetrics) (*Syncer, error)

NewSyncerFromFile instantiates a syncer that reads from a file/bundle instead of an HTTP server.

func (*Syncer) LoadClients

func (s *Syncer) LoadClients() (*pendingCleanup, error)

LoadClients gets configured clients, This function returns clients that have been deconfigured, which are expected to be cleaned up

func (*Syncer) Run

func (s *Syncer) Run() error

Run the main sync loop.

func (*Syncer) RunOnce

func (s *Syncer) RunOnce() (Updated, []error)

RunOnce runs the syncer once, for all clients, without sleeps.

type Updated

type Updated struct {
	Added   uint
	Changed uint
	Deleted uint
}

Updated secrets during a sync. How many secrets were added, changed, or deleted this sync.

func (*Updated) Add

func (u *Updated) Add(rhs Updated)

Add in another update count

func (*Updated) Total

func (u *Updated) Total() uint

Total of changed secrets

Directories

Path Synopsis
package backup handles reading and writing encrypted .tar files from the secretsDirectory to a backupPath using the key backupKey.
package backup handles reading and writing encrypted .tar files from the secretsDirectory to a backupPath using the key backupKey.
cmd
keyrestore
This is the main entry point for Keysync.
This is the main entry point for Keysync.
keysync
This is the main entry point for Keysync.
This is the main entry point for Keysync.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL