pachyderm: github.com/pachyderm/pachyderm/src/server/pfs/s3 Index | Files

package s3

import "github.com/pachyderm/pachyderm/src/server/pfs/s3"

Index

Package Files

auth.go bucket.go driver.go error.go multipart.go object.go s3.go service.go

func Server Uses

func Server(port uint16, driver Driver, clientFactory ClientFactory) (*http.Server, error)

Server runs an HTTP server with an S3-like API for PFS. This allows you to use s3 clients to access PFS contents.

`inputBuckets` specifies which buckets should be served, referencing specific commit IDs. If nil, all PFS branches will be served as separate buckets, of the form `<branch name>.<bucket name>`. Some s3 features are enabled when all PFS branches are served as well; e.g. we add support for some s3 versioning functionality.

This returns an `http.Server` instance. It is the responsibility of the caller to start the returned server. It's possible for the caller to gracefully shutdown the server if desired; see the `http` package for details.

Note: server errors are redirected to logrus' standard log writer. The log writer is never closed. This should not be a problem with logrus' default configuration, which just writes to stdio. But if the standard logger is overwritten (e.g. to write to a socket), it's possible for this to cause problems.

Note: In `s3cmd`, you must set the access key and secret key, even though this API will ignore them - otherwise, you'll get an opaque config error: https://github.com/s3tools/s3cmd/issues/845#issuecomment-464885959

type Bucket Uses

type Bucket struct {
    // Repo is the PFS repo that this bucket points to
    Repo string
    // Commit is the PFS commit that this repo points to
    Commit string
    // Name is the name of the bucket
    Name string
}

Bucket represents an S3 bucket

type ClientFactory Uses

type ClientFactory = func() (*client.APIClient, error)

ClientFactory is a function called by s3g to create request-scoped pachyderm clients

type Driver Uses

type Driver interface {
    // contains filtered or unexported methods
}

Driver implementations drive the underlying bucket-related functionality for an s3gateway instance

type MasterDriver Uses

type MasterDriver struct{}

MasterDriver is the driver for the s3gateway instance running on pachd master

func NewMasterDriver Uses

func NewMasterDriver() *MasterDriver

NewMasterDriver constructs a new master driver

type WorkerDriver Uses

type WorkerDriver struct {
    // contains filtered or unexported fields
}

WorkerDriver is the driver for the s3gateway instance running on pachd workers

func NewWorkerDriver Uses

func NewWorkerDriver(inputBuckets []*Bucket, outputBucket *Bucket) *WorkerDriver

NewWorkerDriver creates a new worker driver. `inputBuckets` is a list of whitelisted buckets to be served from input repos. `outputBucket` is the whitelisted bucket to be served from an output repo. If `nil`, no output bucket will be available.

Package s3 imports 22 packages (graph) and is imported by 4 packages. Updated 2020-08-05. Refresh now. Tools for package owners.