client

package
v2.9.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 8, 2024 License: Apache-2.0 Imports: 62 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxListItemsLog specifies the maximum number of items we log in response to a List* API
	MaxListItemsLog = 10
	// StorageSecretName is the name of the Kubernetes secret in which storage credentials are stored.
	StorageSecretName = "pachyderm-storage-secret"
	// PostgresSecretName is the name of the secret containing the postgres password
	// It must match the secret passed to pachd here: etc/helm/pachyderm/templates/pachd/deployment.yaml
	PostgresSecretName = "postgres"
	// PachctlSecretName is the name of the Kubernetes secret in which
	// pachctl credentials are stored.
	PachctlSecretName = "pachyderm-pachctl-secret"
)
View Source 
const (
	// PPSEtcdPrefixEnv is the environment variable that specifies the etcd
	// prefix that PPS uses.
	PPSEtcdPrefixEnv = "PPS_ETCD_PREFIX"
	// PPSWorkerIPEnv is the environment variable that a worker can use to
	// see its own IP.  The IP address is made available through the
	// Kubernetes downward API.
	PPSWorkerIPEnv = "PPS_WORKER_IP"
	// PPSPodNameEnv is the environment variable that a pod can use to
	// see its own name.  The pod name is made available through the
	// Kubernetes downward API.
	PPSPodNameEnv = "PPS_POD_NAME"
	// PPSProjectNameEnv is the env var that sets the name of the project
	// that the workers are running.
	PPSProjectNameEnv = "PPS_PROJECT_NAME"
	// PPSPipelineNameEnv is the env var that sets the name of the pipeline
	// that the workers are running.
	PPSPipelineNameEnv = "PPS_PIPELINE_NAME"
	// PPSJobIDEnv is the env var that sets the ID of the job that the
	// workers are running (if the workers belong to an orphan job, rather than a
	// pipeline).
	PPSJobIDEnv = "PPS_JOB_ID"
	// PPSSpecCommitEnv is the namespace in which pachyderm is deployed
	PPSSpecCommitEnv = "PPS_SPEC_COMMIT"
	// PPSInputPrefix is the prefix of the path where datums are downloaded
	// to.  A datum of an input named `XXX` is downloaded to `/pfs/XXX/`.
	PPSInputPrefix = "/pfs"
	// PPSScratchSpace is where pps workers store data while it's waiting to be
	// processed.
	PPSScratchSpace = ".scratch"
	// PPSWorkerPortEnv is environment variable name for the port that workers
	// use for their gRPC server
	PPSWorkerPortEnv = "PPS_WORKER_GRPC_PORT"
	// PPSWorkerVolume is the name of the volume in which workers store
	// data.
	PPSWorkerVolume = "pachyderm-worker"
	// PPSWorkerUserContainerName is the name of the container that runs
	// the user code to process data.
	PPSWorkerUserContainerName = "user"
	// PPSWorkerSidecarContainerName is the name of the sidecar container
	// that runs alongside of each worker container.
	PPSWorkerSidecarContainerName = "storage"
	// GCGenerationKey is the etcd key that stores a counter that the
	// GC utility increments when it runs, so as to invalidate all cache.
	GCGenerationKey = "gc-generation"
	// JobIDEnv is an env var that is added to the environment of user pipeline
	// code and indicates the id of the job currently being run.
	JobIDEnv = "PACH_JOB_ID"
	// OutputCommitIDEnv is an env var that is added to the environment of user
	// pipelined code and indicates the id of the output commit.
	OutputCommitIDEnv = "PACH_OUTPUT_COMMIT_ID"
	// DatumIDEnv is an env var that is added to the environment of user
	// pipelined code and indicates the id of the datum.
	DatumIDEnv = "PACH_DATUM_ID"
	// PeerPortEnv is the env var that sets a custom peer port
	PeerPortEnv = "PEER_PORT"

	ReprocessSpecUntilSuccess = "until_success"
	ReprocessSpecEveryJob     = "every_job"
)
View Source 
const DefaultDialTimeout = 30 * time.Second

DefaultDialTimeout is the max amount of time APIClient.connect() will wait for a connection to be established unless overridden by WithDialTimeout()

View Source 
const DefaultMaxConcurrentStreams = 100

DefaultMaxConcurrentStreams defines the max number of Putfiles or Getfiles happening simultaneously

View Source 
const DefaultTTL = 10 * time.Minute

DefaultTTL is the default time-to-live for a temporary fileset.

View Source 
const FileSetsRepoName = "__filesets__"

FileSetsRepoName is the repo name used to access filesets as virtual commits.

Variables

This section is empty.

Functions

func AddFileSet added in v2.8.0 

func AddFileSet(ctx context.Context, c pfs.APIClient, project, repo, branch, commit, ID string) (retErr error)

AddFileSet adds a fileset to a commit in a project.

func ComposeFileSet added in v2.8.0 

func ComposeFileSet(ctx context.Context, c pfs.APIClient, IDs []string, ttl time.Duration) (_ string, retErr error)

func CopyFile added in v2.8.0 

func CopyFile(ctx context.Context, c pfs.APIClient, dstCommit *pfs.Commit, dstPath string, srcCommit *pfs.Commit, srcPath string, opts ...CopyFileOption) error

func DefaultDialOptions 

func DefaultDialOptions() []grpc.DialOption

DefaultDialOptions is a helper returning a slice of grpc.Dial options such that grpc.Dial() is synchronous: the call doesn't return until the connection has been established and it's safe to send RPCs

func DeleteFile added in v2.8.0 

func DeleteFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string, opts ...DeleteFileOption) error

func DiffFile added in v2.8.0 

func DiffFile(ctx context.Context, c pfs.APIClient, newCommit *pfs.Commit, newPath string, oldCommit *pfs.Commit, oldPath string, shallow bool, cb func(*pfs.FileInfo, *pfs.FileInfo) error) (retErr error)

func DiffFileAll added in v2.8.0 

func DiffFileAll(ctx context.Context, c pfs.APIClient, newCommit *pfs.Commit, newPath string, oldCommit *pfs.Commit, oldPath string, shallow bool) (_ []*pfs.FileInfo, _ []*pfs.FileInfo, retErr error)

func FinishCommit added in v2.8.0 

func FinishCommit(ctx context.Context, c pfs.APIClient, projectName, repoName, branchName, commitID string) (retErr error)

func GetDatumTotalTime 

func GetDatumTotalTime(s *pps.ProcessStats) time.Duration

GetDatumTotalTime sums the timing stats from a DatumInfo

func GetFile added in v2.8.0 

func GetFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string, w io.Writer, opts ...GetFileOption) (retErr error)

func GetFileReadSeeker added in v2.8.0 

func GetFileReadSeeker(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string) (io.ReadSeeker, error)

func GetFileReader added in v2.8.0 

func GetFileReader(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string) (io.Reader, error)

func GetFileSet added in v2.8.0 

func GetFileSet(ctx context.Context, c pfs.APIClient, project, repo, branch, commit string) (_ string, retErr error)

func GetFileTAR added in v2.8.0 

func GetFileTAR(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string) (io.ReadCloser, error)

func GetFileURL added in v2.8.0 

func GetFileURL(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path, URL string) (retErr error)

func GetTransaction 

func GetTransaction(ctx context.Context) (*transaction.Transaction, error)

GetTransaction (should be run from the server-side) loads the active transaction from the grpc metadata and returns the associated transaction object - or `nil` if no transaction is set.

func GetUserMachineAddr 

func GetUserMachineAddr(context *config.Context) (addr *grpcutil.PachdAddress, err error)

func GlobFile added in v2.8.0 

func GlobFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, pattern string, cb func(fi *pfs.FileInfo) error) (retErr error)

func GlobFileAll added in v2.8.0 

func GlobFileAll(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, pattern string) (_ []*pfs.FileInfo, retErr error)

func InspectCommitSet added in v2.8.0 

func InspectCommitSet(ctx context.Context, c pfs.APIClient, id string) (_ []*pfs.CommitInfo, retErr error)

func InspectFile added in v2.8.0 

func InspectFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string) (_ *pfs.FileInfo, retErr error)

func ListFile added in v2.8.0 

func ListFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string, cb func(fi *pfs.FileInfo) error) (retErr error)

func ListFileAll added in v2.8.0 

func ListFileAll(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string) (_ []*pfs.FileInfo, retErr error)

func NewBranch 

func NewBranch(projectName, repoName, branchName string) *pfs.Branch

NewBranch creates a pfs.Branch in the given project & repo.

func NewCommit 

func NewCommit(projectName, repoName, branchName, commitID string) *pfs.Commit

NewCommit creates a pfs.Commit in the given project, repo & branch.

func NewCommitSet 

func NewCommitSet(id string) *pfs.CommitSet

NewCommitSet creates a pfs.CommitSet

func NewCronInput 

func NewCronInput(name string, spec string) *pps.Input

NewCronInput returns an input which will trigger based on a timed schedule. It uses cron syntax to specify the schedule. The input will be exposed to jobs as `/pfs/<name>/<timestamp>`. The timestamp uses the RFC 3339 format, e.g. `2006-01-02T15:04:05Z07:00`. It only takes required options.

func NewCronInputOpts 

func NewCronInputOpts(name string, repo string, spec string, overwrite bool, start *timestamppb.Timestamp) *pps.Input

NewCronInputOpts returns an input which will trigger based on a timed schedule. It uses cron syntax to specify the schedule. The input will be exposed to jobs as `/pfs/<name>/<timestamp>`. The timestamp uses the RFC 3339 format, e.g. `2006-01-02T15:04:05Z07:00`. It includes all the options.

func NewCrossInput 

func NewCrossInput(input ...*pps.Input) *pps.Input

NewCrossInput returns an input which is the cross product of other inputs. That means that all combination of datums will be seen by the job / pipeline.

func NewFile 

func NewFile(projectName, repoName, branchName, commitID, path string) *pfs.File

NewFile creates a pfs.File.

func NewGroupInput 

func NewGroupInput(input ...*pps.Input) *pps.Input

NewGroupInput returns an input which groups the inputs by the GroupBy pattern. That means that it will return a datum for each group of input datums matching a particular GroupBy pattern

func NewJob 

func NewJob(projectName, pipelineName, jobID string) *pps.Job

NewJob creates a pps.Job.

func NewJobSet 

func NewJobSet(id string) *pps.JobSet

NewJobSet creates a pps.JobSet.

func NewJoinInput 

func NewJoinInput(input ...*pps.Input) *pps.Input

NewJoinInput returns an input which is the join of other inputs. That means that all combination of datums which match on `joinOn` will be seen by the job / pipeline.

func NewPFSInput 

func NewPFSInput(project, repo, glob string) *pps.Input

NewPFSInput returns a new PFS input. It only includes required options.

func NewPFSInputOpts 

func NewPFSInputOpts(name, project, repo, branch, glob, joinOn, groupBy string, outerJoin, lazy bool, trigger *pfs.Trigger) *pps.Input

NewPFSInputOpts returns a new PFS input. It includes all options.

func NewPipeline 

func NewPipeline(projectName, pipelineName string) *pps.Pipeline

NewPipeline creates a pps.Pipeline.

func NewProject 

func NewProject(name string) *pfs.Project

NewProject creates a pfs.Project

func NewProxyPostgresListener 

func NewProxyPostgresListener(clientFactory func() (proxy.APIClient, error)) col.PostgresListener

func NewRepo 

func NewRepo(projectName, repoName string) *pfs.Repo

NewRepo creates a new PFS repo.

func NewS3PFSInput 

func NewS3PFSInput(name, project, repo, branch string) *pps.Input

NewS3PFSInput returns a new PFS input with 'S3' set.

func NewSystemRepo 

func NewSystemRepo(projectName, repoName, repoType string) *pfs.Repo

NewSystemRepo creates a pfs.Repo of the given type in the given project.

func NewUnionInput 

func NewUnionInput(input ...*pps.Input) *pps.Input

NewUnionInput returns an input which is the union of other inputs. That means that all datums from any of the inputs will be seen individually by the job / pipeline.

func PutFile added in v2.8.0 

func PutFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string, r io.Reader, opts ...PutFileOption) error

func PutFileTAR added in v2.8.0 

func PutFileTAR(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, r io.Reader, opts ...PutFileOption) error

func PutFileURL added in v2.8.0 

func PutFileURL(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path, url string, recursive bool, opts ...PutFileOption) error

func RenewFileSet added in v2.8.0 

func RenewFileSet(ctx context.Context, c pfs.APIClient, ID string, ttl time.Duration) (retErr error)

func SetAuthToken added in v2.8.0 

func SetAuthToken(ctx context.Context, token string) context.Context

func ShardFileSet added in v2.8.0 

func ShardFileSet(ctx context.Context, c pfs.APIClient, ID string) (_ []*pfs.PathRange, retErr error)

func ShardFileSetWithConfig added in v2.9.0 

func ShardFileSetWithConfig(ctx context.Context, c pfs.APIClient, ID string, numFiles, sizeBytes int64) (_ []*pfs.PathRange, retErr error)

ShardFileSetWithConfig shards a file set using the given configuration. `numFiles` is the number of files targeted in each shard. `sizeBytes` is the size (in bytes) targeted for each shard. If a shard configuration field (numFiles, sizeBytes) is 0, the file set's default value is used. If both `numFiles` and `sizeBytes` are non-zero, shards are created based on whichever threshold is surpassed first.

func StartCommit added in v2.8.0 

func StartCommit(ctx context.Context, c pfs.APIClient, projectName, repoName string, branchName string) (_ *pfs.Commit, retErr error)

func WaitCommitSet added in v2.8.0 

func WaitCommitSet(ctx context.Context, c pfs.APIClient, id string, cb func(*pfs.CommitInfo) error) (retErr error)

func WalkFile added in v2.8.0 

func WalkFile(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, path string, cb func(*pfs.FileInfo) error) (retErr error)

func WithCreateFileSetClient added in v2.8.0 

func WithCreateFileSetClient(ctx context.Context, c pfs.APIClient, cb func(ModifyFile) error) (resp *pfs.CreateFileSetResponse, retErr error)

func WithModifyFileClient added in v2.8.0 

func WithModifyFileClient(ctx context.Context, c pfs.APIClient, commit *pfs.Commit, cb func(ModifyFile) error) (retErr error)

func WithRenewer added in v2.8.0 

func WithRenewer(ctx context.Context, c pfs.APIClient, cb func(context.Context, *renew.StringSet) error) error

func WithSystemCAs 

func WithSystemCAs(settings *clientSettings) error

WithSystemCAs uses the system certs for client creation, if no others are provided. This is the default behaviour when the scheme is `https` or `grpcs`.

Types

type APIClient 

type APIClient struct {
	PfsAPIClient
	PpsAPIClient
	AuthAPIClient
	IdentityAPIClient
	VersionAPIClient
	AdminAPIClient
	TransactionAPIClient
	DebugClient
	ProxyClient
	Enterprise enterprise.APIClient // not embedded--method name conflicts with AuthAPIClient
	License    license.APIClient
	// contains filtered or unexported fields
}

An APIClient is a wrapper around pfs, pps and block APIClients.

func NewEnterpriseClientForTest 

func NewEnterpriseClientForTest(ctx context.Context) (*APIClient, error)

NewEnterpriseClientForTest constructs a new APIClient for tests. TODO(actgardner): this should probably live in testutils and accept a testing.TB

func NewEnterpriseClientOnUserMachine 

func NewEnterpriseClientOnUserMachine(prefix string, options ...Option) (*APIClient, error)

NewEnterpriseClientOnUserMachine constructs a new APIClient using $HOME/.pachyderm/config if it exists. This is intended to be used in the pachctl binary to communicate with the enterprise server.

func NewEnterpriseClientOnUserMachineContext 

func NewEnterpriseClientOnUserMachineContext(ctx context.Context, prefix string, options ...Option) (*APIClient, error)

NewEnterpriseClientOnUserMachineContext is like NewEnterpriseClientOnUserMachine, but accepts a context for dialing and future RPCs.

func NewFromPachdAddress 

func NewFromPachdAddress(ctx context.Context, pachdAddress *grpcutil.PachdAddress, options ...Option) (*APIClient, error)

NewFromPachdAddress creates a new client given a parsed GRPC address.

func NewFromURI 

func NewFromURI(ctx context.Context, uri string, options ...Option) (*APIClient, error)

NewFromURI creates a new client given a GRPC URI ex. grpc://test.example.com. If no scheme is specified `grpc://` is assumed. A scheme of `grpcs://` enables TLS.

func NewInCluster 

func NewInCluster(ctx context.Context, options ...Option) (*APIClient, error)

NewInCluster constructs a new APIClient using env vars that Kubernetes creates. This should be used to access Pachyderm from within a Kubernetes cluster with Pachyderm running on it.

func NewInWorker 

func NewInWorker(ctx context.Context, options ...Option) (*APIClient, error)

NewInWorker constructs a new APIClient intended to be used from a worker to talk to the sidecar pachd container.

func NewOnUserMachine 

func NewOnUserMachine(ctx context.Context, prefix string, options ...Option) (*APIClient, error)

NewOnUserMachine constructs a new APIClient using $HOME/.pachyderm/config if it exists. This is intended to be used in the pachctl binary.

func (APIClient) AddFileSet 

func (c APIClient) AddFileSet(project, repo, branch, commit, ID string) (retErr error)

func (*APIClient) AddMetadata 

func (c *APIClient) AddMetadata(ctx context.Context) context.Context

AddMetadata adds necessary metadata (including authentication credentials) to the context 'ctx', preserving any metadata that is present in either the incoming or outgoing metadata of 'ctx'.

func (*APIClient) AuthToken 

func (c *APIClient) AuthToken() string

AuthToken gets the authentication token that is set for this client.

func (APIClient) Binary 

func (c APIClient) Binary(filter *debug.Filter, w io.Writer) (retErr error)

Binary collects a set of binaries.

func (*APIClient) ClientConn added in v2.8.0 

func (c *APIClient) ClientConn() *grpc.ClientConn

ClientConn returns the current grpc client connection.

func (*APIClient) ClientContextName 

func (c *APIClient) ClientContextName() string

ClientContextName returns the name of the context in the client config that produced this client, or an empty string if the client was not produced from a configured client context.

func (*APIClient) Close 

func (c *APIClient) Close() error

Close the connection to gRPC

func (*APIClient) ClusterInfo added in v2.8.0 

func (c *APIClient) ClusterInfo() (info *admin.ClusterInfo, ok bool)

ClusterInfo returns information about the cluster that is retrieved at connection time, saving a redundant call to InspectCluster.

func (APIClient) ComposeFileSet 

func (c APIClient) ComposeFileSet(IDs []string, ttl time.Duration) (_ string, retErr error)

ComposeFileSet composes a file set from a list of file sets.

func (APIClient) CopyFile 

func (c APIClient) CopyFile(dstCommit *pfs.Commit, dstPath string, srcCommit *pfs.Commit, srcPath string, opts ...CopyFileOption) error

CopyFile copies a file from one PFS location to another. It can be used on directories or regular files.

func (APIClient) CreateBranch 

func (c APIClient) CreateBranch(projectName, repoName, branchName, commitBranch, commitID string, provenance []*pfs.Branch) error

CreateBranch creates a new branch

func (APIClient) CreateBranchTrigger 

func (c APIClient) CreateBranchTrigger(projectName, repoName, branchName, commitBranch, commitID string, trigger *pfs.Trigger) error

CreateBranchTrigger creates a branch with a trigger. Note: triggers and provenance are mutually exclusive. See the docs on triggers to learn more about why this is.

func (APIClient) CreatePipeline 

func (c APIClient) CreatePipeline(projectName, pipelineName, image string, cmd []string, stdin []string, parallelismSpec *pps.ParallelismSpec, input *pps.Input, outputBranch string, update bool) error

CreatePipeline creates a new pipeline, pipelines are the main computation object in PPS they create a flow of data from a set of input Repos to an output Repo (which has the same name as the pipeline). Whenever new data is committed to one of the input repos the pipelines will create jobs to bring the output Repo up to data.

image is the Docker image to run the jobs in.

cmd is the command passed to the Docker run invocation. NOTE as with Docker cmd is not run inside a shell that means that things like wildcard globbing (*), pipes (|) and file redirects (> and >>) will not work. To get that behavior you should have your command be a shell of your choice and pass a shell script to stdin.

stdin is a slice of lines that are sent to your command on stdin. Lines need not end in newline characters.

parallelism is how many copies of your container should run in parallel. You may pass 0 for parallelism in which case PPS will set the parallelism based on available resources.

input specifies a set of Repos that will be visible to the jobs during runtime. commits to these repos will cause the pipeline to create new jobs to process them. update indicates that you want to update an existing pipeline.

func (APIClient) CreatePipelineService 

func (c APIClient) CreatePipelineService(projectName, pipelineName, image string, cmd, stdin []string, parallelismSpec *pps.ParallelismSpec, input *pps.Input, update bool, internalPort, externalPort int32, annotations map[string]string) error

CreatePipelineService creates a new pipeline service.

func (APIClient) CreateProject 

func (c APIClient) CreateProject(name string) error

CreateProject creates a new Project object in pfs with the given name.

func (APIClient) CreateRepo 

func (c APIClient) CreateRepo(projectName, repoName string) error

CreateRepo creates a new Repo object in pfs with the given name. Repos are the top level data object in pfs and should be used to store data of a similar type. For example rather than having a single Repo for an entire project you might have separate Repos for logs, metrics, database dumps etc.

func (APIClient) CreateSecret 

func (c APIClient) CreateSecret(file []byte) error

CreateSecret creates a secret on the cluster.

func (*APIClient) Ctx 

func (c *APIClient) Ctx() context.Context

Ctx is a convenience function that returns adds Pachyderm authn metadata to context.Background().

func (APIClient) DeleteAll 

func (c APIClient) DeleteAll() error

DeleteAll deletes everything in the cluster. Use with caution, there is no undo. TODO: rewrite this to use transactions

func (APIClient) DeleteAllEnterprise 

func (c APIClient) DeleteAllEnterprise() error

DeleteAllEnterprise deletes everything in the enterprise server. Use with caution, there is no undo. TODO: rewrite this to use transactions

func (APIClient) DeleteBranch 

func (c APIClient) DeleteBranch(projectName, repoName, branchName string, force bool) error

DeleteBranch deletes a branch, but leaves the commits themselves intact. In other words, those commits can still be accessed via commit IDs and other branches they happen to be on.

func (APIClient) DeleteFile 

func (c APIClient) DeleteFile(commit *pfs.Commit, path string, opts ...DeleteFileOption) error

DeleteFile deletes a file from PFS.

func (APIClient) DeleteJob 

func (c APIClient) DeleteJob(projectName, pipelineName, jobID string) error

DeleteJob deletes a job.

func (APIClient) DeletePipeline 

func (c APIClient) DeletePipeline(projectName, pipelineName string, force bool) error

DeletePipeline deletes a pipeline along with its output Repo.

func (APIClient) DeleteProject 

func (c APIClient) DeleteProject(projectName string, force bool) error

DeleteProject deletes a project.

If "force" is set to true, the project will be removed regardless of errors. This argument should be used with care.

func (APIClient) DeleteRepo 

func (c APIClient) DeleteRepo(projectName, repoName string, force bool) error

DeleteRepo deletes a repo and reclaims the storage space it was using. Note that as of 1.0 we do not reclaim the blocks that the Repo was referencing, this is because they may also be referenced by other Repos and deleting them would make those Repos inaccessible. This will be resolved in later versions.

If "force" is set to true, the repo will be removed regardless of errors. This argument should be used with care.

func (APIClient) DeleteSecret 

func (c APIClient) DeleteSecret(secret string) error

DeleteSecret deletes a secret from the cluster.

func (APIClient) DeleteTransaction 

func (c APIClient) DeleteTransaction(txn *transaction.Transaction) error

DeleteTransaction is an RPC that aborts an existing transaction in the Pachyderm cluster and removes it from the cluster.

func (APIClient) DiffFile 

func (c APIClient) DiffFile(newCommit *pfs.Commit, newPath string, oldCommit *pfs.Commit, oldPath string, shallow bool, cb func(*pfs.FileInfo, *pfs.FileInfo) error) (retErr error)

DiffFile returns the differences between 2 paths at 2 commits. It streams back one file at a time which is either from the new path, or the old path

func (APIClient) DiffFileAll 

func (c APIClient) DiffFileAll(newCommit *pfs.Commit, newPath string, oldCommit *pfs.Commit, oldPath string, shallow bool) (_ []*pfs.FileInfo, _ []*pfs.FileInfo, retErr error)

DiffFileAll returns the differences between 2 paths at 2 commits.

func (APIClient) DropCommitSet 

func (c APIClient) DropCommitSet(id string) error

DropCommitSet drop the commits of a CommitSet and all data included in those commits.

func (APIClient) Dump 

func (c APIClient) Dump(filter *debug.Filter, limit int64, w io.Writer) (retErr error)

Dump collects a standard set of debugging information.

func (APIClient) ExecuteInTransaction 

func (c APIClient) ExecuteInTransaction(f func(c *APIClient) error) (*transaction.TransactionInfo, error)

ExecuteInTransaction executes a callback within a transaction. The callback should use the passed in APIClient. If the callback returns a nil error, then the transaction will be finished. If the callback returns a non-nil error, then the transaction will be deleted.

func (APIClient) FindCommits 

func (c APIClient) FindCommits(req *pfs.FindCommitsRequest) (*FindCommitsResponse, error)

FindCommits searches for commits that reference a supplied file being modified in a branch.

func (APIClient) FinishCommit 

func (c APIClient) FinishCommit(projectName, repoName, branchName, commitID string) (retErr error)

FinishCommit ends the process of committing data to a Repo and persists the Commit. Once a Commit is finished the data becomes immutable and future attempts to write to it with PutFile will error.

func (APIClient) FinishTransaction 

func (c APIClient) FinishTransaction(txn *transaction.Transaction) (*transaction.TransactionInfo, error)

FinishTransaction is an RPC that closes an existing transaction in the Pachyderm cluster and commits its changes to the persisted cluster metadata transactionally.

func (APIClient) Fsck 

func (c APIClient) Fsck(fix bool, cb func(*pfs.FsckResponse) error, opts ...FsckOption) error

Fsck performs checks on pfs. Errors that are encountered will be passed onError. These aren't errors in the traditional sense, in that they don't prevent the completion of fsck. Errors that do prevent completion will be returned from the function.

func (APIClient) FsckFastExit 

func (c APIClient) FsckFastExit() error

FsckFastExit performs checks on pfs, similar to Fsck, except that it returns the first fsck error it encounters and exits.

func (*APIClient) GetAddress 

func (c *APIClient) GetAddress() *grpcutil.PachdAddress

GetAddress returns the pachd host:port with which 'c' is communicating. If 'c' was created using NewInCluster or NewOnUserMachine then this is how the address may be retrieved from the environment.

func (APIClient) GetClusterRoleBinding 

func (c APIClient) GetClusterRoleBinding(ctx context.Context) (*auth.RoleBinding, error)

func (APIClient) GetFile 

func (c APIClient) GetFile(commit *pfs.Commit, path string, w io.Writer, opts ...GetFileOption) (retErr error)

GetFile returns the contents of a file at a specific Commit. offset specifies a number of bytes that should be skipped in the beginning of the file. size limits the total amount of data returned, note you will get fewer bytes than size if you pass a value larger than the size of the file. If size is set to 0 then all of the data will be returned.

func (APIClient) GetFileReadSeeker 

func (c APIClient) GetFileReadSeeker(commit *pfs.Commit, path string) (io.ReadSeeker, error)

GetFileReadSeeker returns a reader for the contents of a file at a specific Commit that permits Seeking to different points in the file.

func (APIClient) GetFileReader 

func (c APIClient) GetFileReader(commit *pfs.Commit, path string) (io.Reader, error)

func (APIClient) GetFileSet 

func (c APIClient) GetFileSet(project, repo, branch, commit string) (_ string, retErr error)

GetFileSet gets a file set for a commit in a project.

func (APIClient) GetFileTAR 

func (c APIClient) GetFileTAR(commit *pfs.Commit, path string) (io.ReadCloser, error)

func (APIClient) GetFileURL 

func (c APIClient) GetFileURL(commit *pfs.Commit, path, URL string) (retErr error)

GetFileURL gets the file at the specified URL

func (APIClient) GetLogs 

func (c APIClient) GetLogs(projectName, pipelineName, jobID string, data []string, datumID string, master, follow bool, since time.Duration) *LogsIter

GetLogs gets logs from a job (logs includes stdout and stderr). 'pipelineName', 'jobID', 'data', and 'datumID', are all filters. To forego any filter, simply pass an empty value, though one of 'pipelineName' and 'jobID' must be set. Responses are written to 'messages'.

func (APIClient) GetLogsLoki 

func (c APIClient) GetLogsLoki(
	pipelineName string,
	jobID string,
	data []string,
	datumID string,
	master bool,
	follow bool,
	since time.Duration,
) *LogsIter

GetLogsLoki gets logs from a job (logs includes stdout and stderr). 'pipelineName', 'jobID', 'data', and 'datumID', are all filters. To forego any filter, simply pass an empty value, though one of 'pipelineName' and 'jobID' must be set. Responses are written to 'messages'.

func (APIClient) GetProjectLogsLoki 

func (c APIClient) GetProjectLogsLoki(projectName, pipelineName, jobID string, data []string, datumID string, master, follow bool, since time.Duration) *LogsIter

GetProjectLogsLoki gets logs from a job (logs includes stdout and stderr). 'pipelineName', 'jobID', 'data', and 'datumID', are all filters. To forego any filter, simply pass an empty value, though one of 'pipelineName' and 'jobID' must be set. Responses are written to 'messages'.

func (APIClient) GetProjectRoleBinding 

func (c APIClient) GetProjectRoleBinding(project string) (*auth.RoleBinding, error)

func (APIClient) GetRepoRoleBinding 

func (c APIClient) GetRepoRoleBinding(projectName, repoName string) (*auth.RoleBinding, error)

Return the roles bound to a repo within a project.

func (*APIClient) GetRequestMetadata 

func (c *APIClient) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error)

func (APIClient) GetTransaction 

func (c APIClient) GetTransaction() (*transaction.Transaction, error)

GetTransaction is a helper function to get the active transaction from the client's context metadata.

func (APIClient) GlobFile 

func (c APIClient) GlobFile(commit *pfs.Commit, pattern string, cb func(fi *pfs.FileInfo) error) (retErr error)

GlobFile returns files that match a given glob pattern in a given commit, calling cb with each FileInfo. The pattern is documented here: https://golang.org/pkg/path/filepath/#Match

func (APIClient) GlobFileAll 

func (c APIClient) GlobFileAll(commit *pfs.Commit, pattern string) (_ []*pfs.FileInfo, retErr error)

GlobFileAll returns files that match a given glob pattern in a given commit. The pattern is documented here: https://golang.org/pkg/path/filepath/#Match

func (APIClient) Health 

func (c APIClient) Health() error

Health health checks pachd, it returns an error if pachd isn't healthy.

func (APIClient) InspectBranch 

func (c APIClient) InspectBranch(projectName, repoName string, branchName string) (*pfs.BranchInfo, error)

InspectBranch returns information on a specific PFS branch.

func (APIClient) InspectCluster 

func (c APIClient) InspectCluster() (*admin.ClusterInfo, error)

InspectCluster retrieves cluster state

func (APIClient) InspectClusterWithVersionAndProject 

func (c APIClient) InspectClusterWithVersionAndProject(v *versionpb.Version, p *pfs.Project) (*admin.ClusterInfo, error)

InspectCluster retrieves cluster state, and sends the server its version for the server to validate.

func (APIClient) InspectCommit 

func (c APIClient) InspectCommit(projectName, repoName, branchName, commitID string) (_ *pfs.CommitInfo, retErr error)

InspectCommit returns info about a specific Commit.

func (APIClient) InspectCommitSet 

func (c APIClient) InspectCommitSet(id string) (_ []*pfs.CommitInfo, retErr error)

InspectCommitSet returns info about a specific CommitSet.

func (APIClient) InspectDatum 

func (c APIClient) InspectDatum(projectName, pipelineName, jobID, datumID string) (*pps.DatumInfo, error)

InspectDatum returns info about a single datum.

func (APIClient) InspectFile 

func (c APIClient) InspectFile(commit *pfs.Commit, path string) (_ *pfs.FileInfo, retErr error)

InspectFile returns metadata about the specified file

func (APIClient) InspectJob 

func (c APIClient) InspectJob(projectName, pipelineName, jobID string, details bool) (_ *pps.JobInfo, retErr error)

InspectJob returns info about a specific job.

'details' indicates that the JobInfo.Details field should be filled out.

func (APIClient) InspectJobSet 

func (c APIClient) InspectJobSet(id string, details bool) (_ []*pps.JobInfo, retErr error)

func (APIClient) InspectPipeline 

func (c APIClient) InspectPipeline(projectName, pipelineName string, details bool) (*pps.PipelineInfo, error)

InspectPipeline returns info about a specific pipeline. The name may include ancestry syntax or be a bare name.

func (APIClient) InspectProject 

func (c APIClient) InspectProject(name string) (*pfs.ProjectInfo, error)

InspectProject returns info about a specific Project.

func (APIClient) InspectRepo 

func (c APIClient) InspectRepo(projectName, repoName string) (_ *pfs.RepoInfo, retErr error)

InspectRepo returns info about a specific Repo.

func (APIClient) InspectSecret 

func (c APIClient) InspectSecret(secret string) (*pps.SecretInfo, error)

InspectSecret returns info about a specific secret.

func (APIClient) InspectTransaction 

func (c APIClient) InspectTransaction(txn *transaction.Transaction) (*transaction.TransactionInfo, error)

InspectTransaction is an RPC that fetches the detailed information for an existing transaction in the Pachyderm cluster.

func (APIClient) IsAuthActive 

func (c APIClient) IsAuthActive(ctx context.Context) (bool, error)

IsAuthActive returns whether auth is activated on the cluster

func (APIClient) ListBranch 

func (c APIClient) ListBranch(projectName, repoName string) ([]*pfs.BranchInfo, error)

ListBranch lists the active branches on a Repo.

func (APIClient) ListCommit 

func (c APIClient) ListCommit(repo *pfs.Repo, to, from *pfs.Commit, number int64) ([]*pfs.CommitInfo, error)

ListCommit lists commits. If only `repo` is given, all commits in the repo are returned. If `to` is given, only the ancestors of `to`, including `to` itself, are considered. If `from` is given, only the descendents of `from`, including `from` itself, are considered. If `to` and `from` are the same commit, no commits will be returned. `number` determines how many commits are returned. If `number` is 0, all commits that match the aforementioned criteria are returned.

func (APIClient) ListCommitByRepo 

func (c APIClient) ListCommitByRepo(repo *pfs.Repo) ([]*pfs.CommitInfo, error)

ListCommitByRepo lists all commits in a repo.

func (APIClient) ListCommitF 

func (c APIClient) ListCommitF(repo *pfs.Repo, to, from *pfs.Commit, number int64, reverse bool, f func(*pfs.CommitInfo) error) error

ListCommitF lists commits, calling f with each commit. If only `repo` is given, all commits in the repo are returned. If `to` is given, only the ancestors of `to`, including `to` itself, are considered. If `from` is given, only the descendents of `from`, including `from` itself, are considered. If `to` and `from` are the same commit, no commits will be returned. `number` determines how many commits are returned. If `number` is 0, `reverse` lists the commits from oldest to newest, rather than newest to oldest all commits that match the aforementioned criteria are passed to f.

func (APIClient) ListDatum 

func (c APIClient) ListDatum(projectName, pipelineName, jobID string, cb func(*pps.DatumInfo) error) (retErr error)

ListDatum returns info about datums in a job.

func (APIClient) ListDatumAll 

func (c APIClient) ListDatumAll(projectName, pipelineName, jobID string) (_ []*pps.DatumInfo, retErr error)

ListDatumAll returns info about datums in a job.

func (APIClient) ListDatumInput 

func (c APIClient) ListDatumInput(input *pps.Input, cb func(*pps.DatumInfo) error) (retErr error)

ListDatumInput returns info about datums for a pipeline with input. The pipeline doesn't need to exist.

func (APIClient) ListDatumInputAll 

func (c APIClient) ListDatumInputAll(input *pps.Input) (_ []*pps.DatumInfo, retErr error)

ListDatumInputAll returns info about datums for a pipeline with input. The pipeline doesn't need to exist.

func (APIClient) ListFile 

func (c APIClient) ListFile(commit *pfs.Commit, path string, cb func(fi *pfs.FileInfo) error) (retErr error)

ListFile returns info about all files in a Commit under path, calling cb with each FileInfo.

func (APIClient) ListFileAll 

func (c APIClient) ListFileAll(commit *pfs.Commit, path string) (_ []*pfs.FileInfo, retErr error)

ListFileAll returns info about all files in a Commit under path.

func (APIClient) ListJob 

func (c APIClient) ListJob(projectName, pipelineName string, inputCommit []*pfs.Commit, history int64, details bool) ([]*pps.JobInfo, error)

ListJob returns info about all jobs.

If projectName & pipelineName are non empty then only jobs that were started by the named pipeline will be returned.

If inputCommit is non-nil then only jobs which took the specific commits as inputs will be returned.

The order of the inputCommits doesn't matter.

If outputCommit is non-nil then only the job which created that commit as output will be returned.

'history' controls whether jobs from historical versions of pipelines are returned, it has the following semantics:

  • 0: Return jobs from the current version of the pipeline or pipelines.
  • 1: Return the above and jobs from the next most recent version
  • 2: etc.
  • -1: Return jobs from all historical versions.

'details' controls whether the JobInfo passed to 'f' includes details from the pipeline spec (e.g. the transform). Leaving this 'false' can improve performance.

func (APIClient) ListJobF 

func (c APIClient) ListJobF(projectName, pipelineName string, inputCommit []*pfs.Commit,
	history int64, details bool,
	f func(*pps.JobInfo) error) error

ListJobF is a previous version of ListJobFilterF, returning info about all jobs and calling f on each JobInfo.

func (APIClient) ListJobFilterF 

func (c APIClient) ListJobFilterF(pipelineName string, inputCommit []*pfs.Commit,
	history int64, details bool, jqFilter string,
	f func(*pps.JobInfo) error) error

ListJobFilterF returns info about all jobs, calling f with each JobInfo.

If f returns an error iteration of jobs will stop and ListJobF will return that error, unless the error is errutil.ErrBreak in which case it will return nil.

If pipelineName is non empty then only jobs that were started by the named pipeline will be returned.

If inputCommit is non-nil then only jobs which took the specific commits as inputs will be returned.

The order of the inputCommits doesn't matter.

If outputCommit is non-nil then only the job which created that commit as output will be returned.

'history' controls whether jobs from historical versions of pipelines are returned, it has the following semantics:

  • 0: Return jobs from the current version of the pipeline or pipelines.
  • 1: Return the above and jobs from the next most recent version
  • 2: etc.
  • -1: Return jobs from all historical versions.

'details' controls whether the JobInfo passed to 'f' includes details from the pipeline spec--setting this to 'false' can improve performance.

func (APIClient) ListPipeline 

func (c APIClient) ListPipeline() ([]*pps.PipelineInfo, error)

ListPipeline returns info about all pipelines.

func (APIClient) ListPipelineHistory 

func (c APIClient) ListPipelineHistory(projectName, pipelineName string, history int64) ([]*pps.PipelineInfo, error)

- 0: Return the current version of the pipeline or pipelines. - 1: Return the above and the next most recent version - 2: etc. - -1: Return all historical versions.

func (APIClient) ListProject 

func (c APIClient) ListProject() (_ []*pfs.ProjectInfo, retErr error)

ListProject lists projects.

func (APIClient) ListProjectJobFilterF 

func (c APIClient) ListProjectJobFilterF(projectName, pipelineName string, inputCommit []*pfs.Commit,
	history int64, details bool, jqFilter string,
	f func(*pps.JobInfo) error) error

ListProjectJobFilterF returns info about all jobs, calling f with each JobInfo.

If f returns an error iteration of jobs will stop and ListJobF will return that error, unless the error is errutil.ErrBreak in which case it will return nil.

If projectName & pipelineName are both non-empty then only jobs that were started by the named pipeline will be returned.

If inputCommit is non-nil then only jobs which took the specific commits as inputs will be returned.

The order of the inputCommits doesn't matter.

If outputCommit is non-nil then only the job which created that commit as output will be returned.

'history' controls whether jobs from historical versions of pipelines are returned, it has the following semantics:

  • 0: Return jobs from the current version of the pipeline or pipelines.
  • 1: Return the above and jobs from the next most recent version
  • 2: etc.
  • -1: Return jobs from all historical versions.

'details' controls whether the JobInfo passed to 'f' includes details from the pipeline spec--setting this to 'false' can improve performance.

func (APIClient) ListProjectRepo 

func (c APIClient) ListProjectRepo(r *pfs.ListRepoRequest) ([]*pfs.RepoInfo, error)

ListProjectRepo returns a list of RepoInfos given a ListRepoRequest, which can include information about which projects to filter with.

func (APIClient) ListRepo 

func (c APIClient) ListRepo() ([]*pfs.RepoInfo, error)

ListRepo returns info about user Repos

func (APIClient) ListRepoByType 

func (c APIClient) ListRepoByType(repoType string) (_ []*pfs.RepoInfo, retErr error)

ListRepoByType returns info about Repos of the given type.

The if repoType is empty, all Repos will be included

func (APIClient) ListSecret 

func (c APIClient) ListSecret() ([]*pps.SecretInfo, error)

ListSecret returns info about all Pachyderm secrets.

func (APIClient) ListTask 

func (c APIClient) ListTask(service string, namespace, group string, cb func(*task.TaskInfo) error) (retErr error)

ListTask lists tasks in the given namespace and group

func (APIClient) ListTransaction 

func (c APIClient) ListTransaction() ([]*transaction.TransactionInfo, error)

ListTransaction is an RPC that fetches a list of all open transactions in the Pachyderm cluster.

func (APIClient) ModifyClusterRoleBinding 

func (c APIClient) ModifyClusterRoleBinding(principal string, roles []string) error

func (APIClient) ModifyProjectRoleBinding 

func (c APIClient) ModifyProjectRoleBinding(projectName, principal string, roles []string) error

ModifyProjectRoleBinding binds a user's roles to a project.

func (APIClient) ModifyRepoRoleBinding 

func (c APIClient) ModifyRepoRoleBinding(projectName, repoName, principal string, roles []string) error

Update the roles bound to a repo within a project.

func (APIClient) NewCreateFileSetClient 

func (c APIClient) NewCreateFileSetClient() (_ *CreateFileSetClient, retErr error)

NewCreateFileSetClient returns a CreateFileSetClient instance backed by this client

func (APIClient) NewModifyFileClient 

func (c APIClient) NewModifyFileClient(commit *pfs.Commit) (_ *ModifyFileClient, retErr error)

NewModifyFileClient creates a new ModifyFileClient.

func (APIClient) Profile 

func (c APIClient) Profile(profile *debug.Profile, filter *debug.Filter, w io.Writer) (retErr error)

Profile collects a set of pprof profiles.

func (APIClient) PutFile 

func (c APIClient) PutFile(commit *pfs.Commit, path string, r io.Reader, opts ...PutFileOption) error

PutFile puts a file into PFS from a reader.

func (APIClient) PutFileTAR 

func (c APIClient) PutFileTAR(commit *pfs.Commit, r io.Reader, opts ...PutFileOption) error

PutFileTAR puts a set of files into PFS from a tar stream.

func (APIClient) PutFileURL 

func (c APIClient) PutFileURL(commit *pfs.Commit, path, url string, recursive bool, opts ...PutFileOption) error

PutFileURL puts a file into PFS using the content found at a URL. The URL is sent to the server which performs the request. recursive allow for recursive scraping of some types of URLs for example on s3:// urls.

func (APIClient) RenewFileSet 

func (c APIClient) RenewFileSet(ID string, ttl time.Duration) (retErr error)

RenewFileSet renews a fileset.

func (*APIClient) RequireTransportSecurity 

func (c *APIClient) RequireTransportSecurity() bool

func (APIClient) RestartDatum 

func (c APIClient) RestartDatum(projectName, pipelineName, jobID string, datumFilter []string) error

RestartDatum restarts a datum that's being processed as part of a job.

datumFilter is a slice of strings which are matched against either the Path or Hash of the datum, the order of the strings in datumFilter is irrelevant.

func (APIClient) RunBatchInTransaction 

func (c APIClient) RunBatchInTransaction(cb func(builder *TransactionBuilder) error) (*transaction.TransactionInfo, error)

RunBatchInTransaction will execute a batch of API calls in a single round-trip transactionally. The callback is used to build the request, which is executed when the callback returns.

func (APIClient) RunCron 

func (c APIClient) RunCron(projectName, pipelineName string) error

RunCron runs a pipeline. It can be passed a list of commit provenance. This will trigger a new job provenant on those commits, effectively running the pipeline on the data in those commits.

func (APIClient) RunPipeline 

func (c APIClient) RunPipeline(projectName, pipelineName string, provenance []*pfs.Commit, jobID string) error

RunPipeline runs a pipeline. It can be passed a list of commit provenance. This will trigger a new job provenant on those commits, effectively running the pipeline on the data in those commits.

func (*APIClient) SetAuthToken 

func (c *APIClient) SetAuthToken(token string)

SetAuthToken sets the authentication token that will be used for all API calls for this client.

func (APIClient) ShardFileSet 

func (c APIClient) ShardFileSet(ID string) (_ []*pfs.PathRange, retErr error)

func (APIClient) SquashCommitSet 

func (c APIClient) SquashCommitSet(id string) error

SquashCommitSet squashes the commits of a CommitSet into their children.

func (APIClient) StartCommit 

func (c APIClient) StartCommit(projectName, repoName string, branchName string) (_ *pfs.Commit, retErr error)

StartCommit begins the process of committing data to a Repo. Once started you can write to the Commit with PutFile and when all the data has been written you must finish the Commit with FinishCommit. NOTE, data is not persisted until FinishCommit is called.

branch is a more convenient way to build linear chains of commits. When a commit is started with a non empty branch the value of branch becomes an alias for the created Commit. This enables a more intuitive access pattern. When the commit is started on a branch the previous head of the branch is used as the parent of the commit.

func (APIClient) StartPipeline 

func (c APIClient) StartPipeline(projectName, pipelineName string) error

StartPipeline restarts a stopped pipeline.

func (APIClient) StartTransaction 

func (c APIClient) StartTransaction() (*transaction.Transaction, error)

StartTransaction is an RPC that registers a new transaction with the Pachyderm cluster and returns the identifier of the new transaction.

func (APIClient) StopJob 

func (c APIClient) StopJob(projectName, pipelineName, jobID string) error

StopJob stops a job.

func (APIClient) StopPipeline 

func (c APIClient) StopPipeline(projectName, pipelineName string) error

StopPipeline prevents a pipeline from processing things; it can be restarted with StartProjectPipeline.

func (APIClient) SubscribeCommit 

func (c APIClient) SubscribeCommit(repo *pfs.Repo, branchName string, from string, state pfs.CommitState, cb func(*pfs.CommitInfo) error) (retErr error)

SubscribeCommit is like ListCommit but it keeps listening for commits as they come in.

func (APIClient) SubscribeJob 

func (c APIClient) SubscribeJob(projectName, pipelineName string, details bool, cb func(*pps.JobInfo) error) error

SubscribeJob calls the given callback with each open job in the given pipeline until cancelled.

func (APIClient) UpdateProject 

func (c APIClient) UpdateProject(projectName, description string) error

UpdateProject upserts a project with the given name.

func (APIClient) UpdateRepo 

func (c APIClient) UpdateRepo(projectName, repoName string) error

UpdateRepo upserts a repo with the given name.

func (APIClient) Version 

func (c APIClient) Version() (string, error)

Version returns the version of pachd as a string.

func (APIClient) WaitCommit 

func (c APIClient) WaitCommit(projectName, repoName, branchName, commitID string) (_ *pfs.CommitInfo, retErr error)

WaitCommit returns info about a specific Commit, but blocks until that commit has been finished.

func (APIClient) WaitCommitSet 

func (c APIClient) WaitCommitSet(id string, cb func(*pfs.CommitInfo) error) (retErr error)

WaitCommitSet blocks until each of a CommitSet's commits are finished, passing them to the given callback as they finish. To wait for an individual commit, use WaitCommit instead.

func (APIClient) WaitCommitSetAll 

func (c APIClient) WaitCommitSetAll(id string) (_ []*pfs.CommitInfo, retErr error)

WaitCommitSetAll blocks until all of a CommitSet's commits are finished. To wait for an individual commit, use WaitCommit instead.

func (APIClient) WaitJob 

func (c APIClient) WaitJob(projectName, pipelineName, jobID string, details bool) (_ *pps.JobInfo, retErr error)

WaitJob is a blocking version of InspectJob that will wait until the job has reached a terminal state.

func (APIClient) WaitJobSet 

func (c APIClient) WaitJobSet(id string, details bool, cb func(*pps.JobInfo) error) (retErr error)

func (APIClient) WaitJobSetAll 

func (c APIClient) WaitJobSetAll(id string, details bool) (_ []*pps.JobInfo, retErr error)

func (APIClient) WalkFile 

func (c APIClient) WalkFile(commit *pfs.Commit, path string, cb func(*pfs.FileInfo) error) (retErr error)

WalkFile walks the files under path.

func (APIClient) WithCreateFileSetClient 

func (c APIClient) WithCreateFileSetClient(cb func(ModifyFile) error) (resp *pfs.CreateFileSetResponse, retErr error)

WithCreateFileSetClient provides a scoped fileset client.

func (*APIClient) WithCtx 

func (c *APIClient) WithCtx(ctx context.Context) *APIClient

WithCtx returns a new APIClient that uses ctx for requests it sends. Note that the new APIClient will still use the authentication token and metrics metadata of this client, so this is only useful for propagating other context-associated metadata.

func (APIClient) WithDefaultTransformImage 

func (c APIClient) WithDefaultTransformImage(x string) *APIClient

WithDefaultTransformImage sets the image used when the empty string "" is passed as the image in calls to CreatePipeline*

func (APIClient) WithDefaultTransformUser 

func (c APIClient) WithDefaultTransformUser(x string) *APIClient

WithDefaultTransformUser sets the user to run the transform container as. This overrides the user set by the image.

func (APIClient) WithModifyFileClient 

func (c APIClient) WithModifyFileClient(commit *pfs.Commit, cb func(ModifyFile) error) (retErr error)

WithModifyFileClient creates a new ModifyFileClient that is scoped to the passed in callback. TODO: Context should be a parameter, not stored in the pach client.

func (APIClient) WithRenewer 

func (c APIClient) WithRenewer(cb func(context.Context, *renew.StringSet) error) error

WithRenewer provides a scoped fileset renewer.

func (APIClient) WithTransaction 

func (c APIClient) WithTransaction(txn *transaction.Transaction) *APIClient

WithTransaction (client-side) returns a new APIClient that will run supported write operations within the specified transaction.

func (APIClient) WithoutTransaction 

func (c APIClient) WithoutTransaction() *APIClient

WithoutTransaction returns a new APIClient which will run all future operations outside of any active transaction Removing from both incoming and outgoing metadata is necessary because Ctx() merges them

type AdminAPIClient 

type AdminAPIClient admin.APIClient

AdminAPIClient is an alias of admin.APIClient

type AuthAPIClient 

type AuthAPIClient auth.APIClient

AuthAPIClient is an alias of auth.APIClient

type CopyFileOption 

type CopyFileOption func(*pfs.CopyFile)

CopyFileOption configures a CopyFile call.

func WithAppendCopyFile 

func WithAppendCopyFile() CopyFileOption

WithAppendCopyFile configures the CopyFile call to append to existing files.

func WithDatumCopyFile 

func WithDatumCopyFile(datum string) CopyFileOption

WithDatumCopyFile configures the CopyFile call to apply to a particular datum.

type CreateFileSetClient 

type CreateFileSetClient struct {
	// contains filtered or unexported fields
}

CreateFileSetClient is used to create a temporary fileset.

func NewCreateFileSetClient added in v2.8.0 

func NewCreateFileSetClient(ctx context.Context, c pfs.APIClient) (_ *CreateFileSetClient, retErr error)

func (*CreateFileSetClient) Close 

func (ctfsc *CreateFileSetClient) Close() (*pfs.CreateFileSetResponse, error)

Close closes the CreateFileSetClient.

func (*CreateFileSetClient) CopyFile 

func (mfc *CreateFileSetClient) CopyFile(dst string, src *pfs.File, opts ...CopyFileOption) error

func (*CreateFileSetClient) DeleteFile 

func (mfc *CreateFileSetClient) DeleteFile(path string, opts ...DeleteFileOption) error

func (*CreateFileSetClient) PutFile 

func (mfc *CreateFileSetClient) PutFile(path string, r io.Reader, opts ...PutFileOption) error

func (*CreateFileSetClient) PutFileTAR 

func (mfc *CreateFileSetClient) PutFileTAR(r io.Reader, opts ...PutFileOption) error

func (*CreateFileSetClient) PutFileURL 

func (mfc *CreateFileSetClient) PutFileURL(path, url string, recursive bool, opts ...PutFileOption) error

type DebugClient 

type DebugClient debug.DebugClient

DebugClient is an alias of debug.DebugClient

type DeleteFileOption 

type DeleteFileOption func(*deleteFileConfig)

DeleteFileOption configures a DeleteFile call.

func WithDatumDeleteFile 

func WithDatumDeleteFile(datum string) DeleteFileOption

WithDatumDeleteFile configures the DeleteFile call to apply to a particular datum.

func WithRecursiveDeleteFile 

func WithRecursiveDeleteFile() DeleteFileOption

WithRecursiveDeleteFile configures the DeleteFile call to recursively delete the files in a directory.

type FindCommitsResponse 

type FindCommitsResponse struct {
	FoundCommits       []*pfs.Commit
	LastSearchedCommit *pfs.Commit
	CommitsSearched    uint32
}

FindCommitsResponse is a merged response of *pfs.FindCommitsResponse items that is presented to users.

type FsckOption 

type FsckOption func(*pfs.FsckRequest)

func WithZombieCheckAll 

func WithZombieCheckAll() FsckOption

func WithZombieCheckTarget 

func WithZombieCheckTarget(c *pfs.Commit) FsckOption

type GetFileOption 

type GetFileOption func(*pfs.GetFileRequest)

GetFileOption configures a GetFile call

func WithDatumGetFile 

func WithDatumGetFile(datum string) GetFileOption

WithDatumGetFile sets the datum for the get file request

func WithOffset 

func WithOffset(offset int64) GetFileOption

type IdentityAPIClient 

type IdentityAPIClient identity.APIClient

IdentityAPIClient is an alias of identity.APIClient

type LogsIter 

type LogsIter struct {
	// contains filtered or unexported fields
}

LogsIter iterates through log messages returned from pps.GetLogs. Logs can be fetched with 'Next()'. The log message received can be examined with 'Message()', and any errors can be examined with 'Err()'.

func (*LogsIter) Err 

func (l *LogsIter) Err() error

Err retrieves any errors encountered in the course of calling 'Next()'.

func (*LogsIter) Message 

func (l *LogsIter) Message() *pps.LogMessage

Message returns the most recently retrieve log message (as an annotated log line, in the form of a pps.LogMessage)

func (*LogsIter) Next 

func (l *LogsIter) Next() bool

Next retrieves the next relevant log message from pachd

type ModifyFile 

type ModifyFile interface {
	// PutFile puts a file into PFS from a reader.
	PutFile(path string, r io.Reader, opts ...PutFileOption) error
	// PutFileTAR puts a set of files into PFS from a tar stream.
	PutFileTAR(r io.Reader, opts ...PutFileOption) error
	// PutFileURL puts a file into PFS using the content found at a URL.
	// recursive allows for recursive scraping of some types of URLs.
	PutFileURL(path, url string, recursive bool, opts ...PutFileOption) error
	// DeleteFile deletes a file from PFS.
	DeleteFile(path string, opts ...DeleteFileOption) error
	// CopyFile copies a file from src to dst.
	CopyFile(dst string, src *pfs.File, opts ...CopyFileOption) error
}

ModifyFile is used for performing a stream of file modifications. The modifications are not persisted until the ModifyFileClient is closed. ModifyFileClient is not thread safe. Multiple ModifyFileClients should be used for concurrent modifications.

type ModifyFileClient 

type ModifyFileClient struct {
	// contains filtered or unexported fields
}

func NewModifyFileClient added in v2.8.0 

func NewModifyFileClient(ctx context.Context, c pfs.APIClient, commit *pfs.Commit) (_ *ModifyFileClient, retErr error)

func NewNoOpModifyFileClient 

func NewNoOpModifyFileClient() *ModifyFileClient

NewNoOpModifyFileClient returns a ModifyFileClient that does nothing; it accepts any operation and does not error.

func (*ModifyFileClient) Close 

func (mfc *ModifyFileClient) Close() error

Close closes the ModifyFileClient.

func (*ModifyFileClient) CopyFile 

func (mfc *ModifyFileClient) CopyFile(dst string, src *pfs.File, opts ...CopyFileOption) error

func (*ModifyFileClient) DeleteFile 

func (mfc *ModifyFileClient) DeleteFile(path string, opts ...DeleteFileOption) error

func (*ModifyFileClient) PutFile 

func (mfc *ModifyFileClient) PutFile(path string, r io.Reader, opts ...PutFileOption) error

func (*ModifyFileClient) PutFileTAR 

func (mfc *ModifyFileClient) PutFileTAR(r io.Reader, opts ...PutFileOption) error

func (*ModifyFileClient) PutFileURL 

func (mfc *ModifyFileClient) PutFileURL(path, url string, recursive bool, opts ...PutFileOption) error

type Option 

type Option func(*clientSettings) error

Option is a client creation option that may be passed to NewOnUserMachine(), or NewInCluster()

func WithAdditionalPachdCert 

func WithAdditionalPachdCert() Option

WithAdditionalPachdCert instructs the New* functions to additionally trust the signed cert mounted in Pachd's cert volume. This is used by Pachd when connecting to itself (if no cert is present, the clients cert pool will not be modified, so that if no other options have been passed, pachd will connect to itself over an insecure connection)

func WithAdditionalRootCAs 

func WithAdditionalRootCAs(pemBytes []byte) Option

WithAdditionalRootCAs instructs the New* functions to additionally trust the given base64-encoded, signed x509 certificates as root certificates. Introduced to pass certs in the Pachyderm config

func WithAdditionalStreamClientInterceptors 

func WithAdditionalStreamClientInterceptors(interceptors ...grpc.StreamClientInterceptor) Option

WithAdditionalStreamClientInterceptors instructs the New* functions to add the provided StreamClientInterceptors to the gRPC dial options when opening a client connection. Internally, all of the provided options are coalesced into one chain, so it is safe to provide this option more than once.

This client creates both Unary and Stream client connections, so you will probably want to supply a corresponding WithAdditionalUnaryClientInterceptors option.

func WithAdditionalUnaryClientInterceptors 

func WithAdditionalUnaryClientInterceptors(interceptors ...grpc.UnaryClientInterceptor) Option

WithAdditionalUnaryClientInterceptors instructs the New* functions to add the provided UnaryClientInterceptors to the gRPC dial options when opening a client connection. Internally, all of the provided options are coalesced into one chain, so it is safe to provide this option more than once.

This client creates both Unary and Stream client connections, so you will probably want to supply a corresponding WithAdditionalStreamClientInterceptors call.

func WithCertPool 

func WithCertPool(pool *x509.CertPool) Option

WithCertPool instructs the New* functions to create a client that uses the provided cert pool to validate the server's identity when connecting with TLS.

func WithDialTimeout 

func WithDialTimeout(t time.Duration) Option

WithDialTimeout instructs the New* functions to use 't' as the deadline to connect to pachd

func WithGZIPCompression 

func WithGZIPCompression() Option

WithGZIPCompression enabled GZIP compression for data on the wire

func WithMaxConcurrentStreams 

func WithMaxConcurrentStreams(streams int) Option

WithMaxConcurrentStreams instructs the New* functions to create client that can have at most 'streams' concurrent streams open with pachd at a time

func WithRootCAs 

func WithRootCAs(path string) Option

WithRootCAs instructs the New* functions to create client that uses the given signed x509 certificates as the trusted root certificates (instead of the system certs). Introduced to pass certs provided via command-line flags

type PfsAPIClient 

type PfsAPIClient pfs.APIClient

PfsAPIClient is an alias for pfs.APIClient.

type PortForwarder 

type PortForwarder struct {
	// contains filtered or unexported fields
}

PortForwarder handles proxying local traffic to a kubernetes pod

func NewPortForwarder 

func NewPortForwarder(context *config.Context, namespace string) (*PortForwarder, error)

NewPortForwarder creates a new port forwarder

func (*PortForwarder) Close 

func (f *PortForwarder) Close()

Close shuts down port forwarding.

func (*PortForwarder) Run 

func (f *PortForwarder) Run(appName string, localPort, remotePort uint16, selectors ...string) (uint16, error)

Run starts the port forwarder. Returns after initialization is begun with the locally bound port and any initialization errors.

func (*PortForwarder) RunForConsole 

func (f *PortForwarder) RunForConsole(localPort, remotePort uint16) (uint16, error)

RunForConsole creates a port forwarder for console

func (*PortForwarder) RunForEnterpriseServer 

func (f *PortForwarder) RunForEnterpriseServer(localPort, remotePort uint16) (uint16, error)

RunForEnterpriseServer creates a port forwarder for the enterprise server

func (*PortForwarder) RunForPachd 

func (f *PortForwarder) RunForPachd(localPort, remotePort uint16) (uint16, error)

RunForDaemon creates a port forwarder for the pachd daemon.

type PpsAPIClient 

type PpsAPIClient pps.APIClient

PpsAPIClient is an alias for pps.APIClient.

type ProxyClient 

type ProxyClient proxy.APIClient

ProxyClient is an alias of proxy.APIClient

type PutFileOption 

type PutFileOption func(*putFileConfig)

PutFileOption configures a PutFile call.

func WithAppendPutFile 

func WithAppendPutFile() PutFileOption

WithAppendPutFile configures the PutFile call to append to existing files.

func WithDatumPutFile 

func WithDatumPutFile(datum string) PutFileOption

WithDatumPutFile configures the PutFile call to apply to a particular datum.

func WithImportConcurrency 

func WithImportConcurrency(importConcurrency uint32) PutFileOption

WithImportConcurrency configures the maximum number of tasks in flight created by PutFileURL.

type TransactionAPIClient 

type TransactionAPIClient transaction.APIClient

TransactionAPIClient is an alias of transaction.APIClient

type TransactionBuilder 

type TransactionBuilder struct {
	APIClient
	// contains filtered or unexported fields
}

TransactionBuilder presents the same interface as a pachyderm APIClient, but captures requests rather than sending to the server. If a request is not supported by the transaction system, it immediately errors.

func (*TransactionBuilder) Close 

func (tb *TransactionBuilder) Close() error

Close does not exist on a TransactionBuilder because it doesn't represent ownership of a connection to the API server. We need this to shadow the inherited Close, though.

func (*TransactionBuilder) GetAddress 

func (tb *TransactionBuilder) GetAddress() *grpcutil.PachdAddress

GetAddress should not exist on a TransactionBuilder because it doesn't represent ownership of a connection to the API server, but it also doesn't return an error, so we just passthrough to the parent client's implementation.

type VersionAPIClient 

type VersionAPIClient versionpb.APIClient

VersionAPIClient is an alias of versionpb.APIClient

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.