gcs

package
v0.0.173 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 18, 2024 License: Apache-2.0 Imports: 32 Imported by: 11

Documentation

Overview

Package gcs provides utilities for interacting with GCS.

This includes basic CRUD operations. It is primarily focused on reading prow build results uploaded to GCS.

Index

Constants

View Source 
const (
	// DefaultACL will use the default acl for this upload.
	DefaultACL = false
	// PublicRead will use a ACL for this upload.
	PublicRead = true
	// NoCache may cache, but only after verifying.
	// See https://cloud.google.com/storage/docs/metadata#cache-control
	NoCache = "no-cache"
)
View Source 
const (
	// MissingPodInfo appears when builds complete without a podinfo.json report.
	MissingPodInfo = "podinfo.json not found in job artifacts (has not uploaded, or Prow's GCS reporter is not enabled)."
	// NoPodUtils appears when builds run without decoration.
	NoPodUtils = "Decoration is not enabled; set `decorate: true` on Prowjob."
)

Variables

This section is empty.

Functions

func ClientWithCreds 

func ClientWithCreds(ctx context.Context, creds ...string) (*storage.Client, error)

ClientWithCreds returns a storage client, optionally authenticated with the specified .json creds

func DownloadGrid added in v0.0.55 

func DownloadGrid(ctx context.Context, opener Opener, path Path) (*statepb.Grid, *storage.ReaderObjectAttrs, error)

DownloadGrid downloads and decompresses a grid from the specified path.

func IsPreconditionFailed added in v0.0.108 

func IsPreconditionFailed(err error) bool

IsPreconditionFailed returns true when the error is an http.StatusPreconditionFailed googleapi.Error.

func LeastRecentlyUpdated added in v0.0.63 

func LeastRecentlyUpdated(ctx context.Context, log logrus.FieldLogger, client Stater, paths []Path) map[Path]int64

LeastRecentlyUpdated sorts paths by their update timestamp, noting generations and any errors.

func MarshalGrid added in v0.0.70 

func MarshalGrid(grid *statepb.Grid) ([]byte, error)

MarshalGrid serializes a state proto into zlib-compressed bytes.

func Sort added in v0.0.39 

func Sort(builds []Build)

Sort the builds by monotonically decreasing original prefix base name.

In other words, 

gs://b/1
gs://a/5
gs://c/10

becomes: 

gs://c/10
gs://a/5
gs://b/1

func StatExisting added in v0.0.110 

func StatExisting(ctx context.Context, log logrus.FieldLogger, client Stater, paths ...Path) []*storage.ObjectAttrs

StatExisting reduces Stat() to an array of ObjectAttrs.

Non-existent objects will return a pointer to a zero storage.ObjectAttrs. Objects that fail to stat will be nil (and log).

func Touch added in v0.0.63 

func Touch(ctx context.Context, client ConditionalClient, path Path, generation int64, genZero []byte) (*storage.ObjectAttrs, error)

Touch attempts to win an update of the object.

Cloud copies the current object to itself when the object already exists. Otherwise uploads genZero bytes.

func Upload 

func Upload(ctx context.Context, client *storage.Client, path Path, buf []byte, worldReadable bool, cacheControl string) (*storage.ObjectAttrs, error)

Upload writes bytes to the specified Path by converting the client and path into an ObjectHandle.

func UploadHandle added in v0.0.34 

func UploadHandle(ctx context.Context, handle *storage.ObjectHandle, buf []byte, worldReadable bool, cacheControl string) (*storage.ObjectAttrs, error)

UploadHandle writes bytes to the specified ObjectHandle

Types

type Build 

type Build struct {
	Path Path
	// contains filtered or unexported fields
}

Build points to a build stored under a particular gcs prefix.

func ListBuilds 

func ListBuilds(parent context.Context, lister Lister, gcsPath Path, after *Path) ([]Build, error)

ListBuilds returns the array of builds under path, sorted in monotonically decreasing order.

func (Build) Artifacts 

func (build Build) Artifacts(ctx context.Context, lister Lister, artifacts chan<- string) error

Artifacts writes the object name of all paths under the build's artifact dir to the output channel.

func (Build) Build added in v0.0.42 

func (build Build) Build() string

Build is the unique invocation id of the job.

func (Build) Finished 

func (build Build) Finished(ctx context.Context, opener Opener) (*Finished, error)

Finished parses the build's finished metadata.

func (Build) Job added in v0.0.42 

func (build Build) Job() string

Job is the name of the job for this build

func (Build) PodInfo added in v0.0.53 

func (build Build) PodInfo(ctx context.Context, opener Opener) (*PodInfo, error)

PodInfo parses the build's pod state.

func (Build) Started 

func (build Build) Started(ctx context.Context, opener Opener) (*Started, error)

Started parses the build's started metadata.

func (Build) String 

func (build Build) String() string

func (Build) Suites 

func (build Build) Suites(ctx context.Context, opener Opener, artifacts <-chan string, suites chan<- SuitesMeta, max int) error

Suites takes a channel of artifact names, parses those representing junit suites, sending the result to the suites channel.

Truncates xml results when set to a positive number of max bytes.

type Client added in v0.0.16 

type Client interface {
	Uploader
	Downloader
	Stater
	Copier
}

A Client can upload, download and stat.

type ConditionalClient added in v0.0.34 

type ConditionalClient interface {
	Client
	// If specifies conditions on the object read from and/or written to.
	If(read, write *storage.Conditions) ConditionalClient
}

A ConditionalClient can limit actions to those matching conditions.

func NewClient added in v0.0.16 

func NewClient(client *storage.Client) ConditionalClient

NewClient returns a flexible (local or GCS) storage client.

func NewGCSClient added in v0.0.53 

func NewGCSClient(client *storage.Client) ConditionalClient

NewGCSClient returns a GCSUploadClient for the storage.Client.

func NewLocalClient added in v0.0.53 

func NewLocalClient() ConditionalClient

NewLocalClient returns a GCSUploadClient for the storage.Client.

type Copier added in v0.0.34 

type Copier interface {
	// Copy an object to the specified path
	Copy(ctx context.Context, from, to Path) (*storage.ObjectAttrs, error)
}

A Copier can cloud copy an object to a new location.

type Downloader added in v0.0.16 

type Downloader interface {
	Lister
	Opener
}

Downloader can list files and open them for reading.

type Finished 

type Finished struct {
	metadata.Finished
	// Running when the job hasn't finished and finished.json doesn't exist
	Running bool
}

Finished holds finished.json data.

type Iterator added in v0.0.16 

type Iterator interface {
	Next() (*storage.ObjectAttrs, error)
}

An Iterator returns the attributes of the listed objects or an iterator.Done error.

type Lister added in v0.0.16 

type Lister interface {
	Objects(ctx context.Context, prefix Path, delimiter, start string) Iterator
}

A Lister returns objects under a prefix.

type Opener added in v0.0.16 

type Opener interface {
	Open(ctx context.Context, path Path) (io.ReadCloser, *storage.ReaderObjectAttrs, error)
}

An Opener opens a path for reading.

type Path 

type Path struct {
	// contains filtered or unexported fields
}

Path parses gs://bucket/obj urls

func NewPath 

func NewPath(path string) (*Path, error)

NewPath returns a new Path if it parses.

func (Path) Bucket 

func (g Path) Bucket() string

Bucket returns bucket in gs://bucket/obj

func (Path) MarshalJSON added in v0.0.66 

func (g Path) MarshalJSON() ([]byte, error)

MarshalJSON encodes Path as a string

func (Path) Object 

func (g Path) Object() string

Object returns path/to/something in gs://bucket/path/to/something

func (Path) ResolveReference 

func (g Path) ResolveReference(ref *url.URL) (*Path, error)

ResolveReference returns the path relative to the current path

func (*Path) Set 

func (g *Path) Set(v string) error

Set updates value from a gs://bucket/obj string, validating errors.

func (*Path) SetURL 

func (g *Path) SetURL(u *url.URL) error

SetURL updates value to the passed in gs://bucket/obj url

func (Path) String 

func (g Path) String() string

String returns the gs://bucket/obj url

func (Path) URL added in v0.0.53 

func (g Path) URL() url.URL

URL returns the url

func (*Path) UnmarshalJSON added in v0.0.66 

func (g *Path) UnmarshalJSON(buf []byte) error

UnmarshalJSON decodes a string into Path

type PodInfo added in v0.0.53 

type PodInfo struct {
	Pod *core.Pod `json:"pod,omitempty"`
}

PodInfo holds podinfo.json (data about the pod).

func (PodInfo) Summarize added in v0.0.53 

func (pi PodInfo) Summarize() (bool, string)

Summarize returns if the pod completed successfully and a diagnostic message.

type Started 

type Started struct {
	metadata.Started
	// Pending when the job has not started yet
	Pending bool
}

Started holds started.json data.

type StatResult added in v0.0.76 

type StatResult struct {
	Attrs *storage.ObjectAttrs
	Err   error
}

StatResult contains the result of calling Stat, including any error

func Stat added in v0.0.76 

func Stat(ctx context.Context, client Stater, workers int, paths ...Path) []StatResult

Stat multiple paths using concurrent workers. Result indexes match paths.

type Stater added in v0.0.33 

type Stater interface {
	Stat(ctx context.Context, prefix Path) (*storage.ObjectAttrs, error)
}

A Stater can stat an object and get its attributes.

type SuitesMeta 

type SuitesMeta struct {
	Suites   *junit.Suites     // suites data extracted from file contents
	Metadata map[string]string // metadata extracted from path name
	Path     string
	Err      error
}

SuitesMeta holds testsuites xml and metadata from the filename

type Uploader added in v0.0.16 

type Uploader interface {
	Upload(ctx context.Context, path Path, buf []byte, public bool, cacheControl string) (*storage.ObjectAttrs, error)
}

Uploader adds upload capabilities to a GCS client.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.