Documentation
¶
Overview ¶
Package gcs handles uploading files and raw data to blob storage and determines where in the GCS bucket data should go given a specific job specification
Index ¶
- Constants
- func AliasForSpec(spec *downwardapi.JobSpec) string
- func LatestBuildForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) []string
- func LocalExport(ctx context.Context, exportDir string, uploadTargets map[string]UploadFunc) error
- func PathForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) string
- func RootForSpec(spec *downwardapi.JobSpec) string
- func Upload(ctx context.Context, bucket, gcsCredentialsFile, s3CredentialsFile string, ...) error
- func WriterOptionsFromFileName(filename string) (string, io.WriterOptions)
- type ReaderFunc
- type RepoPathBuilder
- type UploadFunc
- func DataUpload(newReader ReaderFunc) UploadFunc
- func DataUploadWithMetadata(newReader ReaderFunc, metadata map[string]string) UploadFunc
- func DataUploadWithOptions(newReader ReaderFunc, attrs pkgio.WriterOptions) UploadFunc
- func FileUpload(file string) UploadFunc
- func FileUploadWithOptions(file string, opts pkgio.WriterOptions) UploadFunc
Constants ¶
const ( // PRLogs is the name of the directory we put PR logs in. PRLogs = "pr-logs" // NonPRLogs is the name of directory we put logs that are not PR logs in. NonPRLogs = "logs" )
Variables ¶
This section is empty.
Functions ¶
func AliasForSpec ¶
func AliasForSpec(spec *downwardapi.JobSpec) string
AliasForSpec determines the GCS path aliases for a job spec
func LatestBuildForSpec ¶
func LatestBuildForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) []string
LatestBuildForSpec determines the GCS path for storing the latest build id for a job. pathSegment can be nil so callers of this helper are not required to choose a path strategy but can still get back a result.
func LocalExport ¶
LocalExport copies all of the data in the uploadTargets map to local files in parallel. The map is keyed on file path under the exportDir.
func PathForSpec ¶
func PathForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) string
PathForSpec determines the GCS path prefix for files uploaded for a specific job spec
func RootForSpec ¶
func RootForSpec(spec *downwardapi.JobSpec) string
RootForSpec determines the root GCS path for storing artifacts about the provided job.
func Upload ¶
func Upload(ctx context.Context, bucket, gcsCredentialsFile, s3CredentialsFile string, compressFileTypes []string, uploadTargets map[string]UploadFunc) error
Upload uploads all the data in the uploadTargets map to blob storage in parallel. The map is keyed on blob storage path under the bucket. Files with an extension in the compressFileTypes list will be compressed prior to uploading
func WriterOptionsFromFileName ¶
func WriterOptionsFromFileName(filename string) (string, io.WriterOptions)
WriterOptionsFromFileName guesses file attributes from the filename and returns the writerOptions and a simplified filename. For example, build-log.txt.gz would be:
Content-Type: text/plain; charset=utf-8 Content-Encoding: gzip
and the simplified filename would be build-log.txt (excluding the content encoding extension).
Types ¶
type ReaderFunc ¶
type ReaderFunc func() (io.ReadCloser, error)
type RepoPathBuilder ¶
RepoPathBuilder builds GCS path segments and embeds defaulting behavior
func NewExplicitRepoPathBuilder ¶
func NewExplicitRepoPathBuilder() RepoPathBuilder
NewExplicitRepoPathBuilder returns a builder that handles the path encoding where a path will always have an explicit "org_repo" path segment
func NewLegacyRepoPathBuilder ¶
func NewLegacyRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder
NewLegacyRepoPathBuilder returns a builder that handles the legacy path encoding where a path will only contain an org or repo if they are non-default
func NewSingleDefaultRepoPathBuilder ¶
func NewSingleDefaultRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder
NewSingleDefaultRepoPathBuilder returns a builder that handles the legacy path encoding where a path will contain org and repo for all but one default repo
type UploadFunc ¶
type UploadFunc func(writer dataWriter) error
UploadFunc knows how to upload into an object
func DataUpload ¶
func DataUpload(newReader ReaderFunc) UploadFunc
DataUpload returns an UploadFunc which copies all data from src reader into GCS.
func DataUploadWithMetadata ¶
func DataUploadWithMetadata(newReader ReaderFunc, metadata map[string]string) UploadFunc
DataUploadWithMetadata returns an UploadFunc which copies all data from src reader into GCS and also sets the provided metadata fields onto the object.
func DataUploadWithOptions ¶
func DataUploadWithOptions(newReader ReaderFunc, attrs pkgio.WriterOptions) UploadFunc
DataUploadWithOptions returns an UploadFunc which copies all data from src reader into GCS and also sets the provided attributes on the object.
func FileUpload ¶
func FileUpload(file string) UploadFunc
FileUpload returns an UploadFunc which copies all data from the file on disk to the GCS object
func FileUploadWithOptions ¶
func FileUploadWithOptions(file string, opts pkgio.WriterOptions) UploadFunc
FileUploadWithOptions returns an UploadFunc which copies all data from the file on disk into GCS object and also sets the provided attributes on the object.
Source Files