Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GetFilesSince ¶
func GetFilesSince(ctx context.Context, sClient stiface.Client, prefix string, filter *regexp.Regexp, since time.Time) ([]*storage.ObjectAttrs, int64, error)
GetFilesSince gets list of all storage objects with prefix, created or updated since given date. TODO - similar to code in etl-gardener/cloud/tq/tq.go. Should move to go/cloud/gcs
Types ¶
type Prefix ¶
type Prefix struct {
Bucket string // the GCS bucket name.
Experiment string // the experiment name
DataType string // if empty, this is legacy, and DataType is same as Experiment
DatePath string // the YYYY/MM/DD date path.
Date time.Time // the time.Time corresponding to the datepath.
}
Prefix is a valid gs:// prefix for either legacy or new platform data.
func ParsePrefix ¶
ParsePrefix Parses prefix, returning {bucket, experiment, date string}, error Unless it returns error, the result will be exactly length 3.
Source Files
Click to show internal directories.
Click to hide internal directories.