Documentation
¶
Index ¶
- Variables
- func GCSSourceFactory(c stiface.Client) factory.SourceFactory
- func GetStorageClient(writeAccess bool) (stiface.Client, error)
- func NewRowWriter(ctx context.Context, client stiface.Client, bucket string, path string) (row.Sink, error)
- func NewSinkFactory(client stiface.Client, outputBucket string) factory.SinkFactory
- func NewTestSource(client stiface.Client, dp etl.DataPath, label string) (etl.TestSource, error)
- func ObjectWriter(ctx context.Context, client stiface.Client, bucket string, path string) stiface.Writer
- type Closer
- type GCSSource
- type RowWriter
- type SinkFactory
- type TarReader
Constants ¶
This section is empty.
Variables ¶
var ErrOversizeFile = errors.New("Oversize file")
ErrOversizeFile is returned when exceptionally large files are skipped.
Functions ¶
func GCSSourceFactory ¶
func GCSSourceFactory(c stiface.Client) factory.SourceFactory
GCSSourceFactory returns the default SourceFactory
func GetStorageClient ¶
GetStorageClient provides a storage reader client. This contacts the backend server, so should be used infrequently.
func NewRowWriter ¶
func NewRowWriter(ctx context.Context, client stiface.Client, bucket string, path string) (row.Sink, error)
NewRowWriter creates a RowWriter.
func NewSinkFactory ¶
func NewSinkFactory(client stiface.Client, outputBucket string) factory.SinkFactory
NewSinkFactory returns the default SinkFactory
func NewTestSource ¶
NewTestSource creates an TestSource suitable for injecting into Task. Caller is responsible for calling Close on the returned object.
uri should be of form gs://bucket/filename.tar or gs://bucket/filename.tgz FYI Using a persistent client saves about 80 msec, and 220 allocs, totalling 70kB.
Types ¶
type Closer ¶
type Closer struct {
// contains filtered or unexported fields
}
Closer handles gzip files.
type GCSSource ¶
type GCSSource struct {
FilePath string
Size int64
TarReader // TarReader interface provided by an embedded struct.
io.Closer // Closer interface to be provided by an embedded struct.
RetryBaseTime time.Duration // The base time for backoff and retry.
TableBase string // TableBase is BQ table associated with this source, or "invalid".
PathDate civil.Date // Date associated with YYYY/MM/DD in FilePath.
}
GCSSource wraps a gsutil tar file containing tests.
type RowWriter ¶
type RowWriter struct {
// contains filtered or unexported fields
}
RowWriter implements row.Sink to a GCS file backend.
func (*RowWriter) Commit ¶
Commit commits rows, in order, to the GCS object. The GCS object is not available until Close is called, at which point the entire object becomes available atomically. The returned int is the number of rows written (and pending), or, if error is not nil, an estimate of the number of rows written.
type SinkFactory ¶
type SinkFactory struct {
// contains filtered or unexported fields
}
SinkFactory implements factory.SinkFactory.
Source Files