s3

package
v0.0.0-...-eef0557 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2021 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewS3FileReader

func NewS3FileReader(ctx context.Context, bucket string, key string, cfgs ...*aws.Config) (source.ParquetFile, error)

NewS3FileReader creates an S3 FileReader, to be used with NewParquetReader

func NewS3FileReaderWithClient

func NewS3FileReaderWithClient(ctx context.Context, s3Client s3iface.S3API, bucket string, key string) (source.ParquetFile, error)

NewS3FileReaderWithClient is the same as NewS3FileReader but allows passing your own S3 client

func NewS3FileWriter

func NewS3FileWriter(
	ctx context.Context,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
	cfgs ...*aws.Config,
) (source.ParquetFile, error)

NewS3FileWriter creates an S3 FileWriter, to be used with NewParquetWriter

func NewS3FileWriterWithClient

func NewS3FileWriterWithClient(
	ctx context.Context,
	s3Client s3iface.S3API,
	bucket string,
	key string,
	acl string,
	uploaderOptions []func(*s3manager.Uploader),
) (source.ParquetFile, error)

NewS3FileWriterWithClient is the same as NewS3FileWriter but allows passing your own S3 client.

func SetActiveSession

func SetActiveSession(sess *session.Session)

SetActiveSession sets the current session. If this is unset, the functions of this package will implicitly create a new session.Session and use that. This allows you to control what session is used, particularly useful for testing with a system like localstack.

Types

type S3File

type S3File struct {
	BucketName string
	Key        string
	ACL        string
	// contains filtered or unexported fields
}

S3File is ParquetFile for AWS S3

func (*S3File) Close

func (s *S3File) Close() error

Close signals write completion and cleans up any open streams. Will block until pending uploads are complete.

func (*S3File) Create

func (s *S3File) Create(key string) (source.ParquetFile, error)

Create creates a new S3 File instance to perform writes

func (*S3File) Open

func (s *S3File) Open(name string) (source.ParquetFile, error)

Open creates a new S3 File instance to perform concurrent reads

func (*S3File) Read

func (s *S3File) Read(p []byte) (n int, err error)

Read up to len(p) bytes into p and return the number of bytes read

func (*S3File) Seek

func (s *S3File) Seek(offset int64, whence int) (int64, error)

Seek tracks the offset for the next Read. Has no effect on Write.

func (*S3File) Write

func (s *S3File) Write(p []byte) (n int, err error)

Write len(p) bytes from p to the S3 data stream

Directories

Path Synopsis
Package mocks is a generated GoMock package.
Package mocks is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL