archiver

package

v0.0.0-...-570c91f Latest Latest Go to latest Published: Dec 19, 2023 License: MIT Imports: 15 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/penguin-statistics/backend-next

Links

Open Source Insights

Documentation ¶

Constants ¶

View Source

const (
	FileExtJsonlGzip       = ".jsonl.gz"
	FileExtParquet         = ".parquet"
	LocalTempDirPattern    = "penguin_stats-archiver-*"
	ArchiverChanBufferSize = 1000
)

Variables ¶

View Source

var ErrFileAlreadyExists = errors.New("file already exists")

Functions ¶

This section is empty.

Types ¶

type Archiver ¶

type Archiver struct {
	S3Client *s3.Client
	S3Bucket string

	// S3Prefix is for the files in the bucket with no leading slash but optionally (typically) with trailing slash
	// e.g. "v1/" or simply "" (empty string)
	S3Prefix string

	RealmName string
	// contains filtered or unexported fields
}

func (*Archiver) Cleanup ¶

func (a *Archiver) Cleanup() error

func (*Archiver) Collect ¶

func (a *Archiver) Collect(ctx context.Context) error

Caller MUST use WriterCh() to get the channel and ensure necessary data is sent to the channel before calling this function. Moreover, caller should ensure that Collect runs only once and runs on a different goroutine from the one that sends data to the channel to avoid deadlocks.

func (*Archiver) Prepare ¶

func (a *Archiver) Prepare(ctx context.Context, date time.Time) error

func (*Archiver) WriterCh ¶

func (a *Archiver) WriterCh() chan interface{}

Caller MUST close the channel when it's done

Source Files ¶

View all Source files

archiver.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL