multi_container

package

v0.3.1 Latest Latest Go to latest Published: Dec 4, 2023 License: MIT Imports: 19 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/difof/goul

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
type ArchiveManager
- func NewArchiveManager(rootDir, prefix string, compQueueBufSize, compPoolSize int) (am *ArchiveManager, err error)
type Container
- func NewContainer[P generics.Ptr[RowType], RowType any](rootDir, prefix string, options ...Option) (c *Container[P, RowType], err error)
type MultiContainerFilenameParts
- func NewMultiContainerFilenamePartsFromNow(prefix string) MultiContainerFilenameParts
- func SplitMultiContainerFilename(filename string, tz *time.Location) (parts MultiContainerFilenameParts, err error)
type MultiContainerIteratorKey
type Option
type Options
- func (o *Options) LogPrintf(format string, v ...interface{})

Constants ¶

This section is empty.

Variables ¶

View Source

var ErrNoFileFound = errors.New("no file found")

Functions ¶

This section is empty.

Types ¶

type ArchiveManager ¶

type ArchiveManager struct {
	// contains filtered or unexported fields
}

ArchiveManager manages the archives of the multi container

func NewArchiveManager ¶

func NewArchiveManager(rootDir, prefix string, compQueueBufSize, compPoolSize int) (am *ArchiveManager, err error)

NewArchiveManager creates a new archive manager

func (*ArchiveManager) CleanCompress ¶

func (am *ArchiveManager) CleanCompress() error

CleanCompress does this to root dir:

Ignore files with prefix and .sbt extension if there's a compressed version
Compress any file with prefix and .sbt extension

func (*ArchiveManager) Close ¶

func (am *ArchiveManager) Close() error

Close closes the archive

func (*ArchiveManager) Files ¶

func (am *ArchiveManager) Files(ctx context.Context, start, end time.Time) chan string

Files returns the decompressed filename channel. This function panics on error, make sure to recover.

func (*ArchiveManager) QueueCompression ¶

func (am *ArchiveManager) QueueCompression(filename string)

QueueCompression queues files for compression. filename should be the full path to the file (starting at rootDir)

type Container ¶ added in v0.1.0

type Container[P generics.Ptr[RowType], RowType any] struct {
	// contains filtered or unexported fields
}

Container is a AcquireContainer wrapper allowing data insertion in multiple serial files with archive control to save space.

It use a predefined filename format for identifying files. The format is <prefix>_<date 2006-01-02-15-04>_<unix time>.sbt

func NewContainer ¶ added in v0.1.0

func NewContainer[P generics.Ptr[RowType], RowType any](
	rootDir, prefix string, options ...Option,
) (c *Container[P, RowType], err error)

NewContainer creates a new Container and loads the last container file based on the mode

func (*Container[P, RowType]) AcquireContainer ¶ added in v0.1.0

func (c *Container[P, RowType]) AcquireContainer() *sbt.Container[P, RowType]

AcquireContainer returns the current container in a thread safe way. MAKE SURE to call ReleaseContainer when done, otherwise the container will be locked forever.

func (*Container[P, RowType]) AsIterable ¶ added in v0.1.0

func (c *Container[P, RowType]) AsIterable() generics.Iterable[containers.Tuple[*MultiContainerIteratorKey, P]]

func (*Container[P, RowType]) Close ¶ added in v0.1.0

func (c *Container[P, RowType]) Close() (err error)

Close closes the current container and the archive scheduler

func (*Container[P, RowType]) Iter ¶ added in v0.1.0

func (c *Container[P, RowType]) Iter() *generics.Iterator[containers.Tuple[*MultiContainerIteratorKey, P]]

Iter will begin iteration from the very first found file

func (*Container[P, RowType]) IterHandler ¶ added in v0.1.0

func (c *Container[P, RowType]) IterHandler(
	iter *generics.Iterator[containers.Tuple[*MultiContainerIteratorKey, P]],
)

func (*Container[P, RowType]) IterRange ¶ added in v0.1.0

func (c *Container[P, RowType]) IterRange(
	start, end time.Time,
) *generics.Iterator[containers.Tuple[*MultiContainerIteratorKey, P]]

IterRange will begin iteration from the first found file and will stop at the last found file

func (*Container[P, RowType]) ReleaseContainer ¶ added in v0.1.0

func (c *Container[P, RowType]) ReleaseContainer()

ReleaseContainer releases the current container.

type MultiContainerFilenameParts ¶

type MultiContainerFilenameParts struct {
	// contains filtered or unexported fields
}

func NewMultiContainerFilenamePartsFromNow ¶

func NewMultiContainerFilenamePartsFromNow(prefix string) MultiContainerFilenameParts

NewMultiContainerFilenamePartsFromNow creates a new MultiContainerFilenameParts from now

func SplitMultiContainerFilename ¶

func SplitMultiContainerFilename(filename string, tz *time.Location) (parts MultiContainerFilenameParts, err error)

SplitMultiContainerFilename splits a filename into parts

func (MultiContainerFilenameParts) Date ¶

func (p MultiContainerFilenameParts) Date() time.Time

Date

func (MultiContainerFilenameParts) Prefix ¶

func (p MultiContainerFilenameParts) Prefix() string

Prefix

func (MultiContainerFilenameParts) String ¶

func (p MultiContainerFilenameParts) String() string

String returns the filename

func (MultiContainerFilenameParts) Unix ¶

func (p MultiContainerFilenameParts) Unix() int64

Unix

type MultiContainerIteratorKey ¶

type MultiContainerIteratorKey struct {
	Filename string
	RowId    int64
}

type Option ¶

type Option func(*Options)

func WithCompressionPoolSize ¶

func WithCompressionPoolSize(size int) Option

WithCompressionPoolSize sets the size of the compression worker pool. Defaults to NumCPU / 4.

func WithCompressionScheduler ¶

func WithCompressionScheduler(delaySec int) Option

func WithLog ¶

func WithLog(l *log.Logger) Option

func WithOpenRead ¶ added in v0.1.0

func WithOpenRead() Option

WithOpenRead sets the container to open the file for reading. Defaults to false.

type Options ¶

type Options struct {
	// contains filtered or unexported fields
}

func (*Options) LogPrintf ¶

func (o *Options) LogPrintf(format string, v ...interface{})

LogPrintf

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL