Documentation
¶
Overview ¶
Package common provides functionality to read and write to a archive file and keep track of underlying *os.File to close. It also provides a handy layer to keep archive files under temporary name until they are closed and complete.
Index ¶
- func BufferSize(bufferSize int) func(*BasicArchive) error
- func Compress(c bool) func(*BasicArchive) error
- func Logger(logger *zap.Logger) func(*BasicArchive) error
- type ArchiveFileDetails
- type BasicArchive
- func (rf *BasicArchive) Close() (err error)
- func (rf *BasicArchive) FinalizedFiles() map[string]ArchiveFileDetails
- func (rf *BasicArchive) Flush() (err error)
- func (rf *BasicArchive) Name() string
- func (rf *BasicArchive) ProgressPrinter(statChan chan int64, fileName string, fileSize int64, wg *sync.WaitGroup)
- func (rf *BasicArchive) Read(p []byte) (n int, err error)
- func (rf *BasicArchive) Reset()
- func (rf *BasicArchive) Rotate() (err error)
- func (rf *BasicArchive) TrueContentLength() int64
- func (rf *BasicArchive) Write(buf []byte) (int, error)
- type FinalizerFunc
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BufferSize ¶
func BufferSize(bufferSize int) func(*BasicArchive) error
func Compress ¶
func Compress(c bool) func(*BasicArchive) error
Types ¶
type ArchiveFileDetails ¶ added in v0.1.6
type ArchiveFileDetails struct {
FileName string // Not filled when it is a checksum-only writer.
BytesWritten int64 // Always filled
ChunksWritten int64 // Always filled, but does not accurately represent rows/requests.
RowsWritten int64 // Not filled by archiver
Checksum string // Sometimes filled: only by checksum-only writer.
}
type BasicArchive ¶
type BasicArchive struct {
Logger *zap.Logger
ChunksWritten int64
Finalizer FinalizerFunc
// contains filtered or unexported fields
}
BasicArchive encapsulates some common functionality between S3Archive, FileArchive, and AZArchive
func NewBasicArchive ¶
func NewBasicArchive(stageDir, prefix, extension string, options ...func(*BasicArchive) error) (ba *BasicArchive, err error)
func OpenArchive ¶
func OpenArchive(fileName string, bufferSize int, deleteOnClose bool) (rf *BasicArchive, err error)
OpenArchive opens an archive file for reading. `*BasicArchive` returned is an io.Reader
func (*BasicArchive) Close ¶
func (rf *BasicArchive) Close() (err error)
Close works for both Read and Write. Additional logic for write to finalize the file from temp-name to final-name
func (*BasicArchive) FinalizedFiles ¶ added in v0.1.5
func (rf *BasicArchive) FinalizedFiles() map[string]ArchiveFileDetails
func (*BasicArchive) Flush ¶
func (rf *BasicArchive) Flush() (err error)
Flush complements io.Writer
func (*BasicArchive) Name ¶
func (rf *BasicArchive) Name() string
func (*BasicArchive) ProgressPrinter ¶
func (rf *BasicArchive) ProgressPrinter(statChan chan int64, fileName string, fileSize int64, wg *sync.WaitGroup)
ProgressPrinter is a helper function. Currently only usable with Azure blob upload
func (*BasicArchive) Read ¶
func (rf *BasicArchive) Read(p []byte) (n int, err error)
Read satisfies io.Reader interface - main logic is the transparent read from LZ4, Bufio, or Raw FP depending on how the file was opened
func (*BasicArchive) Reset ¶
func (rf *BasicArchive) Reset()
func (*BasicArchive) Rotate ¶
func (rf *BasicArchive) Rotate() (err error)
Rotate creates a new archive file or if one already exists, then it closes the current one and create another empty file To keep file sizes small, Rotate() must be called at regular intervals (either by size or time depending on your preference)
func (*BasicArchive) TrueContentLength ¶
func (rf *BasicArchive) TrueContentLength() int64
type FinalizerFunc ¶
type FinalizerFunc func() (finalFile ArchiveFileDetails, err error)
FinalizerFunc is a callback that will be called on Close. Typically one would rename the file to the final desired name OR upload the file to S3 or Azure Blobstore. Users of this library must provide this call back implementation.
Source Files