README
¶
🐳 Vaar 📦
Vaala archive is a tar archive tool & library optimized for lots of small files.
Written in Golang, vaar performs operations in parallel & fully utilizes the POSIX APIs to reduce filesystem overheads.
Vaar is capable of tar creation & extraction. It works only on Linux & macOS.
Vaar is in beta. Some bugs are still out there 🙏
Install
Go 1.16+ is required to compile vaar.
As a command
go install github.com/moycat/vaar/cmd/vaar@latest
As a library
import "github.com/moycat/vaar"
CLI Usage
Suppose you have $GOPATH/bin in your PATH.
The common usage to create a tarball is:
vaar create [-c <algorithm>] [-l <level>] [-r <read_ahead>] <tarball> <file ...>
Arguments:
-c <algorithm>: Compression algorithm,lz4orgzip. No compression by default.-l <level>: Compression level,fastest,fast,default,goodorbest.-r <read_ahead>: Read ahead size, the maximum number of files to be walked and stated ahead.512by default.
Examples:
- Create a tarball with LZ4 compression:
vaar c -c lz4 archive.tar.lz4 seagrass kombu - Create a tarball with a large read ahead size:
vaar c -r 4096 archive.tar shrimps
The common usage to extract a tarball is:
vaar extract [-c <algorithm>] [-d <target>] [-s <buffer_threshold>] [-t <thread>] [-r <read_ahead>] <tarball>
Arguments:
-c <algorithm>: Compression algorithm,lz4orgzip. No compression by default.-d <target>: Extraction target path..by default.-s <buffer_threshold>: The size threshold for a file to be buffered in KiB.512by default.-t <thread>: The number of buffered extraction thread.4by default.-r <read_ahead>: Read ahead size, the maximum number of files to be extracted ahead.512by default.
Examples:
- Extract a LZ4-compressed tarball to
/tmp:vaar x -c lz4 -d /tmp archive.tar.lz4 - Extract a tarball with high concurrency:
vaar x -s 4096 -t 32 -r 2048 archive.tar
Appendix
Vaal means whale in Estonian, with Vaala being its genitive form.
Gigantic baleen whales like blue whales eat a large volume of tiny fish and shrimp with extremely high efficiency, by swallowing and filtering tons of water every time they open their mouths.
________
< Yummy! >
--------
\
\
\
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
Documentation
¶
Index ¶
- Variables
- func Resolve(r io.Reader, targetPath string, options ...Option) error
- func Walk(path string, walkFunc WalkFunc) error
- type Algorithm
- type Composer
- type Entry
- func (e *Entry) Group() string
- func (e *Entry) GroupID() uint32
- func (e *Entry) IsDir() bool
- func (e *Entry) Linkname() string
- func (e *Entry) ModTime() time.Time
- func (e *Entry) Mode() os.FileMode
- func (e *Entry) Name() string
- func (e *Entry) Owner() string
- func (e *Entry) OwnerID() uint32
- func (e *Entry) Size() int64
- func (e *Entry) Sys() interface{}
- type Level
- type Option
- type Resolver
- type WalkFunc
Constants ¶
This section is empty.
Variables ¶
Functions ¶
func Resolve ¶
Resolve takes a tarball (optionally compressed) from r and extracts it to targetPath with options. The leading slashes of the files are trimmed and path traversal is forbidden.
func Walk ¶
Walk is a walking function similar to filepath.Walk, but differs in these aspects:
- It takes WalkFunc instead of filepath.WalkFunc.
- The passed-in path must be a directory.
- The error that occurred during walking is directly returned, without passing to WalkFunc.
- Lots of magic targeting *nix systems. See the comments for details.
Types ¶
type Composer ¶
type Composer struct {
// contains filtered or unexported fields
}
Composer is a tarball creation context.
func NewComposer ¶
NewComposer creates a Composer with options, writing the tarball to w.
func (*Composer) Add ¶
Add adds a path to the tarball. The path argument is the root path of files being added. If it's a file, only the file is added. The paths of all files are trimmed from the prefix filepath.Dir(path). The base argument is prepended to the paths of all files using filepath.Join. In other words, if you call Add("/a/b/c", "d/e"), all files in /a/b/c are added as d/e/c/... including /a/b/c itself.
type Entry ¶ added in v0.0.3
type Entry struct {
// contains filtered or unexported fields
}
Entry represents a file, used in WalkFunc, specialized for tar headers.
type Option ¶
type Option func(i private) error
func WithCompression ¶
WithCompression provides the compression algorithm of tar creation and extraction.
func WithReadAhead ¶
WithReadAhead specifies the maximum number of files to be read ahead.
func WithThread ¶
WithThread specifies the worker number during extraction.
func WithThreshold ¶
WithThreshold specifies the threshold size in bytes of buffered files during extraction.
type Resolver ¶
type Resolver struct {
// contains filtered or unexported fields
}
Resolver is the context used when a tarball is being extracted. It shouldn't be used directly. Use Resolve instead.
type WalkFunc ¶
type WalkFunc func(path string, entry *Entry, r io.ReadCloser) error
WalkFunc is the type of the function called by Walk to visit each file or directory. The path argument is the absolute path of the current file. The linkName argument is the link target if this file is a symlink, otherwise empty. The entry argument is an Entry struct about this file. The r argument is the reader of this file, if it's a regular one. It must be closed if it's not nil.
Source Files
Directories