README
¶
Chunky
Chunky is a high performance, deduplicated, object store. It is designed to be used as a backend for efficiently sharing machine learning datasets and weights between machines while minimizing network bandwidth.
Chunky is in a very early stage of development. It is not ready for general production use.
Features
- FastCDC content defined chunking.
- LZ4 transit compression.
- HighwayHash based content hashing.
- Offline network bypass mode.
Links
License
This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtai one at https://mozilla.org/MPL/2.0/.
For more information see the LICENSE file.
Documentation
¶
Index ¶
Constants ¶
const ( // DefaultChunkSize is the default chunk size (1 MB). DefaultChunkSize = 1000000 // DefaultBufferSize is the number of chunks to buffer in memory. DefaultBufferSize = 4 // DefaultNetworkThreads is the number of connections to use for network ops. DefaultNetworkThreads = 4 // DefaultRetryAttempts is the number of times to retry a failed operation. DefaultRetryAttempts = 3 )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ChunkClient ¶
type ChunkClient struct {
// contains filtered or unexported fields
}
ChunkClient is a client for fetching and pushing chunks.
func NewChunkClient ¶
NewChunkClient creates a new chunk client.
func (*ChunkClient) Fetch ¶
func (c *ChunkClient) Fetch(ctx context.Context, id contenthash.Hash) ([]byte, error)
Fetch retrieves a chunk from the server.
func (*ChunkClient) Push ¶
func (c *ChunkClient) Push(ctx context.Context, id contenthash.Hash, data []byte) error
Push pushes a chunk to the server.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is an object storage client.
func NewClient ¶
func NewClient(grpcObjectClient pb.ObjectClient, chunkClient *ChunkClient, cache storage.Storage) *Client
NewClient creates a new object storage client.
type FetchOpts ¶
type FetchOpts struct {
// NetworkThreads is the number of parallel threads to use for fetching chunks.
NetworkThreads *int
// BufferSize is the maximum number of chunks to buffer in memory.
// Larger values will result in greater throughput, but will use more memory.
BufferSize *int
// RetryAttempts is the number of times to retry fetching a chunk before giving up.
RetryAttempts *uint
}
FetchOpts is used to configure the behaviour of fetching an object.
type PushOpts ¶
type PushOpts struct {
// ChunkSize is the nominal size of each chunk.
ChunkSize int64
// NetworkThreads is the number of parallel threads to use for pushing chunks.
NetworkThreads *int
// RetryAttempts is the number of times to retry pushing a chunk before giving up.
RetryAttempts *uint
}
PushOpts is used to configure the behaviour of pushing an object.
Source Files
Directories