README
¶
blobby
Blobby is an open-source, distributed blob storage solution written in go. Node consensus is made possible using hashicorp's raft implementation. File data is exposed via a gRPC API.
Project status
Blobby is in active development, breaking changes may happen. It is not recommented to use blobby in production (yet). There are quite a few missing tests and the project has not been rigorously battle-tested. However, if you do intend to use blobby in its current state and run into any problems, please raise an issue
Getting started
This section assumes you have the following installed:
Installing from source
To start, clone the repository:
$ git clone git@github.com:uw-labs/blobby.git
The repository contains a Makefile with commands for compiling the server and command-line interface.
$ make build-server
$ make build-cli
From here, you can find the compiled binaries under the bin folder within the root of the project. Commands also exist for linting which use golangci-lint, enforcing a coding style.
$ make lint
You can similarly run all tests
$ make test
Starting a node
For local development on a single node, you can use the run-server recipe
$ make run-server
This will start a single node, serving gRPC traffic on port 12000 and raft traffic on port 11000.
Creating a cluster
Creating a cluster requires some orchestration, it is recommented to use the docker-compose file provided to run a local development cluster.
$ make docker-compose
In other environments, you will need to configure the cluster using environment variables/command-line flags. You can view the full list of configuration options using the --help flag when running the server binary.
# Start the first node as the leader. These
# commands assume you're running all three
# nodes on the same host.
$ server --raft-local-id=0 \
--raft-bind-addr=:11000 \
--port=12000 \
--raft-single-node \
--fs-root=data/0 \
--raft-dir=raft/0
# Create two follower nodes, on startup
# they ping the first node to join the
# cluster.
$ server --raft-local-id=1 \
--raft-bind-addr=:11001 \
--port=12001 \
--raft-join-addr=:11000 \
--fs-root=data/1 \
--raft-dir=raft/1
# Each node needs a unique local id
$ server --raft-local-id=2 \
--raft-bind-addr=:11001 \
--port=12001 \
--raft-join-addr=:11000 \
--fs-root=data/2 \
--raft-dir=raft/2
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
hack
|
|
|
load-test
Package main contains the entry point to the load-test command-line app.
|
Package main contains the entry point to the load-test command-line app. |
|
load-test/runner
Package runner contains the implementation of the load-test test runner.
|
Package runner contains the implementation of the load-test test runner. |
|
internal
|
|
|
filesystem
Package filesystem contains types and methods for interacting with the underlying file system.
|
Package filesystem contains types and methods for interacting with the underlying file system. |
|
metadata
Package metadata contains types for managing custom metadata of files in the file system.
|
Package metadata contains types for managing custom metadata of files in the file system. |
|
proto/commands
Package commands is a generated protocol buffer package.
|
Package commands is a generated protocol buffer package. |
|
proto/contract
Package contract is a generated protocol buffer package.
|
Package contract is a generated protocol buffer package. |
|
services
Package services contains gRPC service implementations.
|
Package services contains gRPC service implementations. |
|
store
Package store contains logic for managing the file system and maintaining consensus via the raft implementation.
|
Package store contains logic for managing the file system and maintaining consensus via the raft implementation. |
|
pkg
|
|
|
blobby
Package blobby contains the client implementation for the blobby file system.
|
Package blobby contains the client implementation for the blobby file system. |
|
encryptedstream
Package encryptedstream contains methods for wrapping streaming interfaces like io.Reader and io.Writer in encryption.
|
Package encryptedstream contains methods for wrapping streaming interfaces like io.Reader and io.Writer in encryption. |
|
multicontext
Package multicontext provides the ability to merge multiple context.Context implementations into a single context.Context implementation.
|
Package multicontext provides the ability to merge multiple context.Context implementations into a single context.Context implementation. |
|
operational
Package operational contains types and methods for monitoring the operations of the blobby cluster.
|
Package operational contains types and methods for monitoring the operations of the blobby cluster. |