fbox

fbox - Files in a Box

fbox is a distributed file system written in Go that has the the following features:

Current Features:

• Single portable binary
• Simple to setup and maintain
• A Web Interface with Drag 'n Drop
• Data redundancy with erasure coding and sharding

Planned Features:

• POSIX compatible FUSE interface
• S3 Object Storage interface
• Docker Volume Driver

There is also a publicly (freely) available demo instance available at:

• https://files.mills.io/

NOTE: I, James Mills, run this instance on pretty cheap hardware on a limited budget. Please use it fairly so everyone can enjoy using it equally!

Sponsor this project to support the development of new features, improving existings ones and fix bugs!

Table of Contents

• fbox - Files in a Box
• Table of Contents
  • Why?
  • Getting Started
    • Install from Releases
    • Install from Homebrew
    • Install from Source
  • Usage
    • Prepare storage
    • Start a master node
    • Start some data nodes
  • Production Deployments
    • Docker Swarm
  • In the News
  • Sponsor
  • Contributing
  • Contributors
  • Stargazers over time
  • Attributions and Prior Art
  • License

Created by gh-md-toc

Why?

fbox was written primarily and firstly as en education exercise in distributed file systems and a lot of time add effort went into understanding how many distributed file systems in the wild are designed.

Most other distributed file systems are either complex and hard to set up and maintain or come with really expensive license feeds. Therefore fbox is also an attempt at designing and implementing a distributed file system that is free and open source with the reliability, scale and durability of other distributed file systems.

Getting Started

Install from Releases

You can install fbox by simply downloading the latest version from the Release page for your platform and placing the binary in your $PATH.

Install from Homebrew

On macOS you can install fbox using Homebrew:

brew tap prologic/fbox https://git.mills.io/prologic/homebrew-fbox.git
brew install fbox

Install from Source

To install fbox from source you can run go get directly if you have a Go environment setup:

go get git.mills.io/prologic/fbox

NOTE: Be sure to have $GOBIN (if not empty) or your $GOPATH/bin in your $PATH. See Compile and install packages and dependencies

Or grab the source code and build:

git clone https://git.mills.io/prologic/fbox.git
cd fbox
make build

And optionally run make install to place the binary fbox in your $GOBIN or $GOPATH/bin (again see note above).

Usage

Prepare storage

mkdir data1 data2 data3

Start a master node

fbox -a 127.0.0.1:8000 -b :8000 -d ./data1

Start some data nodes

fbox -a 127.0.0.1:8001 -b :8001 -d ./data2 -m http://127.0.0.1:8000
fbox -a 127.0.0.1:8002 -b :8002 -d ./data3 -m http://127.0.0.1:8000

This will set up a 3-node cluster on your local machine (for testing only).

You can open the Web UI by navigating to:

http://127.0.0.1:8000/ui/

You can also use the command-line client fbox itself; See usage:

$ ./fbox --help
Usage: fbox [options] [command [arguments]]

fbox is a simple distributed file system...

Valid commands:
 - cat <name> -- Downloads the given file given by <name> to stdout
 - put <name> -- Upload the given file given by <name>

Valid options:
  -a, --advertise-addr string   [interface]:port to advertise
  -b, --bind string             [interface]:port to bind to (default "0.0.0.0:8000")
  -d, --data string             path to store data in (default "./data")
  -s, --data-shards int         no. of data shards (default 3)
  -D, --debug                   enable debug logging
  -m, --master string           address:port of master to join (empty if master)
  -p, --parity-shards int       no. of parity shards (default 1)
  -u, --store string            blob store uri (default "disk://data")
  -V, --version                 display version information and exit
pflag: help requested

For example to store a file:

echo "Hello World" > hello.txt
fbox -m http://127.0.0.1:8000 put hello.txt

And to retrieve the file:

fbox -m http://127.0.0.1:8000 cat hello.txt

Production Deployments

Docker Swarm

You can deploy fbox to a Docker Swarm cluster by utilising the provided fbox.yml Docker Stack. This also depends on and uses the Traefik ingress load balancer so you must also have that configured and running in your cluster appropriately.

export DOMAIN=files.yourdomain.tld
docker stack deploy -c fbox.yml fbox

In the News

Sponsor

Support the ongoing development of fbox!

Sponsor

• Become a Sponsor
• Contribute! See Issues

Contributing

Interested in contributing to this project? You are welcome! Here are some ways you can contribute:

• File an Issue -- For a bug, or interesting idea you have for a new feature or just general questions.
• Submit a Pull-Request or two! We welcome all PR(s) that improve the project!

Please see the Contributing Guidelines and checkout the Roadmap for inspiration, ideas and the general vision and goals.

Contributors

Thank you to all those that have contributed to this project, battle-tested it, used it in their own projects or products, fixed bugs, improved performance and even fix tiny typos in documentation! Thank you and keep contributing!

You can find an AUTHORS file where we keep a list of contributors to the project. If you contribute a PR please consider adding your name there. There is also Github's own Contributors statistics.

Stargazers over time

Attributions and Prior Art

• nicolagi/dino -- Some code was borrowed and much inspiration for metadata and FUSE interface design with performance in mind.
• seaweedfs -- Some inspiration taken from design and access interfaces (e.g: S3).
• GlusterFS, Portworx, Storidge, StorageOS, MooseFS -- All distributed filesystems evalauted, trialed in anger and some used to drive design goals.

License

fbox is licensed under the terms of the MIT License