README
¶
shade
shade (the SHA Drive Engine) stores files in the cloud, in a flexible fashion, optionally encrypted.
The primary interface is a FUSE filesystem for interacting with shade. There is also a command line "dropbox" tool which can cheaply add new files to shade, but cannot read them. There is also a shadeutil command line debugging tool for investigating the contents.
The basic method of file storage
- Represent the file as a series of chunks, of a configurable size (16MB by default).
- Calculate a SHA-256 hash for each chunk.
- Store the chunk in 1 or more Drive clients
- Create a manifest file (a shade.File struct) with:
- Filename
- Chunk size
- Indexed list of chunks
- Calculate a SHA-256 hash of the manifest.
- Store the shade.File in 1 or more Drive implementations (just like Chunk, but retrievable separately).
Retrieving a file works much the same, just in reverse:
- Download all of the manifest files.
- Find the filename which matches the request.
- If necessary, decrypt the chunk(s).
shade/drive Drive interface
The Drive interface provides a way to store and retrieve two separate buckets of bytes, Files and Chunks, each identified by their sha256sum. It also provides a way to list the sha256sum of all known Files.
The interface also provides a bit of metadata about the implementation, such as a name for identifying it, if it stores files persistently and/or remotely, and a way to retrieve the configuration that intialized the implementation.
drive.Drive implementations
There are several implementations of drive.Drive clients. Some are only for testing (eg. drive/win, drive/fail), some are for local caching (drive/memory, drive/local), and some are for remote/cloud storage (drive/amazon, drive/google). There are a few special implementations which allow you to combine (drive/cache) or augment (drive/encrypt) the other implementations.
These can be combined in novel ways by the config package. Trust your local machine? You can create a config which will encrypt only the bytes the leave your machine and go to a remote provider. Want to always encrypt bytes at rest? You can build a config which will encrypt even the local disk storage, but still cache all File objects unencrypted in memory for more efficient reads.
Encryption overview
The drive/encrypt module will encrypt writes to its child client. It will AES-256 encrypt the chunked contents of the files, the File objects that describe the metadata, and even the sha256sums of the chunks. It then RSA encrypts the AES-256 key and stores the encrypted key with the File object.
RSA public and private keypairs are provided via the config package. It is supported to provide only a public RSA key pair. This is useful with cmd/throw/throw.go a "write only" tool which cannot read back any of the data once it is writen.
For additional details on the implementation, see the godoc for the drive/encrypt module.
NB: Encrypting the contents stored in Drive comes with two penalties:
- Modest CPU usage to encrypt/decrypt on the way in/out.
- The chunks of identical files will not be deduplicated.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConfigDir ¶
func ConfigDir() string
ConfigDir identifies the correct path to store persistent configuration data on various operating systems.
func NewNonce ¶
func NewNonce() []byte
NewNonce generates a random Nonce for AES-GCM. It panics if the source of randomness fails.
func NewSymmetricKey ¶
func NewSymmetricKey() *[32]byte
NewSymmetricKey generates a random 256-bit AES key for File{}s. It panics if the source of randomness fails.
Types ¶
type Chunk ¶
type Chunk struct {
Index int
Sha256 []byte
Nonce []byte // If encrypted, use this Nonce to store/retrieve the Sum.
}
Chunk represents a portion of the content of the File being stored.
type File ¶
type File struct {
// Filename is a fully qualified path, with no leading slash.
Filename string
Filesize int64 // Bytes
// ModifiedTime represents the "commit" time of this File object. A given
// Filename is represented by the valid File with the latest ModifiedTime.
ModifiedTime time.Time
// Chunks represets an ordered list of the bytes in the file.
Chunks []Chunk
// Chunksize is the maximum size of each plaintext Chunk, in bytes.
Chunksize int
// LastChunksize is the size of the last chunk in the File. Storing this
// explicity avoids the need to fetch the last chunk to update the Filesize.
LastChunksize int
// Deleted indicates all previous versions of this file should be suppressed.
Deleted bool
// AesKey is a 256 bit key used to encrypt the Chunks with AES-GCM. If no
// key is provided, the blocks are not encrypted. The GCM nonce is stored at
// the front of the encrypted Chunk using gcm.Seal(); use gcm.Open() to
// recover the Nonce when decrypting. Nb: This increases the encrypted
// Chunk's size by gcm.NonceSize(), currently 12 bytes.
AesKey *[32]byte
}
File represents the metadata of a file stored in Shade. It is stored and retrieved by the drive.Client API, and boiled down
func (*File) FromJSON ¶
FromJSON populates the fields of this File struct from a JSON representation. It primarily provides a convenient error message if this fails.
func (*File) UpdateFilesize ¶
func (f *File) UpdateFilesize()
UpdateFilesize calculates the size of the assocaited Chunks and sets the Filesize member of the struct.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
shade
shade presents a fuse filesystem interface.
|
shade presents a fuse filesystem interface. |
|
shadeutil
shadeutil contains tools for inspecting shade repositories.
|
shadeutil contains tools for inspecting shade repositories. |
|
throw
throw stores a file in the cloud, encrypted.
|
throw stores a file in the cloud, encrypted. |
|
cache
Package cache is an interface to multiple storage backends for Shade.
|
Package cache is an interface to multiple storage backends for Shade. |
|
encrypt
Package encrypt is an interface to manage encrypted storage backends.
|
Package encrypt is an interface to manage encrypted storage backends. |
|
fail
Package fail is a test client.
|
Package fail is a test client. |
|
local
Package local is a persistent local storage backend for Shade.
|
Package local is a persistent local storage backend for Shade. |
|
memory
Package memory is an in memory storage backend for Shade.
|
Package memory is an in memory storage backend for Shade. |
|
win
Package win is a test client.
|
Package win is a test client. |