README
¶
reedlib
A high-level Reed Solomon erasure encoding library and set of command-line tools written in Go.
Quick Start
Installing the library
Either run:
go get git.mills.io/prologic/reedlib
Or just import the library in your project:
import "git.mills.io/prologic/reedlib"
NOTE: There isn't much of a library yet here at the time of writing this (2020-12-27). Coming soon...
Installing the command-line tools
Install reed-encode:
go get git.mills.io/prologic/reedlib/cmd/reed-encode/....
Install reed-decode:
go get git.mills.io/prologic/reedlib/cmd/reed-decode/....
Usage
Encoding a file (command-line)
To encode a file using Reed-Solomon erasure encoding using the default Data and Parity Shards (3 + 1 respectively):
reed-encode ./testdata/IMG_7895.JPG
This will result in a number of output files from the result of splitting up and encoding the original input file using the specified number of data and parity shards:
$ ls -lah ./testdata/
total 6.4M
drwxr-xr-x 8 prologic staff 256 Dec 27 09:10 .
drwxr-xr-x 17 prologic staff 544 Dec 27 09:12 ..
-rw-r--r-- 1 prologic staff 6.1K Dec 27 09:10 .DS_Store
-rw-r--r-- 1 prologic staff 2.7M Dec 27 09:10 IMG_7895.JPG
-rw-r--r-- 1 prologic staff 916K Dec 27 09:10 IMG_7895.JPG.0
-rw-r--r-- 1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.1
-rw-r--r-- 1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.2
-rw-r--r-- 1 prologic staff 916K Dec 27 08:55 IMG_7895.JPG.3
Keep the pieces (data and parity shards) on different storage devices or storage nodes which can then later be used to reconstruct the original file, even if one of the shards is lost or corrupt (parity of one).
Decoding shards (command-line)
To decode a number of shards from a previous encoding using the default Data and Parity Shards (3 + 1 respectively):
First remove the original input file to demonstrate recovery:
rm -f ./testdata/IMG_7895.JPG
Now reconstruct the original input file using the shards:
reed-decode ./testdata/IMG_7895.JPG
You should now have the original file recovered and intact:
$ ls -lah ./testdata/
total 6.3M
drwxr-xr-x 7 prologic staff 224 Dec 27 09:33 .
drwxr-xr-x 15 prologic staff 480 Dec 27 09:33 ..
-rw-r--r-- 1 prologic staff 2.7M Dec 27 09:33 IMG_7895.JPG
-rw-r--r-- 1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.0
-rw-r--r-- 1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.1
-rw-r--r-- 1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.2
-rw-r--r-- 1 prologic staff 916K Dec 27 09:33 IMG_7895.JPG.3
You can even remove the original input file and either remove or corrupt one of the shards and the original input file is still recoverable from the remaining shards.
Notes
This is a high-level wrapper library and set of command-line tools that uses @klauspost's reedsolomon library which itself is a Go port of the JavaReedSolomon library released by Backblaze, with some additional optimizations.
For an introduction on erasure coding, see the post on the Backblaze blog.
Some interesting properties to note:
- The number of
data+parityshards MUST NOT exceed 256 as the default bit-field size is 8 for Reed Solomon erasure codes. - You can detect errors or recover from errors from up to
parityshards. For example, with a data/parity of3+1you can recover from 1 lost or corrupt shard. With a data/parity of3+2you can recover from up to two shard failures.
License
reedlib is licensed under the terms of the MIT License
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // Version release version Version = "0.0.1" // Commit will be overwritten automatically by the build system Commit = "HEAD" )
Functions ¶
Types ¶
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface.
func (*Encoder) Encode ¶
Encode splits data into equal-length shards, some containing the original data and some containing parity data.
func (*Encoder) EncodeShards ¶
EncodeShards creates the parity shards for an already sharded input.
func (*Encoder) MinShards ¶
MinShards return the minimum number of shards that must be present to recover the original data.
type ErasureCoder ¶
type ErasureCoder interface {
// NumShards is the number of shards returned by Encode.
NumShards() int
// MinShards is the minimum number of shards that must be present to
// recover the original data.
MinShards() int
// Encode splits data into equal-length shards, with some shards
// containing parity data.
Encode(data []byte) ([][]byte, error)
// EncodeShards encodes the input data like Encode but accepts an already
// sharded input.
EncodeShards(data [][]byte, shardSize uint64) ([][]byte, error)
// Recover recovers the original data from shards and writes it to w.
// shards should be identical to the slice returned by Encode (length and
// order must be preserved), but with missing elements set to nil. n is
// the number of bytes to be written to w; this is necessary because
// shards may have been padded with zeros during encoding.
Recover(shards [][]byte, n uint64, w io.Writer) error
}
ErasureCoder is an error-correcting encoder and decoder.
func NewEncoder ¶
func NewEncoder(dataShards, parityShards int) (ErasureCoder, error)
NewEncoder creates a new Reed-Solomon encoder/decoder using the supplied parameters.
Source Files
Directories