README
¶
Encrypted storage tools
This library implements a collection of tools to create an encrypted storage:
- Functions to encrypt and decrypt, using
AES-256, with the option to compress the data usingZLIB. - Read and write streams to create anb read encrypted files in chunks.
- Read and write streams to pack multiple small encrypted files into a single container file.
Installation
In order to add it to your project, use
go get github.com/AgustinSRG/encrypted-storage
File encryption
You can encrypt a buffer of data using EncryptFileContents, with a key of 64 bytes.
You can then decrypt it using DecryptFileContents and the same key.
Details
The encrypted data returned by EncryptFileContents and accepted by DecryptFileContents is binary-encoded, with the following structure:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
0 |
2 |
Algorithm ID | Identifier of the algorithm, stored as a Big Endian unsigned integer |
2 |
H |
Header | Header containing any parameters required by the encryption algorithm. The size depends on the algorithm used. |
2 + H |
N |
Body | Body containing the raw encrypted data. The size depends on the initial unencrypted data and algorithm used. |
The system is flexible enough to allow multiple encryption algorithms. Currently, there are 2 supported ones:
AES256_FLAT: ID =1, Uses ZLIB (RFC 1950) to compress the data, and then uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header containing the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
2 + H |
4 |
Compressed plaintext size | Size of the compressed plaintext, in bytes, used to remove padding |
2 + H + 4 |
16 |
IV | Initialization vector for AES_256_CBC algorithm |
AES256_FLAT: ID =2, Uses AES with a key of 256 bits to encrypt the data, CBC as the mode of operation and an IV of 128 bits. This algorithm uses a header containing the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
2 + H |
4 |
Plaintext size | Size of the plaintext, in bytes, used to remove padding |
2 + H + 4 |
16 |
IV | Initialization vector for AES_256_CBC algorithm |
Block-Encrypted Files
Block encrypted files are used to encrypt an arbitrarily large file, splitting it's contents in blocks (or chunks) with a set max size. Each block is then encrypted using the file encryption method detailed above.
For creating / writing files:
- You can create a file using
CreateFileBlockEncryptWriteStream, a function that returns a new instance ofFileBlockEncryptWriteStream. - After it's creation, you must call
FileBlockEncryptWriteStream.Initializeto set the file size, the block size and the encryption key. - Once it is initialized, you may call
FileBlockEncryptWriteStream.Writeto write data into the file. When the data reached a block limit, that block is encrypted and stored into the file. - After you wrote all the data, you must call
FileBlockEncryptWriteStream.Closeto close the file.
For reading files:
- You can open a file calling
CreateFileBlockEncryptReadStream, a function that returns an instance ofFileBlockEncryptReadStream - After it's opened, you may call
FileBlockEncryptReadStream.FileSize,FileBlockEncryptReadStream.BlockSizeorFileBlockEncryptReadStream.BlockCountto retrieve the parameters of the file. - You may call
FileBlockEncryptReadStream.Readto decrypt and read the data. - You can call
FileBlockEncryptReadStream.Seekto change the cursor position. You may also callFileBlockEncryptReadStream.Cursorto retrieve the cursor position if needed. - After you are done, you must call
FileBlockEncryptReadStream.Closeto close the file.
Details
They are binary files consisting of 3 contiguous sections: The header, the chunk index and the encrypted chunks.
The header contains the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
0 |
8 |
File size | Size of the original file, in bytes, stored as a Big Endian unsigned integer |
8 |
8 |
Chunk size limit | Max size of a chunk, in bytes, stored as a Big Endian unsigned integer |
After the header, the chunk index is stored. For each chunk the file was split into, the chunk index will store a metadata entry, withe the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
0 |
8 |
Chunk pointer | Starting byte of the chunk, stored as a Big Endian unsigned integer |
8 |
8 |
Chunk size | Size of the chunk, in bytes, stored as a Big Endian unsigned integer |
After the chunk index, the encrypted chunks are stored following the same structure described above.
This chunked structure allows to randomly access any point in the file as a low cost, since you don't need to decrypt the entire file, only the corresponding chunks.
Multi-File Pack
Multi-file pack container files are used to store multiple small files inside a single container.
For creating / writing files:
- You can create a file by calling
CreateMultiFilePackWriteStream, a function that returns an instance ofMultiFilePackWriteStream - You must call
MultiFilePackWriteStream.Initialize, setting the number of files you want to store - You may call
MultiFilePackWriteStream.PutFilefor each file you want to store, in order. - After all files are written, you must call
MultiFilePackWriteStream.Closeto close the file.
For reading files:
- You can open a file by calling
CreateMultiFilePackReadStream, a function that returns an instance ofMultiFilePackReadStream - You may call
MultiFilePackReadStream.FileCountto retrieve the number of stored files. - You may call
MultiFilePackReadStream.GetFileto read a file, by its index. - After you are done, you must call
MultiFilePackReadStream.Closeto close the file.
Details
They are binary files consisting of 3 contiguous sections: The header, the file table and the encrypted files.
The header contains the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
0 |
8 |
File count | Number of files stored by the asset, stored as a Big Endian unsigned integer |
After the header, a file table is stored. For each file stored by the asset, a metadata entry is stored, with the following fields:
| Starting byte | Size (bytes) | Value name | Description |
|---|---|---|---|
0 |
8 |
File data pointer | Starting byte of the file encrypted data, stored as a Big Endian unsigned integer |
8 |
8 |
File size | Size of the encrypted file, in bytes, stored as a Big Endian unsigned integer |
After the file table, each file is stored following the same structure described above.
Documentation
¶
Index ¶
- func DecryptFileContents(data []byte, key []byte) (ret_pain_text []byte, ret_error error)
- func EncryptFileContents(data []byte, method FileEncryptionMethod, key []byte) (ret_cipher_text []byte, ret_error error)
- type FileBlockEncryptReadStream
- func (file *FileBlockEncryptReadStream) BlockCount() int64
- func (file *FileBlockEncryptReadStream) BlockSize() int64
- func (file *FileBlockEncryptReadStream) Close()
- func (file *FileBlockEncryptReadStream) Cursor() int64
- func (file *FileBlockEncryptReadStream) FileSize() int64
- func (file *FileBlockEncryptReadStream) Read(buf []byte) (int, error)
- func (file *FileBlockEncryptReadStream) Seek(pos int64, whence int) (int64, error)
- type FileBlockEncryptWriteStream
- type FileEncryptionMethod
- type MultiFilePackReadStream
- type MultiFilePackWriteStream
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DecryptFileContents ¶
Decrypts file contents data - Cipher text key - Decryption key Returns the original file data, or an error
func EncryptFileContents ¶
func EncryptFileContents(data []byte, method FileEncryptionMethod, key []byte) (ret_cipher_text []byte, ret_error error)
Encrypts file contents data - File data method - algorithm to use key - Encryption key Returns the cipher text, or an error
Types ¶
type FileBlockEncryptReadStream ¶
type FileBlockEncryptReadStream struct {
// contains filtered or unexported fields
}
Status of a read stream
func CreateFileBlockEncryptReadStream ¶
func CreateFileBlockEncryptReadStream(file string, key []byte, perm fs.FileMode) (*FileBlockEncryptReadStream, error)
Creates a read stream file - Path to the file key - Decryption key perm - File mode
func (*FileBlockEncryptReadStream) BlockCount ¶
func (file *FileBlockEncryptReadStream) BlockCount() int64
Returns the block count
func (*FileBlockEncryptReadStream) BlockSize ¶
func (file *FileBlockEncryptReadStream) BlockSize() int64
Returns the block size
func (*FileBlockEncryptReadStream) Close ¶
func (file *FileBlockEncryptReadStream) Close()
Closes the read stream
func (*FileBlockEncryptReadStream) Cursor ¶
func (file *FileBlockEncryptReadStream) Cursor() int64
Returns the cursor position
func (*FileBlockEncryptReadStream) FileSize ¶
func (file *FileBlockEncryptReadStream) FileSize() int64
Returns the file size
func (*FileBlockEncryptReadStream) Read ¶
func (file *FileBlockEncryptReadStream) Read(buf []byte) (int, error)
Reads from stream, returns the amount of bytes obtained Normally reads until the buffer is full, unless the file ends buf - Buffer to fill Returns the number of bytes read
func (*FileBlockEncryptReadStream) Seek ¶
func (file *FileBlockEncryptReadStream) Seek(pos int64, whence int) (int64, error)
Moves the cursor pos - Position for the cursor to move whence - Position interpretation method. Can be 0 = absolute, 1 = Relative to current position, 2 = Relative to file end Returns the new cursor position (absolute)
type FileBlockEncryptWriteStream ¶
type FileBlockEncryptWriteStream struct {
// contains filtered or unexported fields
}
Status of a write stream
func CreateFileBlockEncryptWriteStream ¶
func CreateFileBlockEncryptWriteStream(file string, perm fs.FileMode) (*FileBlockEncryptWriteStream, error)
Creates a write stream file - Path of the file to create perm - File mode
func (*FileBlockEncryptWriteStream) Close ¶
func (file *FileBlockEncryptWriteStream) Close() error
Closes the file, writing any pending data in the buffer
func (*FileBlockEncryptWriteStream) Initialize ¶
func (file *FileBlockEncryptWriteStream) Initialize(file_size int64, block_size int64, key []byte) error
Initializes the file Must be called before any writes file_size - Size of the original file to encrypt block_size - Block size in bytes key - Encryption key
func (*FileBlockEncryptWriteStream) Write ¶
func (file *FileBlockEncryptWriteStream) Write(data []byte) error
Writes data data - Chunk of data to write
type FileEncryptionMethod ¶
type FileEncryptionMethod uint16
const ( AES256_ZIP FileEncryptionMethod = 1 // Compress data, then encrypt it with AES-256-CBC AES256_FLAT FileEncryptionMethod = 2 // Just encrypt the data with AES-256-CBC )
type MultiFilePackReadStream ¶
type MultiFilePackReadStream struct {
// contains filtered or unexported fields
}
Read stream to retrieve files from a packed file
func CreateMultiFilePackReadStream ¶
func CreateMultiFilePackReadStream(file string, perm fs.FileMode) (*MultiFilePackReadStream, error)
Creates read stream to get files from a packed file file - path to the file perm - File mode
func (*MultiFilePackReadStream) Close ¶
func (file *MultiFilePackReadStream) Close()
Closes the read stream
func (*MultiFilePackReadStream) FileCount ¶
func (file *MultiFilePackReadStream) FileCount() int64
Returns the number of files in the container
type MultiFilePackWriteStream ¶
type MultiFilePackWriteStream struct {
// contains filtered or unexported fields
}
Write stream data
func CreateMultiFilePackWriteStream ¶
func CreateMultiFilePackWriteStream(file string, perm fs.FileMode) (*MultiFilePackWriteStream, error)
Creates stream to write multiple files in a packed file file - Path of the file perm - file mode
func (*MultiFilePackWriteStream) Close ¶
func (file *MultiFilePackWriteStream) Close() error
Closes the stream
func (*MultiFilePackWriteStream) Initialize ¶
func (file *MultiFilePackWriteStream) Initialize(file_count int64) error
Initializes write stream (must be called before writing any files) file_count - Number of files to write
func (*MultiFilePackWriteStream) PutFile ¶
func (file *MultiFilePackWriteStream) PutFile(content []byte) error
Writes a file into the packed file content - Content of the file Calling this increases the current file index
Source Files