svfs

The Swift Virtual File System

SVFS is a Virtual File System over Openstack Swift built upon fuse. It is compatible with hubiC, OVH Public Cloud Storage and basically every endpoint using a standard Openstack Swift setup. SVFS brings a layer of abstraction over object storage, making it as accessible and convenient as a filesystem, without being intrusive on the way your data is stored. Last but not least, it can encrypt your data using AES-GCM AEAD.

Disclaimer

This is not an official project of the Openstack community.

Installation

Download and install the latest release packaged for your distribution.

Usage

You can either use standard mount conventions or use the svfs binary directly.

Using the mount command :

mount -t svfs -o username=..,password=..,tenant=..,region=..,container=.. myName /mountpoint

Using /etc/fstab :

myName   /mountpoint   svfs   username=..,password=..,tenant=..,region=..,container=..  0 0

Using svfs directly :

svfs --os-username=.. --os-password=.. ... myName /mountpoint &

Usage with OVH products

• Usage with OVH Public Cloud Storage is explained here.
• Usage with hubiC is explained here.

Options

Keystone options

• identity_url: keystone URL (default is https://auth.cloud.ovh.net/v2.0).
• username: your keystone user name.
• password: your keystone password.
• tenant: your project name.
• region: the region where your tenant is.
• version: authentication version (0 means auto-discovery which is the default).
• storage_url: the storage endpoint holding your data.
• token: a valid token.

Options region, version, storage_url and token are guessed during authentication if not provided.

Hubic options

• hubic_auth: hubic authorization token as returned by the hubic-application command.
• hubic_token : hubic refresh token as returned by the hubic-application command.

Swift options

• container: which container should be selected while mounting the filesystem. If not set, all containers within the tenant will be available under the chosen mountpoint.
• segment_size: large object segments size in MB. When an object has a content larger than this setting, it will be uploaded in multiple parts of the specified size. Default is 256 MB. Segment size should not exceed 5 GB.
• timeout: connection timeout to the swift storage endpoint. If an operation takes longer than this timeout and no data has been seen on open sockets, an error is returned. This can happen when copying non-segmented large files server-side. Default is 5 minutes.

Prefetch options

• readahead_size: Readahead size in KB. Default is 128 KB.
• readdir: Overall concurrency factor when listing segmented objects in directories (default is 20).
• extra_attr: Fetch extended attributes (default is false). Required with security options.

Cache options

• cache_access: cache entry access count before refresh. Default is -1 (unlimited access).
• cache_entries: maximum entry count in cache. Default is -1 (unlimited).
• cache_ttl: cache entry timeout before refresh. Default is 1 minute.

Ownership options

• allow_other: Bypass allow_root.
• allow_root: Restrict access to root and the user mounting the filesystem.
• default_perm: Restrict access based on file mode (useful with allow_other).
• uid: default files uid (default is 0 i.e. root).
• gid: default files gid (default is 0 i.e. root).
• mode: default files permissions (default is 0700).

Security options

• aes_key : path to a private key. Allowed private key lengths are 16, 24 and 32 bytes. Option extra_attr should also be enabled or this is an error. AES-GCM uses symetric encryption. As such, the provided key will be used for both encryption and decryption operations, ensuring a complete control of the process to the end user since absolutely noone but him has access to the key that was used to secure the data. Encrypted chunks are also authenticated using a randomized nonce. As a consequence, encrypting two indentical files results in storing two different byte streams.
• aes_block : chunk size to use while encrypting data, in KB (default is 512 KB).

Debug options

• debug: set it to true to enable debug log.
• profile_addr: Golang profiling information will be served at this address (ip:port) if set.
• profile_cpu: Golang CPU profiling information will be stored to this file if set.
• profile_ram: Golang RAM profiling information will be stored to this file if set.

Performance options

• go_gc: set garbage collection target percentage. A garbage collection is triggered when the heap size exceeds, by this rate, the remaining heap size after the previous collection. A lower value triggers frequent GC, which means memory usage will be lower at the cost of higher CPU usage. Setting a higher value will let the heap size grow by this percent without collection, reducing GC frequency. A Garbage collection is forced if none happened for 2 minutes. Note that unused heap memory is not reclaimed after collection, it is returned to the operating system only if it appears unused for 5 minutes.

Limitations

Be aware that SVFS doesn't transform object storage to block storage.

• SVFS does not support creating, moving or deleting containers.
• SVFS does not support opening a file in append mode.
• SVFS does not support moving directories.
• SVFS does not support SLO (but supports DLO).
• SVFS does not support per-file uid/gid/permissions (but per-mountpoint).

Take a look at the docs for further discussions about SVFS approach.

Hacking

Make sure to use the latest version of go and follow contribution guidelines of SVFS.

License

This work is under the BSD license, see the LICENSE file for details.