gcfs

Go-Couchbase File Server (GCFS)

GCFS allows easy metadata management for file servers.

About

GCFS is a tool for building cloud file systems. It provides default Couchbase utilities to store data about your files, located on a distant server.

GCFS is the link between your UI and the distant file. It will provide client adequate information for retrieving any files stored on your custom server.

GCFS Usage

You can use GCFS in two modes : api mode and methods mode. The first gives you more easy setups, while the second aims for deeper integration in your server application.

You can use both modes together, depending on your configuration.

Basic configuration

To use GCFS, you can provide an optional configuration file, in JSON format. Pass the path to the configuration file in your ENV variables.

From your terminal:

export GCFS_CONFIG="/path/to/my/configuration/file"

It is recommended to pass absolute path to the file, and not relative one.

No parameter is required. You can run GCFS without any config file, though you'll have to comply to default values.

Full example (with default values)

{
  "database": {
    "address": "127.0.0.1",
    "username": "",
    "password": "",
    "bucket_name": "metadata"
  },
  "global": {
    "debug": false,
    "auto_provide": false
  },
  "server": {
    "port": "8080",
    "logs": {
      "default": "",
      "error": ""
    }
  },
  "routes": {
    "ping": "",
    "ping_database": "",
    "insert": "",
    "delete": "",
    "get": "",
    "update": "",
    "search": ""
  },
  "metadata": {}
}

Database

Couchbase server configuration.

Global

Server

Server configuration.

Routes

If any route is set, gcfs will create a pre-configured route for the specified action. You don't have to set every route, only the ones you need.

Metadata (params)

An additional set of metadata to check on insertion. See the Metadata section for more details.

Api mode

Api mode let you define pre-configured routes for your application. Just provide a route with the correct url parameters when needed, and GCFS will handle all the work for you.

💡 Tip : url parameter in go is given between curly braces. /url/path/{url_parameter}/etc.

Methods mode

Methods mode is the default mode of gcfs. You can always use it, even when api mode is set up.

Methods mode provides you some functions to interact directly with Couchbase. Thus, you have full control of your api behavior.

Insert

fileId, err := gcfs_methods.Insert(fileMetadata)

Insert metadata and returns the id of the generated tuple.

arguments

return value

Delete

err := gcfs_methods.Delete(fileId)

Delete metadata from database. Only returns an error.

arguments

return value

Get

fileMetadata, err := gcfs_methods.Get(fileId)

Retrieve metadata from database.

arguments

return value

Search

queryResults, err := gcfs_methods.Search(start, end, options)

Search function using nefts.

arguments

(1) Config.Cluster and Config.Bucket options are overridden by gcfs configuration.

return value

Update

timestamp, err := methods.Update(fileId, fileMetadata)

Partial update of a document.

arguments

return value

Partial metadata

GCFS provides an easy, declarative way to perform a partial update of your metadata, using gocb MutateIn function.

partialMetadata := gcfs_methods.UpdateSpec{
    Remove: []string{"key1", "key2", "key4.key4-2"},
    Upsert: map[string]interface{}{
        "key3": "new value",
        "key4": map[string]interface{}{
            "key4-1": "new value"
        }
    },
    Append: map[string]interface{}{
        "array-key": ["newValue1", "newValue2"]
    }
}

A list of keys to remove from the document. Use the dot syntax for nested keys.

Update a list of values. Each value in the given map will replace the equivalent one in the original document. If a path doesn't exist, it will be created.

Append a list of values to an array key.

💡 Tip : you can declare a non array value to append a single value. An array value will always be treaten with the HasMultiple flag set to true (see gocb specs). Add a nested array if you want to append an array as an array, and not a list of values.

AutoProvide

autoProvided, err = gcfs.metadata.AutoProvide(fileMetadata)

Auto fill metadata with default values. More details in the Metadata section.

CheckIntegrity

coreMetadata, err := gcfs.metadata.CheckIntegrity(fileMetadata)

Check integrity of a dataset.

💡 Tip : coreMetadata represents the default and required metadata. More details in the Metadata section.

Metadata

Core metadata

Metadata is a configurable JSON object that will link file to the client. Some metadata is required for video creation.

Required metadata is provided below. You don't need to set them in config file, but you have to provide the ones marked as required (you can provide the others, or let gcfs fill them by default).

Custom metadata

Additionally, you can add a set of custom REQUIRED metadata. All data sent by your client will be uploaded to database. This parameter just serve to enforce some data to be present on insertion.

Custom metadata are set of key-value pairs. Key is the metadata key, and value its required type. Use interface for undefined type.

{
  "metadata": {
    "labels": "[]string",
    "stats": {
      "download_count": "uint64",
      "status": "string"
    }
  }
}

💡 Tip : types are string representing a go type. Custom types from your application are supported, however you'll need to prefix it with your package name, and run your code in go module mode.

Error handling

NEFTS returns a pointer to an error object adapted to web servers.

Copyright

2020 Alvarios - MIT license