filestore

filestore: Go file storage implementations

Features

• Identify uploaded files as hashes (e.g. for reference in a database)
• Implementations
  • Local file storage (in a directory with subdirectories derived from hash prefix)
  • S3 based file storage (tested with MinIO and AWS S3)
  • Memory based file storage (for testing)

Scope

An abstracted file storage in webapps is the main use case. It is well suited to handle file uploads, retrieval and deletion. With the integration of image processing like imgproxy it provides a simple and efficient way to handle files and images in webapps. When storing files in S3 a webapp can be run stateless which simplifies container deployments.

Install

go get github.com/networkteam/filestore

Usage

Local filestore

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/networkteam/filestore/local"
)

func main() {
	ctx := context.Background()

	fStore, err := local.NewFilestore("./tmp", "./assets")
	if err != nil {
		log.Fatal(err)
	}

	// Storing
	hash, err := fStore.Store(ctx, strings.NewReader("Hello World"))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(hash)

	// Fetching
	r, err := fStore.Fetch(ctx, hash)
	if err != nil {
		log.Fatal(err)
	}
	defer r.Close()
	content, err := io.ReadAll(r)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(content))
}

S3 filestore

package main

import (
  "context"
  "fmt"
  "io"
  "log"
  "strings"

  "github.com/networkteam/filestore/s3"
)

func main() {
  ctx := context.Background()

  fStore, err := s3.NewFilestore(
    ctx,
    "s3.eu-central-1.amazonaws.com",
    "my-bucket",
    s3.WithCredentialsV4("my-access-key", "********", ""),
  )
  if err != nil {
    log.Fatal(err)
  }

  // Storing
  text := "Hello World"
  sr := strings.NewReader(text)
  // Wrap reader with s3.SizedReader to set content length in advance
  hash, err := fStore.Store(ctx, s3.SizedReader(sr, int64(len(text))))
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(hash)

  // Fetching
  r, err := fStore.Fetch(ctx, hash)
  if err != nil {
    log.Fatal(err)
  }
  defer r.Close()
  content, err := io.ReadAll(r)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(content))
}

Dependencies

The filestore module provides each implementation in its own package to reduce the amount of transitive dependencies (e.g. you don't need a S3 client if not using s3.Filestore).

Development

Tests

S3 filestore

• S3 filestore will use a fake S3 server by default
• An external S3 server and bucket can be used by setting the environment variables S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY and S3_SECRET_KEY
• Tests can be run against AWS S3 like this (make sure to allow sufficient access to bucket):

  S3_ENDPOINT=s3.eu-central-1.amazonaws.com S3_BUCKET=my-bucket-name S3_ACCESS_KEY=my-access-key S3_SECRET_KEY=******** go test
  

• Tests can be run against a MinIO server like this:

  S3_ENDPOINT=localhost:9000 S3_BUCKET=my-bucket-name S3_ACCESS_KEY=my-access-key S3_SECRET_KEY=******** go test
  

License

MIT