play-with-litestream

play-with-litestream

Try Litestream setup with AWS ECS + EFS(for shared volume between containers).

SEE infrastructure/README.md for AWS CDK based implementation of resource deployment.

Prerequisites

• macOS (Not tested with other OS)

How to develop (with Docker)

# COPY env file.
cp .env.example .env

# COPY Makefile
# Replace `--profile` argument to your AWS Account's profile name.
# Then you can use these handy make commands instead of manually constructing it, eg: `make diff`, `make ecr-push`, `make deploy`
cp Makefile.example Makefile
vi Makefile

# Start s3 service
docker-compose up -d s3

# Create initial bucket
docker run --network play-with-litestream_default --rm --entrypoint bash minio/mc -c "mc alias set minio http://s3:9000 DUMMY_ROOT_USER DUMMY_ROOT_PASSWORD && mc mb minio/pwl-example-bucket"

# Start others also
docker-compose up

# Open app
open http://localhost:3000

# Try restore replica as local file by Litestream
docker-compose run ls restore -if-replica-exists -o /opt/work/db/restored-db.sqlite /opt/work/db/db.sqlite

# Try query to both server.
# Try "brew install watch" if no watch command found for macOS.
watch -n 1 curl -s http://localhost:3000
watch -n 1 curl -s http://localhost:3001

How to develop (locally)

Needs to install golang runtime.

go mod tidy

# Run main.go
go run .

# Will generate "server" binary from main.
make build

./server

Remaining issues

I found two issue(NFS related "file locking issue" and "slow performance issue") for current setup, I may try other alternative in later.

• App server and Litestream should use same locking protocol("vfs=unix" ?) How To Corrupt An SQLite Database File
  • I'm not sure about Litestream uses which locking protocol.
  • linux - Why does sqlite3 not work on Amazon Elastic File System? - Stack Overflow says use of "vfs=unix-excl" would be better. but for ECS usage, we need to allow at least Two process of container access to same "SQLite DB file" for replacement phase of ECS Service.
  • Google Cloud Run: cannot apply wal: disk I/O error: invalid argument · Issue #183 · benbjohnson/litestream Or embedding Litestream to Go app would make this problem better, but still needs to think of issue above(Two process accessing to same SQLite)
• Better alternative to use of NFS
  • "Bind mounts", means Bind ECS Instance(or Fargate)'s host directory to container, that would have better performance compared to current NFS usage.
    • For enabling that, we need to constrain task placement to same Host. otherwise each container can't share same SQLite file.
  • Current NFS usage takes around 100~150ms for single pair of "INSERT & SELECT count(*)" query. while No NFS mount(uses directory inside docker container) takes less than 1ms for same query.