README
¶
CLI Card Importer
Command line application that downloads the latest card data set from mtgjson.com and imports it into a local postgres database.
Run locally
Database
First start a local postgres database e.g. via docker-compose up -d. This will start the database on port 15432.
Use docker-compose pull to update the image version.
Import Dataset
Run go run cmd/dataset/main.go to start the tool with the default configuration file (configs/application.yaml).
Flags:
| Flag | Usage | Default Value | Description |
|---|---|---|---|
-c,--config |
-c configs/application-prod.yaml |
configs/application.yaml | path to the configuration file |
-u,--url |
-u https://localhost/dataset.zip |
not set | dataset download url (only json and zip is supported) |
-f,--file |
-f ./dataset.json |
not set | path to local dataset json file, has precedence over the url flag or configuration file |
Import Images
Run go run cmd/images/main.go to start the tool with the default configuration file (configs/application.yaml).
Flags:
| Flag | Usage | Default Value | Description |
|---|---|---|---|
-c,--config |
-c configs/applicationLocal.yaml |
configs/application.yaml | path to the configuration file |
-p,--page |
-p 21 |
1 | start page number |
-s,--size |
-s 100 |
20 | amount of entries per page |
Serve images
Run docker run --name card-images -p 8080:80 -v $(pwd)/images:/usr/share/nginx/html:ro nginx:1.23 to make the images
accessible via nginx at localhost:8080.
Test
- Run all tests with
go test -v ./... - Run unit tests
go test -v -short ./... - Run integration tests
go test -v -run Integration ./...
Integration tests require docker to be installed.
Hint
Run the test with flag --count=1 to disable caching.
Build
Import Dataset
Build it with go build -o card-dataset-cli cmd/dataset/main.go
Import Images
Build it with go build -o card-images-cli cmd/images/main.go
Dependencies
Update all dependencies with go get -u ./.... Run go mod tidy afterwards to update and cleanup the go.mod file.
For mor information check: https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies
Misc
Docker linting
The linter Hadolint can be used to apply best practice on your Dockerfile.
Just run docker run --rm -i hadolint/hadolint < Dockerfile to check your Dockerfile.
Golang linting
The lint aggregator golangci-lint can be used to apply best practice and find errors in your golang code.
Just run docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:latest golangci-lint run -v inside the root
dir of the project to start the linting process.
TODOS
- Provide a make file
- Make import more reliable
- Retry on database connection loss
- Retry if external API returns an error
- Don't use face ids to identify images
- Faces could be deleted
- Maybe just the name?
- Use attribute side to identify card face?
First idea how to serve the images:
- Download all images via this tool
- Compress everything into one zip file
- Upload to a server to host the files
- Maybe use Google Drive (got 15GB for free and the api can be accessed via golang https://developers.google.com/drive/api/quickstart/go)
- Build container that is able to server the images from file system
- Startup process of the container downloads the hosted compressed file
- Extract all images into served dir
Directories