mal-db

module

v0.5.4 Latest Latest Go to latest Published: Sep 18, 2021 License: MIT

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/rl404/mal-db

Links

Open Source Insights

README ¶

mal-db

MyAnimeList database dump and REST API.

Powered by my go-malscraper.

Features

Save anime detail, character list, staff list, and stats.
Save manga detail, character list, and stats.
Save character detail, animeography list, mangaography list, and voice actor list.
Save people detail, anime character role list, anime staff role list, and published manga list.
Database (postgresql)
Caching (in-memory, redis, memcache)
Logging (elasticsearch)
PubSub (redis, nsq, rabbitmq)
Swagger
Docker

More will be coming soon...

Requirement

Database (postgresql)
PubSub (redis or nsq or rabbitmq)
(optional) Caching (redis or memcache)

Installation

With Go

Clone the repository.

git clone github.com/rl404/mal-db

Update .env file.
Prepare the database.

make install

Build and run worker.

make worker

In another console/terminal, build and run server.

make server

With Docker & Docker compose

Clone the repository.

git clone github.com/rl404/mal-db

Update .env file.
Prepare the database.

make docker-install

Build and run server & worker.

make docker-up

To stop containers.

make docker-stop

Config

Env are optional except the ones with *.

Env	Default	Description
`MAL_WEB_PORT`	`8006`	HTTP port
`MAL_WEB_READ_TIMEOUT`	`5`	HTTP read timeout (in seconds)
`MAL_WEB_WRITE_TIMEOUT`	`5`	HTTP write timeout (in seconds)
`MAL_WEB_GRACEFUL_TIMEOUT`	`10`	HTTP server shutdown timeout (in seconds)
`MAL_WORKER_AGE_LIMIT`	`604800`	Entry age that needs to be re-parsed (in seconds)
`MAL_WORKER_BREAK_TIME`	`5`	Break time between parsing (in seconds)
`MAL_CACHE_DIALECT`	`inmemory`	Cache type (`nocache`, `inmemory`, `redis`, `memcache`)
`MAL_CACHE_ADDRESS`		Cache address
`MAL_CACHE_PASSWORD`		Cache password
`MAL_CACHE_TIME`	`86400`	Cache time (in seconds)
`MAL_LOG_LEVEL`	`4`	Log all
`MAL_LOG_COLOR`	`true`	Log color
`MAL_DB_ADDRESS*`		Postgresql host and port
`MAL_DB_NAME`		Database name
`MAL_DB_USER`		Database username
`MAL_DB_PASSWORD`		Database password
`MAL_DB_MAX_CONN_OPEN`	`10`	Max database open connection
`MAL_DB_MAX_CONN_IDLE`	`10`	Max database idle connection
`MAL_DB_MAX_CONN_LIFETIME`	`60`	Max database connection lifetime
`MAL_ES_ADDRESS`		Elasticsearch host and port
`MAL_ES_USER`		Elasticsearch user
`MAL_ES_PASSWORD`		Elasticsearch password
`MAL_PUBSUB_DIALECT*`		PubSub type (`redis`, `nsq`, `rabbitmq`)
`MAL_PUBSUB_ADDRESS*`		PubSub address
`MAL_PUBSUB_USER`		PubSub user
`MAL_PUBSUB_PASSWORD`		PubSub password

*Required

Log Level

Level	Trace	Debug	Info	Warn	Error	Fatal
`0`	❌	❌	❌	❌	❌	❌
`1`	❌	❌	❌	❌	✔	✔
`2`	❌	❌	✔	❌	✔	✔
`3`	❌	✔	✔	✔	✔	✔
`4`	✔	✔	✔	✔	✔	✔

Disclamer

mal-db is meant for educational purpose and personal usage only. Although there is no limit in using the API, do remember that every scraper method is accessing MyAnimeList page so use it responsibly according to MyAnimeList's Terms Of Service.

All data (including anime, manga, people, etc) belong to their respective copyrights owners. mal-db does not have any affiliation with content providers.

License

MIT License

Directories ¶

Path	Synopsis
api
cmd
mal
internal
cacher
config
constant
database
errors
image
loader
loader/cacher
loader/database
loader/validator
logger
model
model/join
model/raw
pkg/http
pkg/middleware
pkg/utils
pubsub
router/api
router/image
router/ping
router/swagger
saver
tool

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Level	Trace	Debug	Info	Warn	Error	Fatal
`0`	❌	❌	❌	❌	❌	❌
`1`	❌	❌	❌	❌	✔	✔
`2`	❌	❌	✔	❌	✔	✔
`3`	❌	✔	✔	✔	✔	✔
`4`	✔	✔	✔	✔	✔	✔

Level	Trace	Debug	Info	Warn	Error	Fatal
`0`	❌	❌	❌	❌	❌	❌
`1`	❌	❌	❌	❌	✔	✔
`2`	❌	❌	✔	❌	✔	✔
`3`	❌	✔	✔	✔	✔	✔
`4`	✔	✔	✔	✔	✔	✔

Level	Trace	Debug	Info	Warn	Error	Fatal
`0`	❌	❌	❌	❌	❌	❌
`1`	❌	❌	❌	❌	✔	✔
`2`	❌	❌	✔	❌	✔	✔
`3`	❌	✔	✔	✔	✔	✔
`4`	✔	✔	✔	✔	✔	✔