sync-log-files-to-db

Sync Structured Log Files to a Time-series Database

Purpose

To consolidate incoming logs into a single time-series database

Readers (where sync-log-files-to-db can read logs from)

• Docker container logs
• Monitis results
• BelugaCDN real-time logs
• BelugaCDN logs in S3
• AWS CloudTrail logs in S3
• Systemd journal logs

Writers (where sync-log-files-to-db can write logs to)

• InfluxDB, a time-series database
• Google BigQuery, a serverless "big data" column store, useful for long-term archiving and large queries
• PostgreSQL database