logsearchapi

Log Search API Server for MinIO

Development setup

1. Start Postgresql server in container:

docker run --rm -it -e "POSTGRES_PASSWORD=example" -p 5432:5432 postgres:13-alpine -c "log_statement=all"

1. Start logsearchapi server:

export LOGSEARCH_PG_CONN_STR="postgres://postgres:example@localhost/postgres"
export LOGSEARCH_AUDIT_AUTH_TOKEN=xxx
export MINIO_LOG_QUERY_AUTH_TOKEN=yyy
export LOGSEARCH_DISK_CAPACITY_GB=5
go build && ./logsearchapi

1. Minio setup:

mc admin config set myminio audit_webhook:1 'endpoint=http://localhost:8080/api/ingest?token=xxx'

mc admin service restart myminio

1. Sample search/list queries:

curl -v "http://localhost:8080/api/query?token=yyy&q=raw&pageNo=0&pageSize=10&timeStart=2020-11-04T22:26:12.732402319Z"

Log Storage

Logs are stored in a PostgreSQL database, partitioned such that there are four tables for each month of data. When disk usage approaches the LOGSEARCH_DISK_CAPACITY_GB value, the oldest tables are automatically deleted so as to not run out of disk space.

Raw audit logs are stored as JSON columns. These tables can be queried by specifying the query parameter q=raw.

Additionally, a set of useful request parameters are extracted from the audit logs and stored in separate tables. These tables can be queried by specifying the query parameter q=reqinfo.

API Documentation

Ingest API

POST /api/ingest?token=xxx

This API is used to send MinIO audit logs for ingestion into the API service.

The token parameter is used to authenticate the request and should be equal to the LOGSEARCH_AUDIT_AUTH_TOKEN environment variable passed to the server.

The body must be a JSON object representing a single audit log object created by a MinIO server.

This endpoint must be configured as an audit log endpoint in the MinIO server.

Query API

GET /api/query?token=xxx&...

This API is used to query MinIO audit logs stored by the API service.

The token parameter is used to authenticate the request and should be equal to the MINIO_LOG_QUERY_AUTH_TOKEN environment variable passed to the server.

Additional query parameters specify the logs to be retrieved and the format of their output.

For example, to get the last 24 hours of request-info logs dumped in line-delimited JSON format:

# If your token contains URL-unsafe characters, it must be URL-encoded appropriately as shown with curl below:
curl -XGET -s \
   'http://logsearch:8080/api/query?q=reqinfo&timeAsc&export=ndjson&last=24h' \
   --data-urlencode 'token=xxx' > output.ndjson

When using an export format (csv/json), pagination parameters (pageSize and pageNo) are not used and all data matching data is returned.

Filter Parameters

Filter parameters allow filtering records based on pattern matching on the values of audit log fields.

The format for each filter pattern is key:value-pattern.

Allowed values for the key are:

The value is a glob expression using . to signify any single character and * to match any number of characters. For example bucket:photos-* matches any bucket with a photos- prefix. To match a literal . or * prefix it with a \. To match a literal \, just double it: \\. The value pattern is case-sensitive.

curl -XGET -s \
   'http://logsearch:8080/api/query?q=reqinfo&timeAsc&export=ndjson&last=24h&fp=bucket:photos&fp=api_name:Put*' \
   --data-urlencode 'token=xxx' > output.ndjson

curl -XGET -s \
   'http://logsearch:8080/api/query?q=raw&timeAsc&pageSize=1000&last=1000h&fp=api_name:*Decom*' \
   --data-urlencode 'token=xxx'