README
¶
dp-dataset-exporter
Takes a filter job and produces a filtered dataset.
Getting started
Ensure you have vault running.
brew install vault
vault server -dev
- Setup AWS credentials. The app uses the default provider chain. When running locally this typically means they are provided by the
~/.aws/credentialsfile. Alternatively you can inject the credentials via environment variables as described in the configuration section - Run
make debug - Run the auth-stub-api
Kafka scripts
Scripts for updating and debugging Kafka can be found here(dp-data-tools)
Configuration
An overview of the configuration options available, either as a table of environment variables, or with a link to a configuration guide.
| Environment variable | Default (example) | Description |
|---|---|---|
| BIND_ADDR | :22500 | The host and port to bind to |
| KAFKA_ADDR | localhost:9092 |
The address of (TLS-ready) Kafka brokers (comma-separated values) |
| KAFKA_VERSION | 1.0.2 |
The version of (TLS-ready) Kafka |
| KAFKA_SEC_PROTO | unset (only TLS) |
if set to TLS, kafka connections will use TLS |
| KAFKA_SEC_CLIENT_KEY | unset | PEM [2] for the client key (optional, used for client auth) [1] |
| KAFKA_SEC_CLIENT_CERT | unset | PEM [2] for the client certificate (optional, used for client auth) [1] |
| KAFKA_SEC_CA_CERTS | unset | PEM [2] of CA cert chain if using private CA for the server cert [1] |
| KAFKA_SEC_SKIP_VERIFY | false | ignore server certificate issues if set to true [1] |
| FILTER_JOB_CONSUMER_TOPIC | filter-job-submitted | The name of the topic to consume messages from |
| FILTER_JOB_CONSUMER_GROUP | dp-dataset-exporter | The consumer group this application to consume filter job messages |
| DATASET_API_URL | http://localhost:22000 | The URL of the dataset API |
| DATASET_API_AUTH_TOKEN | FD0108EA-825D-411C-9B1D-41EF7727F465 | The auth token for the dataset API |
| FILTER_API_URL | http://localhost:22100 | The URL of the filter API |
| FILTER_API_AUTH_TOKEN | FD0108EA-825D-411C-9B1D-41EF7727F465 | The auth token for the filter API |
| AWS_REGION | eu-west-1 | The AWS region to use |
| S3_BUCKET_URL | unset (e.g. https://cf.host) |
If set, the URL prefix for public, exported downloads |
| S3_BUCKET_NAME | csv-exported | The name of the public S3 bucket to store exported files |
| S3_PRIVATE_BUCKET_NAME | csv-exported | The name of the private s3 bucket to store exported files |
| CSV_EXPORTED_PRODUCER_TOPIC | common-output-created | The topic to add messages to when a job is complete |
| ERROR_PRODUCER_TOPIC | filter-error | The topic to add messages to when an error occurs |
| HEALTHCHECK_INTERVAL | 30s | The time between doing health checks |
| HEALTHCHECK_CRITICAL_TIMEOUT | 90s | The time taken for the health changes from warning state to critical due to subsystem check failures |
| GRACEFUL_SHUTDOWN_TIMEOUT | time.Second * 10 | How long to wait for the service to shutdown gracefully |
| DOWNLOAD_SERVICE_URL | http://localhost:23600 | The URL of the download service |
| VAULT_ADDR | http://localhost:8200 | The address of vault |
| VAULT_TOKEN | - | Use make debug to set a vault token |
| VAULT_PATH | secret/shared/psk | The vault path to store psks |
| SERVICE_AUTH_TOKEN | 0f49d57b-c551-4d33-af1e-a442801dd851 | The service token for this app |
| ZEBEDEE_URL | http://localhost:8082 | The URL to zebedee |
| AWS_ACCESS_KEY_ID | - | The AWS access key credential |
| AWS_SECRET_ACCESS_KEY | - | The AWS secret key credential |
| FULL_DATASET_FILE_PREFIX | full-datasets | The prefix added to full dataset download files |
| FILTERED_DATASET_FILE_PREFIX | filtered-dataset | The prefix added to filtered dataset download files |
Notes:
-
Ignored unless using TLS (i.e.
KAFKA_SEC_PROTOhas a value enabling TLS) -
PEM values are identified as those starting with
-----BEGINand can use\n(sic) instead of newlines (they will be converted to newlines before use). Any other value will be treated as a path to the given PEM file.
Healthcheck
The /health endpoint returns the current status of the service. Dependent services are health checked on an interval defined by the HEALTHCHECK_INTERVAL environment variable.
On a development machine a request to the health check endpoint can be made by:
curl localhost:22500/health
Contributing
See CONTRIBUTING for details.
License
Copyright © 2016-2019, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.
Directories