README
¶
dp-import-cantabular-dataset
Begins the process of retrieving categories frpm Catabular API to being the population of dimensions into our datastore
Getting started
- Run
make debug
The service runs in the background consuming messages from Kafka.
An example event can be created using the helper script, make produce.
Dependencies
- Requires running…
- No further dependencies other than those defined in
go.mod
Docker-Compose
Alternatively this service can be brought up with all it's dependencies as part of the Cantabular dataset import journey with the docker-compose environment found at: https://github.com/ONSdigital/dp-compose/cantabular-import
Running Tests
Unit Tests
make test
Component Tests
make test-component
Configuration
| Environment variable | Default | Description |
|---|---|---|
| BIND_ADDR | localhost:26100 | The host and port to bind to |
| GRACEFUL_SHUTDOWN_TIMEOUT | 5s | The graceful shutdown timeout in seconds (time.Duration format) |
| HEALTHCHECK_INTERVAL | 30s | Time between self-healthchecks (time.Duration format) |
| HEALTHCHECK_CRITICAL_TIMEOUT | 90s | Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (time.Duration format) |
| KAFKA_ADDR | localhost:9092 | The kafka broker addresses (can be comma separated) |
| KAFKA_VERSION | "1.0.2" | The kafka version that this service expects to connect to |
| KAFKA_OFFSET_OLDEST | true | Start processing Kafka messages in order from the oldest in the queue |
| KAFKA_NUM_WORKERS | 1 | The maximum number of parallel kafka consumers |
| KAFKA_DATASET_INSTANCE_STARTED_GROUP | dp-import-cantabular-dataset | The consumer group this application to consume ImageUploaded messages |
| KAFKA_DATASET_INSTANCE_STARTED_TOPIC | cantabular-dataset-instance-started | The name of the topic to consume messages from |
| CANTABULAR_DATASET_CATEGORY_DIMENSION_IMPORT | cantabular-dataset-category-dimension-import | The name of the topic to produce messages to |
| DATASET_API_URL | http://localhost:22000 | HOST URL for dp-dataset-api |
| RECIPE_API_URL | http://localhost:22300 | HOST URL for dp-recipe-api |
| IMPORT_API_URL | http://localhost:21800 | HOST URL for dp-import-api |
| CANTABULAR_URL | http://localhost:8491 | HOST URL for dp-cantabular-server |
| SERVICE_AUTH_TOKEN | "" | Service auth token for authorizing requests |
| COMPONENT_TEST_USE_LOG_FILE | false | Output component test logs to temporary file instead of stdout. Used for displaying output for component tests in Concourse. |
| KAFKA_SEC_PROTO | unset | if set to TLS, kafka connections will use TLS [1] |
| KAFKA_SEC_CA_CERTS | unset | CA cert chain for the server cert [1] |
| KAFKA_SEC_CLIENT_KEY | unset | PEM for the client key [1] |
| KAFKA_SEC_CLIENT_CERT | unset | PEM for the client certificate [1] |
| KAFKA_SEC_SKIP_VERIFY | false | ignores server certificate issues if true [1] |
Notes:
1. <a name="notes_1">For more info, see the [kafka TLS examples documentation](https://github.com/ONSdigital/dp-kafka/tree/main/examples#tls)</a>
Healthcheck
The /health endpoint returns the current status of the service. Dependent services are health checked on an interval defined by the HEALTHCHECK_INTERVAL environment variable.
On a development machine a request to the health check endpoint can be made by:
curl localhost:8125/health
Contributing
See CONTRIBUTING for details.
License
Copyright © 2021, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.
Documentation
¶
There is no documentation for this package.
Source Files
Directories