dp-cantabular-csv-exporter

command module

v1.13.0 Latest Latest Go to latest Published: Jun 15, 2023 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/ONSdigital/dp-cantabular-csv-exporter

Links

Open Source Insights

README ¶

dp-cantabular-csv-exporter

Consumes a Kafka message (cantabular-export-start) to begin the process of retrieving the counts from Cantabular.

Getting started

Dependencies

Ensure you have vault running.

brew install vault vault server -dev

Setup AWS credentials. The app uses the default provider chain. When running locally this typically means they are provided by the ~/.aws/credentials file. Alternatively you can inject the credentials via environment variables as described in the configuration section.

#### Run the service

Run make debug

The service runs in the background consuming messages from Kafka. An example event can be created using the helper script, make produce.

Dependencies

Requires running…
- kafka
No further dependencies other than those defined in go.mod

Configuration

Environment variable	Default	Description
BIND_ADDR	localhost:26300	The host and port to bind to
GRACEFUL_SHUTDOWN_TIMEOUT	5s	The graceful shutdown timeout in seconds (`time.Duration` format)
HEALTHCHECK_INTERVAL	30s	Time between self-healthchecks (`time.Duration` format)
HEALTHCHECK_CRITICAL_TIMEOUT	90s	Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (`time.Duration` format)
DEFAULT_REQUEST_TIMEOUT	10s	Default timeout for graphQL queries against Cantabular API extension and HTTP requests for the Cantabular Client only
SERVICE_AUTH_TOKEN		The service token for this app
CANTABULAR_URL	http://localhost:8491	The Cantabular server URL
CANTABULAR_EXT_API_URL	http://localhost:8492	The Cantabular API extension URL
DATASET_API_URL	http://localhost:22000	The Dataset API URL
DOWNLOAD_SERVICE_URL	http://localhost:23600	The Download Service URL, only used to generate download links
CANTABULAR_HEALTHCHECK_ENABLED	false	Flag to enable/disable healthchecks against Cantabular server and API extension
AWS_REGION	eu-west-1	The AWS region to use
PUBLIC_UPLOAD_BUCKET_NAME	public-bucket	The name of the S3 bucket to store published csv files
PRIVATE_UPLOAD_BUCKET_NAME	private-bucket	The name of the S3 bucket to store un-published csv files
VAULT_ADDR	http://localhost:8200	The address of vault
VAULT_TOKEN	-	Use `make debug` to set a vault token
VAULT_PATH	secret/shared/psk	The vault path to store psks
ENCRYPTION_DISABLED	false	Flag to enable/disable encryption for un-published csv files
STOP_CONSUMING_ON_UNHEALTHY	true	Flag to enable/disable kafka-consumer consumption depending on health status. If true, the consumer will stop consuming on 'WARNING' and 'CRITICAL' and it will start consuming on 'OK'
S3_PUBLIC_URL	http://public-bucket	The name of the S3 public url
KAFKA_ADDR	localhost:9092	The kafka broker addresses (can be comma separated)
KAFKA_CONSUMER_MIN_BROKERS_HEALTHY	1	The minimum number of healthy kafka consumer brokers
KAFKA_PRODUCER_MIN_BROKERS_HEALTHY	2	The minimum number of healthy kafka producer brokers
KAFKA_VERSION	"1.0.2"	The kafka version that this service expects to connect to
KAFKA_OFFSET_OLDEST	true	Start processing Kafka messages in order from the oldest in the queue
KAFKA_NUM_WORKERS	1	The maximum number of parallel kafka consumers
KAFKA_MAX_BYTES	2000000	the maximum number of bytes per kafka message
KAFKA_GROUP_CANTABULAR_EXPORT_START	dp-cantabular-csv-exporter	The consumer group this application to consume ImageUploaded messages
KAFKA_TOPIC_CANTABULAR_EXPORT_START	cantabular-export-start	The name of the topic to consume messages from
KAFKA_TOPIC_CSV_CREATED	cantabular-csv-created	The name of the topic that is produced after a CSV file has been successfully generated
KAFKA_SEC_PROTO	unset	if set to `TLS`, kafka connections will use TLS [1]
KAFKA_SEC_CA_CERTS	unset	CA cert chain for the server cert [1]
KAFKA_SEC_CLIENT_KEY	unset	PEM for the client key [1]
KAFKA_SEC_CLIENT_CERT	unset	PEM for the client certificate [1]
KAFKA_SEC_SKIP_VERIFY	false	ignores server certificate issues if `true` [1]

Notes:

1. <a name="notes_1">For more info, see the [kafka TLS examples documentation](https://github.com/ONSdigital/dp-kafka/tree/main/examples#tls)</a>

Healthcheck

The /health endpoint returns the current status of the service. Dependent services are health checked on an interval defined by the HEALTHCHECK_INTERVAL environment variable.

On a development machine a request to the health check endpoint can be made by:

curl localhost:8125/health