dp-filter-api

module

v0.0.0-...-87f5b26 Latest Latest Go to latest Published: Jul 31, 2023 License: MIT

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/ONSdigital/dp-filter-api

README ¶

dp-filter-api

Getting started

mongo

Run brew install mongodb
Run brew services start mongodb

Kafka

Run brew install zookeeper
Download kafka version 0.10.2.1
Run ./kafka-server-start.sh ../config/server.properties

Scripts for updating and debugging Kafka can be found here(dp-data-tools)

Configuration

Run brew install neo4j
Disable authentication in the config
Run brew services start neo4j

Environment variable	Default	Description
BIND_ADDR	:22100	The host and port to bind to
HOST	http://localhost:22100	The host name used to build URLs
KAFKA_ADDR	localhost:9092	The kafka broker addresses (can be comma separated)
KAFKA_MAX_BYTES	2000000	The maximum permitted size of a message. Should be set equal to or smaller than the broker's `message.max.bytes`
KAFKA_SEC_PROTO	unset	if set to `TLS`, kafka connections will use TLS [1]
KAFKA_SEC_CLIENT_KEY	unset	PEM for the client key [1]
KAFKA_SEC_CLIENT_CERT	unset	PEM for the client certificate [1]
KAFKA_SEC_CA_CERTS	unset	CA cert chain for the server cert [1]
KAFKA_SEC_SKIP_VERIFY	false	ignores server certificate issues if `true` [1]
FILTER_JOB_SUBMITTED_TOPIC	filter-job-submitted	The kafka topic to write messages to
MONGODB_BIND_ADDR	localhost:27017	The MongoDB bind address
MONGODB_USERNAME		The MongoDB Username
MONGODB_PASSWORD		The MongoDB Password
MONGODB_DATABASE	filters	The MongoDB database
MONGODB_COLLECTIONS	FiltersCollection:filters, OutputsCollection:filterOutputs	The MongoDB collections
MONGODB_REPLICA_SET		The name of the MongoDB replica set
MONGODB_ENABLE_READ_CONCERN	false	Switch to use (or not) majority read concern
MONGODB_ENABLE_WRITE_CONCERN	true	Switch to use (or not) majority write concern
MONGODB_CONNECT_TIMEOUT	5s	The timeout when connecting to MongoDB (`time.Duration` format)
MONGODB_QUERY_TIMEOUT	15s	The timeout for querying MongoDB (`time.Duration` format)
MONGODB_IS_SSL	false	Switch to use (or not) TLS when connecting to mongodb
SHUTDOWN_TIMEOUT	5s	The graceful shutdown timeout (`time.Duration` format)
DATASET_API_URL	http://localhost:22000	The URL of the Dataset API
HEALTHCHECK_INTERVAL	30s	Time between self-healthchecks (`time.Duration` format)
HEALTHCHECK_CRITICAL_TIMEOUT	90s	The time taken for the health changes from warning state to critical due to subsystem check failures
SERVICE_AUTH_TOKEN	FD0108EA-825D-411C-9B1D-41EF7727F465	The token used to identify this service when authenticating
ZEBEDEE_URL	"http://localhost:8082"	Zebedee URL
ENABLE_PRIVATE_ENDPOINTS	false	true if private endpoints should be enabled
DOWNLOAD_SERVICE_URL	http://localhost:23600	The URL of the download service
DOWNLOAD_SERVICE_SECRET_KEY	QB0108EZ-825D-412C-9B1D-41EF7747F462	The service token for the download service

Notes:

For more info, see the kafka TLS examples documentation

Healthchecking

Currently checked each $HEALTHCHECK_INTERVAL and reported on endpoint /healthcheck:

Neo4j
MongoDB
Dataset API

Acceptance tests

To run the acceptance tests for this project, use the following commands

Start Kafka and Mongodb
Run the dp-filter-api using make acceptance
Run the tests in dp-api-test

Contributing

See CONTRIBUTING for details.

License

Released under MIT license, see LICENSE for details.

Directories ¶

Path	Synopsis
api
mock
cmd
dp-filter-api
config
filterOutputQueue
filters
kafkaadapter
middleware
mock
models
mongo
schema
service
mock
utils

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL