dp-search-reindex-api

dp-search-reindex-api

Provides detail about search reindex jobs and enables creation of a new job and triggering the reindex of data for Search Service. See search service architecture docs here

Getting started

Run make help to see full list of make targets, otherwise read the following:

• Set up dependencies locally as follows:

In dp-compose repo run docker-compose up -d to run MongoDB on port 27017.

NB. The above command will also run Site Wide ElasticSearch, on port 11200, which is required by the Search API.

Run vault server -dev (this is required by Zebedee)

In the zebedee repo run ./run.sh to run Zebedee

In the dp-search-api repo set the ELASTIC_SEARCH_URL environment variable as follows (to use the Site Wide ElasticSearch):

export ELASTIC_SEARCH_URL="http://localhost:11200"

Also in the dp-search-api repo run make debug

Make sure that you have a valid local SERVICE_AUTH_TOKEN environment variable value; if not then set one up by following these instructions: https://github.com/ONSdigital/zebedee

• Then in the dp-search-reindex-api repo run make debug

Dependencies

• Requires MongoDB running on port 27017
• Requires Kafka running on port 9092
• Requires Zebedee running on port 8082
• Requires Search API running on port 23900
• No further dependencies other than those defined in go.mod

Configuration

Notes:

1. For more info, see the kafka TLS examples documentation

Testing

• Run the component tests with this command go test -component
• Run the unit tests with this command make test
• For all details of the service endpoints use a swagger editor such as this one to view the swagger specification

When running the service (see 'Getting Started') then one can use command line tool (cURL) or REST API client (e.g. Postman) to test the endpoints:

• POST: http://localhost:25700/search-reindex-jobs (should post a default job into the data store and return a JSON representation of it, should also create a new ElasticSearch index)
• GET: http://localhost:25700/search-reindex-jobs/ID NB. Use the id returned by the POST call above,e.g., http://localhost:25700/search-reindex-jobs/bc7b87de-abf5-45c5-8e3c-e2a575cab28a (should get a job from the data store)
• GET: http://localhost:25700/search-reindex-jobs (should get all the jobs from the data store)
• GET: http://localhost:25700/search-reindex-jobs?offset=1&limit=2 (should get no more than 2 jobs, from the data store, starting from index 1)
• PUT: http://localhost:25700/search-reindex-jobs/ID/number-of-tasks/10 (should put a value of 10 in the number_of_tasks field for the job with that particular id)
• GET: http://localhost:25700/health (should show the health check details)
• POST: http://localhost:25700/search-reindex-jobs/ID/tasks (should post a task into the data store and return a JSON representation of it) NB. This endpoint requires authorisation. Choose 'Bearer Token' as the authorisation type in Postman. The token to use can be found by this command echo $SERVICE_AUTH_TOKEN The endpoint also requires a body, which should contain the task name, and the number of documents e.g. { "task_name": "dataset-api", "number_of_documents": 29 }
• GET: http://localhost:25700/search-reindex-jobs/ID/tasks/TASK_NAME (should get a task from the data store)
• GET: http://localhost:25700/search-reindex-jobs/ID/tasks (should get all the tasks, for a particular job, from the data store)
• PATCH: http://localhost:25700/search-reindex-jobs/41f72483-aeee-4693-9fed-a45ebaa370f2 -H 'Authorization: Bearer fc4089e2e12937861377629b0cd96cf79298a4c5d329a2ebb96664c88df77b67' -H 'If-Match: *' -H 'Content-Type: application/json' -d '[{"op":"replace","path":"/state","value":"created"},{"op":"replace","path":"/total_search_documents","value":208},{"op":"replace","path":"/number-of-tasks","value":2}]'

Contributing

See CONTRIBUTING for details.

License

Copyright © 2022, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.