curator

=================================================
``curator`` -- Artifact and Repository Management
=================================================

Overview
--------

Curator is a tool that we use at MongoDB to generate our package
repositories (e.g. ``repo.mongodb.org`` and
``repo.mongodb.com``). Additionally, curator provides tooling to
support the automated publication and management of build artifacts as
part of our continuous integration system.

Components
----------

Please refer to the ``--help`` output for ``curator`` and its
sub-commands. The following sections provides overviews of these
components and their use.

S3 Tools
~~~~~~~~

Curator includes a basic set of AWS S3 operations on the
command line. Although the get, put, and delete operations are faily
basic the ``sync-to`` and ``sync-from`` operations provide parallel
directory tree sync operations between S3 buckets and the local file
system. These operations compare objects using MD5 checksums, do
multi-threaded file uploads, and retry failed operations with
exponential backoff, for efficient and robust transfers.

Repobuilder
~~~~~~~~~~~

The repobuilder is a helper package to submit Barque jobs to build RPM
and DEB package repositories in S3. These jobs: sync files from an
existing repository, add packages from the local filesystem to the
repository, sign packages (for RPM), regenerate package metadata, sign
package metadata, generate html pages for web-based display, and sync
the changed files to the remote repository.

The current implementation of the repobuilder process depends on
external repository generation tools (e.g. ``createrepo`` and
``apt`` tools.) Additionally, the repobuilder currently depends on
MongoDB's internal signing service.

Artifacts
~~~~~~~~~

The artifacts functionality uses the release metadata feeds
(e.g. ``https://downloads.mongodb.org/full.json``) to fetch and
extract release build artifacts for local use. It is particularly
useful for fetching artifacts for and maintaining local caches of
MongoDB builds for multiple releases. Set the
``CURATOR_ARTIFACTS_DIRECTORY`` environment variable or pass the
``--path`` option to a flag, and then use the ``curator artifacts
download`` command to download files.

The ``artifacts`` command also includes two exploration subcommands
for discovering available builds: Use the ``list-map`` for specific
lists of available edition, target, and architecture combinations and
``list-all`` for a list of available target and architectures. Both
list operations are specific to a single version.

Combine the artifacts tool with the prune tool to avoid unbounded
cache growth.

Prune
~~~~~

Prune is based on the `lru <https://github.com/evergreen-ci/lru>`_
library, and takes a file system tree and removes files, based on age,
until the total size of the files is less than a specified maximum
size. Prune uses modified time for age, in an attempt to have
consistent behavior indepenent of operation system and file system.

There are two modes of operation, a recursive mode which removes
objects from the tree recursively, but skips directory objects, and
directory mode, which does not collect objects recursively, but tracks
the size for the contents--recursively--of top-level directories.

Development
-----------

Design Principles and Goals
~~~~~~~~~~~~~~~~~~~~~~~~~~~

- All operations in the continuous integration environment should be
  easily reproducible outside of the environment. In practice, curator
  exists to build repositories inside of Evergreen tasks; however, it
  is possible to run all stages of the repository building process by
  hand. The manual abilities are useful and required for publishing
  package revisions and repairing corrupt repositories.

- Leverage, as possible, third party libraries and tools. For example,
  the cache pruning and artifact management functionality is entirely
  derived from third-party repositories maintained separately from
  curator.

- Major functionality is implemented and executed in terms of `amboy
  <https://github.com/mongodb/amboy>`_ jobs and queues. Not only does
  this provide a framework for task execution, but leaves the door
  open to provide curator functionality as a highly available service
  with minimal modification.

APIs and Documentation
~~~~~~~~~~~~~~~~~~~~~~

See the `godoc API documentation <http://godoc.org/github.com/mongodb/curator>`_
for more information about curator interfaces and internals.

Issues
~~~~~~

Please file all issues in the `EVG project
<https://jira.mongodb.org/browse/EVG>`_ in the `MongoDB Jira
<https://jira.mongodb.org/>`_ instance.