prometheus-storage-migrator

prometheus-storage-migrator

Tool to migrate Prometheus 1.x data directories to the 2.0 format.

Build

go build

Run

./prometheus-storage-migrator -v1-path=./data-old -v2-path=./data-new -lookback=24h

Flags

./prometheus-storage-migrator -h

How it works

To maximize control over memory usage when reading data from the old storage, as well as to be able to partition the series space fully and equally, this tool uses the following strategy to migrate the data:

1. A modified version of the v1 storage code is used to persist all remaining v1 head chunks to their series files. This means that the subsequent migration only needs to consider data from series files now. It does mean that the v1 storage is being modified before the migration, but the result is still a valid v1 storage (just with drained heads).
2. The tool enumerates all series files on disk and uses the archived-fingerprint-to-metrics LevelDB index of the v1 storage to look up the full label set for each file. It keeps this per-series metadata in memory for the run time of the migration.
3. The v2 storage expects data to arrive for all series in a given (short) time window, so we cannot just migrate an entire series at once and then migrate the next one. So the tool partitions the total migration range into safely appendable time windows and migrates all series at once for each window. It reads raw chunk data from the v1 series files and for each chunk, it appends all samples that fall into the current window.

Memory usage considerations

The memory requirements of the first step (persisting existing head chunks) is expected to use at most as much as the existing Prometheus 1.x server did (since that also had to keep the head chunks into memory), while the appending of data into the v2 storage is expected to require at most as much memory as the new Prometheus 2.x server. The middle step, reading v1 sample data, is what we have most control over and where we try to keep memory overhead minimal: by reading only single sample chunks at a time, and only having to keep the metadata for every series in memory. In case there are a very large number of series (many millions) in the v1 storage, keeping all metadata in memory might still be too expensive. If this becomes a problem, one optimization possibility would be to look up each series' metadata repeatedly from the v1 LevelDB index instead of keeping it in memory (an LRU cache would likely not be helpful, as we have to cycle through all series for each time step).

Contributing

Please see the contribution guidelines