legacyflumeoffset

Overview ¶

Package legacyflumeoffset implements the first iteration of an offset file. as implemented by https://github.com/flumedb/flumelog-offset Format

data is stored in a append only log, where the byte index of the start of a record is the primary key (offset). offset-><data.length (UInt32BE)> <data ...> <data.length (UInt32BE)> <file_length (UInt32BE or Uint48BE or Uint53BE)> by writing the length of the data both before and after each record it becomes possible to scan forward and backward (like a doubly linked list) It's very handly to be able to scan backwards, as often you want to see the last N items, and so you don't need an index for this.