record

Overview ¶

Package record reads and writes sequences of records. Each record is a stream of bytes that completes before the next record starts.

When reading, call Next to obtain an io.Reader for the next record. Next will return io.EOF when there are no more records. It is valid to call Next without reading the current record to exhaustion.

When writing, call Next to obtain an io.Writer for the next record. Calling Next finishes the current record. Call Close to finish the final record.

Optionally, call Flush to finish the current record and flush the underlying writer without starting a new record. To start a new record after flushing, call Next.

Neither Readers or Writers are safe to use concurrently.

Example code:

func read(r io.Reader) ([]string, error) {
	var ss []string
	records := record.NewReader(r)
	for {
		rec, err := records.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Printf("recovering from %v", err)
			r.Recover()
			continue
		}
		s, err := ioutil.ReadAll(rec)
		if err != nil {
			log.Printf("recovering from %v", err)
			r.Recover()
			continue
		}
		ss = append(ss, string(s))
	}
	return ss, nil
}

func write(w io.Writer, ss []string) error {
	records := record.NewWriter(w)
	for _, s := range ss {
		rec, err := records.Next()
		if err != nil {
			return err
		}
		if _, err := rec.Write([]byte(s)), err != nil {
			return err
		}
	}
	return records.Close()
}

The wire format is that the stream is divided into 32KiB blocks, and each block contains a number of tightly packed chunks. Chunks cannot cross block boundaries. The last block may be shorter than 32 KiB. Any unused bytes in a block must be zero.

A record maps to one or more chunks. There are two chunk formats: legacy and recyclable. The legacy chunk format:

+----------+-----------+-----------+--- ... ---+
| CRC (4B) | Size (2B) | Type (1B) | Payload   |
+----------+-----------+-----------+--- ... ---+

CRC is computed over the type and payload Size is the length of the payload in bytes Type is the chunk type

There are four chunk types: whether the chunk is the full record, or the first, middle or last chunk of a multi-chunk record. A multi-chunk record has one first chunk, zero or more middle chunks, and one last chunk.

The recyclyable chunk format is similar to the legacy format, but extends the chunk header with an additional log number field. This allows reuse (recycling) of log files which can provide significantly better performance when syncing frequently as it avoids needing to update the file metadata. Additionally, recycling log files is a prequisite for using direct IO with log writing. The recyclyable format is:

+----------+-----------+-----------+----------------+--- ... ---+
| CRC (4B) | Size (2B) | Type (1B) | Log number (4B)| Payload   |
+----------+-----------+-----------+----------------+--- ... ---+

Recyclable chunks are distinguished from legacy chunks by the addition of 4 extra "recyclable" chunk types that map directly to the legacy chunk types (i.e. full, first, middle, last). The CRC is computed over the type, log number, and payload.

The wire format allows for limited recovery in the face of data corruption: on a format error (such as a checksum mismatch), the reader moves to the next block and looks for the next full or first chunk.