luci: Index | Files

package pubsubprotocol

import ""

Package pubsubprotocol implements the LogDog pubsub wire protocol. This protocol wraps messages that are published to Cloud Pub/Sub for LogDog consumption.

The protocol begins with a set of header bytes to identify and parameterize the remaining data, followed by the data itself.

Note that the Pub/Sub layer is assumed to provide both a total length (so no need to length-prefix) and integrity (so no need to checksum).


The header is a fixed four bytes which positively identify the message as a Butler Pub/Sub message and describe the remaining data. Variant parameters can use different magic numbers to identify different parameters.

Two magic numbers are currently defined:

- 0x10 0x6d 0x06 0x00 (LogDog protocol ButlerLogBundle Raw)
- 0x10 0x6d 0x06 0x62 (LogDog protocol ButlerLogBundle GZip compressed)


The data component is described by the header, and consists of all data in the Pub/Sub message past the last header byte.


Package Files

doc.go proto.go


const (
    // DefaultCompressThreshold is the byte size threshold for compressing message
    // data. Messages whose byte count is less than or equal to this threshold
    // will not be compressed.
    // This is the value used by Akamai for its compression threshold:
    // "The reasons 860 bytes is the minimum size for compression is twofold:
    // (1) The overhead of compressing an object under 860 bytes outweighs
    // performance gain. (2) Objects under 860 bytes can be transmitted via a
    // single packet anyway, so there isn't a compelling reason to compress them."
    DefaultCompressThreshold = 860

type Reader Uses

type Reader struct {

    // Metadata is the unpacked ButlerMetadata. It is populated when the
    // metadata has been read.
    Metadata *logpb.ButlerMetadata

    // Bundle is the unpacked ButlerLogBundle. It is populated when the
    // protocol data has been read and the Metadata indicates a ButlerLogBundle
    // type.
    Bundle *logpb.ButlerLogBundle
    // contains filtered or unexported fields

Reader is a protocol reader instance.

func (*Reader) Read Uses

func (r *Reader) Read(ir io.Reader) error

type Writer Uses

type Writer struct {

    // ProtoVersion is the protocol version string to use. If empty, the current
    // ProtoVersion will be used.
    ProtoVersion string

    // Compress, if true, allows the Writer to choose to compress data when
    // applicable.
    Compress bool

    // CompressThreshold is the minimum size that data must be in order to
    CompressThreshold int
    // contains filtered or unexported fields

Writer writes Butler messages that the Reader can read.

func (*Writer) Write Uses

func (w *Writer) Write(iw io.Writer, b *logpb.ButlerLogBundle) error

WriteWith writes a ButlerLogBundle to the supplied Writer.

func (*Writer) WriteWith Uses

func (w *Writer) WriteWith(fw recordio.Writer, b *logpb.ButlerLogBundle) error

WriteWith writes a ButlerLogBundle to the supplied recordio.Writer.

Package pubsubprotocol imports 9 packages (graph) and is imported by 4 packages. Updated 2021-01-18. Refresh now. Tools for package owners.