Documentation ¶
Overview ¶
Package for working with IPFIX messages. Supports decoding and encoding from and to IPFIX according to RFC 7011.
Overview ¶
IPFIX message format is defined in RFC 7011. go-ipfix implements decoding and encoding of messages adhering to that RFC. Additionally, most other major IPFIX RFCs are also supported feature-wise, namely
- RFC 5103: Bidirectional Flow Export Using IP Flow Information Export (IPFIX)
- RFC 5610: Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements
- RFC 5655: Specification of the IP Flow Information Export (IPFIX) File Format
- RFC 6313: Export of Structured Data in IP Flow Information Export (IPFIX)
Below are some examples of how some common use-cases of this library may look like.
Historical Background ¶
This library was factored out of a 2023 master's thesis' codebase for working with IPFIX flow records. The ipfix package works on its own, and was used for implementing collectors and further processing tools for IPFIX flow records, in particular in combination with enterprise-specific information elements for proprietary flow information, and structured data types, which are prominent when using yaf (https://tools.netsa.cert.org/yaf/) in combination with DPI information.
When factoring out, the Decode API was overhauled to mirror Go's io.Reader style. Additionally, some high-level types such as TemplateSet, DataSet, and OptionsTemplateSet were cleaned up.
Also, TCP and UDP listeners were not originally part of the ipfix module. While UDP is much simpler to implement as one UDP packet generally corresponds to one IPFIX message, and such a listener is easy to implement using Go's rich net.PacketConn tooling, IPFIX's prescription of how collection via a _single long-lived_ TCP connection works is a bit more involved and we decided to include our contribution of such a TCP listener in this package. The below example shows how to use it, and in particular shows how the TCP and UDP listener API is identical.
Data Structures ¶
IPFIX messages are nested to contain as much possible data in a single message as possible. On a top-level, an IPFIX message contains 3 kinds of typed sets of records, namely template sets, data sets, and options template sets. These sets have a header-like data structure at the start denoting the "ID" of the set. In the case of template records, the ID is expected to be 2, for options template sets it is 3, and data sets are freely taken from 255 up to 65535. The IDs of data sets are associated with IDs defined in previously sent template records.
Each set contains one or more records of the same type, i.e., template records, data records, options template records.
Each record contains one or more fields, whose semantics are defined either in the standard registry 0 of IANA-IPFIX, or vendor-specific/proprietary are Information Elements.
Values of such fields are typed according to RFC 7011 or RFC 6313 (which defines higher-level data types). In go-ipfix, all those data types implement the DataType interface. For more details on the data types consult the corresponding RFCs.
Notably, RFC 6313 defines mechanisms for (recursively) nesting records in data records. The SubTemplateList and SubTemplateMultiList data types allow for nesting data records created from *possibly different* templates than the containing data set, thus creating a tree-like structure, where records can be nested up to a certain extent (not exceeding the maximum message size of 2^16-1 bytes minus a couple of bytes for headers). Note that these message sizes are also bound by the transport protocol of your choice: via TCP, this is no problem, but considering that UDP packet fragmentation due to IP MTUs might cause the loss of entire packets, make sure that you pick a protocol that ensures reliable message transfer when sending IPFIX messages that exceed ~1500 Bytes of payload.
The coupling of templates and records (read: data semantics are detached from the actual data) requires a stateful management of the templates such that incoming data records can be decoded if and only if the corresponding template was received before. RFC 7011 prescribes that decoders (collectors) may store records not able to decode due to absense of the template up to a certain point if e.g. asymmetric paths create such a race condition. go-ipfix *does not* implement such a system, but the error returned from the Decoder if a template is not known can be used to queue up such messages and work off that queue at any later point.
Example (CollectorTCP) ¶
Collect IPFIX messages via TCP listener. The code's layout is idiomatic of the package: Consume messages in a goroutine from a channel and use a Decoder instance to create message objects to work with. The example code simply logs the messages to Stdout. Other use-cases might be forwarding these objects in a stateless format, e.g., JSON or Protobuf, to a message queue, such as Kafka (this is what similar Go libraries such as goflow2 and vflow) do.
var ( BindAddr string = "[::]:4739" ) ctx, cancel := context.WithCancel(context.TODO()) defer cancel() c := make(chan os.Signal, 2) signal.Notify(c, os.Interrupt, syscall.SIGTERM) go func() { <-c log.Println("Received shutdown signal, initiating shutdown...") cancel() <-c os.Exit(1) }() tcpListener := ipfix.NewTCPListener(BindAddr) go func() { log.Printf("Starting TCP listener for IPFIX messages on %s", BindAddr) tcpListener.Listen(ctx) }() templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) go func() { for { select { case raw := <-tcpListener.Messages(): msg, err := decoder.Decode(ctx, bytes.NewBuffer(raw)) if err != nil { log.Println(fmt.Errorf("failed to decode IPFIX message: %w", err)) } log.Println(msg) case <-ctx.Done(): return } } }() <-ctx.Done()
Output:
Example (CollectorUDP) ¶
Collect IPFIX messages via UDP listener. The example is exactly the same as the TCP example, except for the transport protocol used. For more description see the TCP collector example.
var ( BindAddr string = "[::]:4739" ) ctx, cancel := context.WithCancel(context.TODO()) defer cancel() c := make(chan os.Signal, 2) signal.Notify(c, os.Interrupt, syscall.SIGTERM) go func() { <-c log.Println("Received shutdown signal, initiating shutdown...") cancel() <-c os.Exit(1) }() tcpListener := ipfix.NewUDPListener(BindAddr) go func() { log.Printf("Starting UDP listener for IPFIX messages on %s", BindAddr) tcpListener.Listen(ctx) }() templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) go func() { for { select { case raw := <-tcpListener.Messages(): msg, err := decoder.Decode(ctx, bytes.NewBuffer(raw)) if err != nil { log.Println(fmt.Errorf("failed to decode IPFIX message: %w", err)) } log.Println(msg) case <-ctx.Done(): return } } }() <-ctx.Done()
Output:
Example (Decoder) ¶
A simple decoder of IPFIX messages read from a file. The example uses the IPFIXFileReader, which asserts the file must contain IPFIX messages according to RFC 5655. The decoder is fed with messages from the reader the same way we previously did in the TCP and UDP listener example, using a indefinitely running goroutine with message channels.
ctx, cancel := context.WithCancel(context.Background()) defer cancel() f, _ := os.Open("demo_flow_records.ipfix") defer f.Close() r := ipfix.NewIPFIXFileReader(f) go r.Start(ctx) templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) go func() { for { select { case raw := <-r.Messages(): msg, err := decoder.Decode(ctx, bytes.NewBuffer(raw)) if err != nil { log.Println(fmt.Errorf("failed to decode IPFIX message: %w", err)) } log.Println(msg) case err := <-r.Errors(): log.Println(fmt.Errorf("failed to read IPFIX message: %w", err)) case <-ctx.Done(): return } } }() <-ctx.Done()
Output:
Example (IpfixFileReader) ¶
A simple decoder of IPFIX messages read from a file. The example uses the IPFIXFileReader, which asserts the file must contain IPFIX messages according to RFC 5655.
ctx, cancel := context.WithCancel(context.Background()) defer cancel() f, _ := os.Open("demo_flow_records.ipfix") r := ipfix.NewIPFIXFileReader(f) go r.Start(ctx) templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) go func() { for { select { case raw := <-r.Messages(): msg, err := decoder.Decode(ctx, bytes.NewBuffer(raw)) if err != nil { log.Println(fmt.Errorf("failed to decode IPFIX message: %w", err)) } log.Println(msg) case err := <-r.Errors(): log.Println(fmt.Errorf("failed to read IPFIX message: %w", err)) case <-ctx.Done(): return } } }() <-ctx.Done()
Output:
Example (ReadFull) ¶
f, _ := os.Open("demo_flow_records.ipfix") messages, err := ipfix.ReadFull(f) if err != nil { log.Fatalln(err) } templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) for _, rawMessage := range messages { msg, err := decoder.Decode(context.TODO(), bytes.NewBuffer(rawMessage)) if err != nil { log.Fatalln(err) } log.Println(msg) }
Output:
Example (TransformerNormalizeRecords) ¶
Transforms IPFIX messages containing more than one record and template set per message into a stream of messages that only contain one record in one typed set per message. Note that while this obviously includes a lot of overhead, it is helpful in scenarios where we want _individual records_ to end up in a hypothetical database, because queries and filters are much easier implemented, and the grouping of records in homogeneous sets is mostly only done for reducing message overhead from redundancy. In practice, due to the possibly complex timing of flow records, sets often only contain single records anyways and rarely do messages contain more than one set. Software exporters such as yaf behave differently in terms of message packing when writing to files or sending via TCP or UDP. To ease this difference, normalization appears reasonable.
package main import ( "bytes" "context" "fmt" "log" "os" "github.com/zoomoid/go-ipfix" ) // Transforms IPFIX messages containing more than one record and template set per message into // a stream of messages that only contain one record in one typed set per message. // Note that while this obviously includes a lot of overhead, it is helpful in scenarios where // we want _individual records_ to end up in a hypothetical database, because queries and filters // are much easier implemented, and the grouping of records in homogeneous sets is mostly only // done for reducing message overhead from redundancy. // In practice, due to the possibly complex timing of flow records, sets often only contain single // records anyways and rarely do messages contain more than one set. // Software exporters such as yaf behave differently in terms of message packing when writing // to files or sending via TCP or UDP. To ease this difference, normalization appears reasonable. func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() in, _ := os.Open("demo_flow_records.ipfix") defer in.Close() out, _ := os.CreateTemp("", "normalized_flow_records.ipfix") r := ipfix.NewIPFIXFileReader(in) go r.Start(ctx) templateCache := ipfix.NewDefaultEphemeralCache() fieldCache := ipfix.NewEphemeralFieldCache(templateCache) decoder := ipfix.NewDecoder(templateCache, fieldCache, ipfix.DecoderOptions{OmitRFC5610Records: false}) go func() { for { select { case raw := <-r.Messages(): msg, err := decoder.Decode(ctx, bytes.NewBuffer(raw)) if err != nil { log.Fatalln(fmt.Errorf("failed to decode IPFIX message: %w", err)) } normalizedMessages, err := NormalizeIPFIXMessage(msg) if err != nil { log.Fatalln(fmt.Errorf("failed to normalize IPFIX message: %w", err)) } for _, newMsg := range normalizedMessages { _, err := newMsg.Encode(out) if err != nil { log.Fatalln(fmt.Errorf("failed to encode IPFIX message: %w", err)) } } case err := <-r.Errors(): log.Println(fmt.Errorf("failed to read IPFIX message: %w", err)) case <-ctx.Done(): return } } }() <-ctx.Done() } const ( ipfixPacketHeaderLength int = 16 ipfixSetHeaderLength int = 4 ) var ( sequenceNumber uint32 = 0 ) func NormalizeIPFIXMessage(old *ipfix.Message) (new []*ipfix.Message, err error) { new = make([]*ipfix.Message, 0) for _, fs := range old.Sets { switch fss := fs.Set.(type) { case *ipfix.TemplateSet: for _, rr := range fss.Records { flow := &bytes.Buffer{} n, err := rr.Encode(flow) // we use this to determine the NEW set length! if err != nil { return nil, err // skip entire packet } pp := &ipfix.Message{ Version: 10, ExportTime: old.ExportTime, SequenceNumber: uint32(sequenceNumber), // this needs to be rewritten! ObservationDomainId: old.ObservationDomainId, Length: uint16(n + ipfixPacketHeaderLength + ipfixSetHeaderLength), Sets: []ipfix.Set{ { SetHeader: ipfix.SetHeader{ Id: fs.Id, Length: uint16(n + ipfixSetHeaderLength), // single record length + set header length }, Set: &ipfix.TemplateSet{ Records: []ipfix.TemplateRecord{rr}, }, }, }, } // sequenceNumber++ - RFC 7011: "Template and Options Template Records do not increase the Sequence Number." new = append(new, pp) } case *ipfix.OptionsTemplateSet: for _, rr := range fss.Records { flow := &bytes.Buffer{} n, err := rr.Encode(flow) // we use this to determine the NEW set length! if err != nil { return nil, err // skip entire packet } pp := &ipfix.Message{ Version: 10, ExportTime: old.ExportTime, SequenceNumber: uint32(sequenceNumber), // this needs to be rewritten! ObservationDomainId: old.ObservationDomainId, Length: uint16(n + ipfixPacketHeaderLength + ipfixSetHeaderLength), Sets: []ipfix.Set{ { SetHeader: ipfix.SetHeader{ Id: fs.Id, Length: uint16(n + ipfixSetHeaderLength), // single record length + set header length }, Set: &ipfix.OptionsTemplateSet{ Records: []ipfix.OptionsTemplateRecord{rr}, }, }, }, } // sequenceNumber++ - RFC 7011: "Template and Options Template Records do not increase the Sequence Number." new = append(new, pp) // recordCounter++ } case *ipfix.DataSet: for _, rr := range fss.Records { flow := &bytes.Buffer{} n, err := rr.Encode(flow) // we use this to determine the *new* set length! if err != nil { return nil, err // skip entire packet } pp := &ipfix.Message{ Version: 10, ExportTime: old.ExportTime, SequenceNumber: uint32(sequenceNumber), // this needs to be rewritten! ObservationDomainId: old.ObservationDomainId, Length: uint16(n + ipfixPacketHeaderLength + ipfixSetHeaderLength), Sets: []ipfix.Set{ { SetHeader: ipfix.SetHeader{ Id: fs.Id, Length: uint16(n + ipfixSetHeaderLength), // single record length + set header length }, Set: &ipfix.DataSet{ Records: []ipfix.DataRecord{rr}, }, }, }, } sequenceNumber++ new = append(new, pp) } } } return }
Output:
Index ¶
- Constants
- Variables
- func FromContext(ctx context.Context, keysAndValues ...interface{}) logr.Logger
- func MustReadCSV(r io.Reader) map[uint16]*InformationElement
- func MustReadXML(r io.Reader) map[uint16]InformationElement
- func MustReadYAML(r io.Reader) map[uint16]*InformationElement
- func MustWriteYAML(w io.Writer, m map[uint16]*InformationElement)
- func NewDataTypeBuilder(constructor DataTypeConstructor) *dataTypeBuilder
- func NewIPFIXFileReader(f io.ReadCloser) *ipfixFileReader
- func ReadCSV(r io.Reader) (map[uint16]*InformationElement, error)
- func ReadXML(r io.Reader) (map[uint16]InformationElement, error)
- func ReadYAML(r io.Reader) (map[uint16]*InformationElement, error)
- func SetLogger(l logr.Logger)
- func WriteYAML(w io.Writer, m map[uint16]*InformationElement) error
- type BasicList
- func (t *BasicList) Clone() DataType
- func (t *BasicList) Decode(r io.Reader) (n int, err error)
- func (*BasicList) DefaultLength() uint16
- func (t *BasicList) Elements() []Field
- func (t *BasicList) Encode(w io.Writer) (n int, err error)
- func (t *BasicList) FieldID() uint16
- func (t *BasicList) IsReducedLength() bool
- func (t *BasicList) Length() uint16
- func (t *BasicList) MarshalJSON() ([]byte, error)
- func (t *BasicList) NewBuilder() listTypeBuilder
- func (t *BasicList) Semantic() ListSemantic
- func (t *BasicList) SetFieldID(s uint16) *BasicList
- func (t *BasicList) SetLength(length uint16) DataType
- func (t *BasicList) SetSemantic(s ListSemantic) *BasicList
- func (t *BasicList) SetValue(v any) DataType
- func (t *BasicList) String() string
- func (t *BasicList) Type() string
- func (t *BasicList) UnmarshalJSON(in []byte) error
- func (t *BasicList) Value() interface{}
- func (t *BasicList) WithLength(length uint16) DataTypeConstructor
- func (t *BasicList) WithManager(mgr FieldCache) DataTypeConstructor
- type BidirectionalField
- type Boolean
- func (t *Boolean) Clone() DataType
- func (t *Boolean) Decode(in io.Reader) (int, error)
- func (*Boolean) DefaultLength() uint16
- func (t *Boolean) Encode(w io.Writer) (int, error)
- func (*Boolean) IsReducedLength() bool
- func (t *Boolean) Length() uint16
- func (t *Boolean) MarshalJSON() ([]byte, error)
- func (t *Boolean) SetLength(length uint16) DataType
- func (t *Boolean) SetValue(v any) DataType
- func (t *Boolean) String() string
- func (Boolean) Type() string
- func (t *Boolean) UnmarshalJSON(in []byte) error
- func (t *Boolean) Value() interface{}
- func (*Boolean) WithLength(length uint16) DataTypeConstructor
- type DataRecord
- func (d *DataRecord) Clone() DataRecord
- func (dr *DataRecord) Decode(r io.Reader) (n int, err error)
- func (dr *DataRecord) Encode(w io.Writer) (n int, err error)
- func (d *DataRecord) Length() uint16
- func (dr *DataRecord) String() string
- func (dr *DataRecord) UnmarshalJSON(in []byte) error
- func (dr *DataRecord) With(t *Template) *DataRecord
- type DataSet
- type DataType
- func NewBasicList() DataType
- func NewBoolean() DataType
- func NewDateTimeMicroseconds() DataType
- func NewDateTimeMilliseconds() DataType
- func NewDateTimeNanoseconds() DataType
- func NewDateTimeSeconds() DataType
- func NewDefaultSubTemplateList() DataType
- func NewDefaultSubTemplateMultiList() DataType
- func NewFloat32() DataType
- func NewFloat64() DataType
- func NewIPv4Address() DataType
- func NewIPv6Address() DataType
- func NewMacAddress() DataType
- func NewOctetArray() DataType
- func NewSigned16() DataType
- func NewSigned32() DataType
- func NewSigned64() DataType
- func NewSigned8() DataType
- func NewString() DataType
- func NewUnsigned16() DataType
- func NewUnsigned32() DataType
- func NewUnsigned64() DataType
- func NewUnsigned8() DataType
- type DataTypeConstructor
- type DateTimeMicroseconds
- func (t *DateTimeMicroseconds) Clone() DataType
- func (t *DateTimeMicroseconds) Decode(in io.Reader) (int, error)
- func (t *DateTimeMicroseconds) DefaultLength() uint16
- func (t *DateTimeMicroseconds) Encode(w io.Writer) (int, error)
- func (*DateTimeMicroseconds) IsReducedLength() bool
- func (t DateTimeMicroseconds) Length() uint16
- func (t *DateTimeMicroseconds) MarshalJSON() ([]byte, error)
- func (t *DateTimeMicroseconds) SetLength(length uint16) DataType
- func (t *DateTimeMicroseconds) SetValue(v any) DataType
- func (t *DateTimeMicroseconds) String() string
- func (t DateTimeMicroseconds) Type() string
- func (t *DateTimeMicroseconds) UnmarshalJSON(in []byte) error
- func (t *DateTimeMicroseconds) Value() interface{}
- func (*DateTimeMicroseconds) WithLength(length uint16) DataTypeConstructor
- type DateTimeMilliseconds
- func (t *DateTimeMilliseconds) Clone() DataType
- func (t *DateTimeMilliseconds) Decode(in io.Reader) (int, error)
- func (t *DateTimeMilliseconds) DefaultLength() uint16
- func (t *DateTimeMilliseconds) Encode(w io.Writer) (int, error)
- func (*DateTimeMilliseconds) IsReducedLength() bool
- func (t DateTimeMilliseconds) Length() uint16
- func (t *DateTimeMilliseconds) MarshalJSON() ([]byte, error)
- func (t *DateTimeMilliseconds) SetLength(length uint16) DataType
- func (t *DateTimeMilliseconds) SetValue(v any) DataType
- func (t *DateTimeMilliseconds) String() string
- func (t DateTimeMilliseconds) Type() string
- func (t *DateTimeMilliseconds) UnmarshalJSON(in []byte) error
- func (t *DateTimeMilliseconds) Value() interface{}
- func (*DateTimeMilliseconds) WithLength(length uint16) DataTypeConstructor
- type DateTimeNanoseconds
- func (t *DateTimeNanoseconds) Clone() DataType
- func (t *DateTimeNanoseconds) Decode(in io.Reader) (int, error)
- func (t *DateTimeNanoseconds) DefaultLength() uint16
- func (t *DateTimeNanoseconds) Encode(w io.Writer) (int, error)
- func (*DateTimeNanoseconds) IsReducedLength() bool
- func (t DateTimeNanoseconds) Length() uint16
- func (t *DateTimeNanoseconds) MarshalJSON() ([]byte, error)
- func (t *DateTimeNanoseconds) SetLength(length uint16) DataType
- func (t *DateTimeNanoseconds) SetValue(v any) DataType
- func (t *DateTimeNanoseconds) String() string
- func (t *DateTimeNanoseconds) Type() string
- func (t *DateTimeNanoseconds) UnmarshalJSON(in []byte) error
- func (t *DateTimeNanoseconds) Value() interface{}
- func (*DateTimeNanoseconds) WithLength(length uint16) DataTypeConstructor
- type DateTimeSeconds
- func (t *DateTimeSeconds) Clone() DataType
- func (t *DateTimeSeconds) Decode(in io.Reader) (int, error)
- func (t *DateTimeSeconds) DefaultLength() uint16
- func (t *DateTimeSeconds) Encode(w io.Writer) (int, error)
- func (*DateTimeSeconds) IsReducedLength() bool
- func (t *DateTimeSeconds) Length() uint16
- func (t *DateTimeSeconds) MarshalJSON() ([]byte, error)
- func (t *DateTimeSeconds) SetLength(length uint16) DataType
- func (t *DateTimeSeconds) SetValue(v any) DataType
- func (t *DateTimeSeconds) String() string
- func (*DateTimeSeconds) Type() string
- func (t *DateTimeSeconds) UnmarshalJSON(in []byte) error
- func (t *DateTimeSeconds) Value() interface{}
- func (*DateTimeSeconds) WithLength(length uint16) DataTypeConstructor
- type DecayingEphemeralCache
- func (ts *DecayingEphemeralCache) Add(ctx context.Context, key TemplateKey, template *Template) error
- func (ts *DecayingEphemeralCache) Close(context.Context) error
- func (t *DecayingEphemeralCache) Delete(ctx context.Context, key TemplateKey) error
- func (ts *DecayingEphemeralCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
- func (ts *DecayingEphemeralCache) GetAll(ctx context.Context) map[TemplateKey]*Template
- func (ts *DecayingEphemeralCache) MarshalJSON() ([]byte, error)
- func (ts *DecayingEphemeralCache) Name() string
- func (ts *DecayingEphemeralCache) SetTimeout(d time.Duration)
- func (ts *DecayingEphemeralCache) Start(ctx context.Context) error
- func (ts *DecayingEphemeralCache) Type() string
- type Decoder
- type DecoderOptions
- type EphemeralCache
- func (ts *EphemeralCache) Add(ctx context.Context, key TemplateKey, template *Template) error
- func (ts *EphemeralCache) Close(context.Context) error
- func (ts *EphemeralCache) Delete(ctx context.Context, key TemplateKey) error
- func (ts *EphemeralCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
- func (ts *EphemeralCache) GetAll(ctx context.Context) map[TemplateKey]*Template
- func (ts *EphemeralCache) Initialize(context.Context) error
- func (ts *EphemeralCache) MarshalJSON() ([]byte, error)
- func (ts *EphemeralCache) Name() string
- func (ts *EphemeralCache) Prepare() error
- func (ts *EphemeralCache) Start(ctx context.Context) error
- func (ts *EphemeralCache) Type() string
- type EphemeralFieldCache
- func (fm *EphemeralFieldCache) Add(ctx context.Context, element InformationElement) error
- func (fm *EphemeralFieldCache) Delete(ctx context.Context, key FieldKey) error
- func (fm *EphemeralFieldCache) Get(ctx context.Context, key FieldKey) (*InformationElement, error)
- func (fm *EphemeralFieldCache) GetAll(ctx context.Context) map[FieldKey]*InformationElement
- func (fm *EphemeralFieldCache) GetAllBuilders(ctx context.Context) map[FieldKey]*FieldBuilder
- func (fm *EphemeralFieldCache) GetBuilder(ctx context.Context, key FieldKey) (*FieldBuilder, error)
- func (fm *EphemeralFieldCache) MarshalJSON() ([]byte, error)
- type Field
- type FieldBuilder
- func (b *FieldBuilder) Complete() Field
- func (b *FieldBuilder) GetIE() *InformationElement
- func (b *FieldBuilder) MarshalJSON() ([]byte, error)
- func (b *FieldBuilder) SetFieldManager(fieldManager FieldCache) *FieldBuilder
- func (b *FieldBuilder) SetLength(length uint16) *FieldBuilder
- func (b *FieldBuilder) SetObservationDomain(id uint32) *FieldBuilder
- func (b *FieldBuilder) SetPEN(pen uint32) *FieldBuilder
- func (b *FieldBuilder) SetReversed(isReverse bool) *FieldBuilder
- func (b *FieldBuilder) SetTemplateManager(templateManager TemplateCache) *FieldBuilder
- func (b *FieldBuilder) UnmarshalJSON(in []byte) error
- type FieldCache
- type FieldKey
- type FixedLengthField
- func (f *FixedLengthField) Clone() Field
- func (f *FixedLengthField) Constructor() DataTypeConstructor
- func (f *FixedLengthField) Decode(r io.Reader) (int, error)
- func (f *FixedLengthField) Encode(w io.Writer) (int, error)
- func (f *FixedLengthField) Id() uint16
- func (f *FixedLengthField) IsScope() bool
- func (f *FixedLengthField) Length() uint16
- func (f *FixedLengthField) Lift() *VariableLengthField
- func (f *FixedLengthField) MarshalJSON() ([]byte, error)
- func (f *FixedLengthField) Name() string
- func (f *FixedLengthField) ObservationDomainId() uint32
- func (f *FixedLengthField) PEN() uint32
- func (f *FixedLengthField) Prototype() *InformationElement
- func (f *FixedLengthField) Reversed() bool
- func (f *FixedLengthField) Reversible() bool
- func (f *FixedLengthField) SetScoped() Field
- func (f *FixedLengthField) SetValue(v any) Field
- func (f *FixedLengthField) String() string
- func (f *FixedLengthField) Type() string
- func (f *FixedLengthField) UnmarshalJSON(in []byte) error
- func (f *FixedLengthField) Value() DataType
- type Float32
- func (t *Float32) Clone() DataType
- func (t *Float32) Decode(in io.Reader) (int, error)
- func (*Float32) DefaultLength() uint16
- func (t *Float32) Encode(w io.Writer) (int, error)
- func (*Float32) IsReducedLength() bool
- func (t *Float32) Length() uint16
- func (t *Float32) MarshalJSON() ([]byte, error)
- func (t *Float32) SetLength(length uint16) DataType
- func (t *Float32) SetValue(v any) DataType
- func (t *Float32) String() string
- func (*Float32) Type() string
- func (t *Float32) UnmarshalJSON(in []byte) error
- func (t *Float32) Value() interface{}
- func (*Float32) WithLength(length uint16) DataTypeConstructor
- type Float64
- func (t *Float64) Clone() DataType
- func (t *Float64) Decode(in io.Reader) (int, error)
- func (*Float64) DefaultLength() uint16
- func (t *Float64) Encode(w io.Writer) (int, error)
- func (*Float64) IsReducedLength() bool
- func (t *Float64) Length() uint16
- func (t *Float64) MarshalJSON() ([]byte, error)
- func (t *Float64) SetLength(length uint16) DataType
- func (t *Float64) SetValue(v any) DataType
- func (t *Float64) String() string
- func (*Float64) Type() string
- func (t *Float64) UnmarshalJSON(in []byte) error
- func (t *Float64) Value() interface{}
- func (*Float64) WithLength(length uint16) DataTypeConstructor
- type IPv4Address
- func (t *IPv4Address) Clone() DataType
- func (t *IPv4Address) Decode(in io.Reader) (n int, err error)
- func (*IPv4Address) DefaultLength() uint16
- func (t *IPv4Address) Encode(w io.Writer) (int, error)
- func (*IPv4Address) IsReducedLength() bool
- func (t *IPv4Address) Length() uint16
- func (t *IPv4Address) MarshalJSON() ([]byte, error)
- func (t *IPv4Address) SetLength(length uint16) DataType
- func (t *IPv4Address) SetValue(v any) DataType
- func (t *IPv4Address) String() string
- func (*IPv4Address) Type() string
- func (t *IPv4Address) UnmarshalJSON(in []byte) error
- func (t *IPv4Address) Value() interface{}
- func (*IPv4Address) WithLength(length uint16) DataTypeConstructor
- type IPv6Address
- func (t *IPv6Address) Clone() DataType
- func (t *IPv6Address) Decode(in io.Reader) (n int, err error)
- func (t *IPv6Address) DefaultLength() uint16
- func (t *IPv6Address) Encode(w io.Writer) (int, error)
- func (*IPv6Address) IsReducedLength() bool
- func (t IPv6Address) Length() uint16
- func (t *IPv6Address) MarshalJSON() ([]byte, error)
- func (t *IPv6Address) SetLength(length uint16) DataType
- func (t *IPv6Address) SetValue(v any) DataType
- func (t *IPv6Address) String() string
- func (t IPv6Address) Type() string
- func (t *IPv6Address) UnmarshalJSON(in []byte) error
- func (t *IPv6Address) Value() interface{}
- func (t *IPv6Address) WithLength(length uint16) DataTypeConstructor
- type InformationElement
- type InformationElementRange
- type ListSemantic
- type MacAddress
- func (t *MacAddress) Clone() DataType
- func (t *MacAddress) Decode(in io.Reader) (n int, err error)
- func (*MacAddress) DefaultLength() uint16
- func (t *MacAddress) Encode(w io.Writer) (int, error)
- func (*MacAddress) IsReducedLength() bool
- func (t *MacAddress) Length() uint16
- func (t *MacAddress) MarshalJSON() ([]byte, error)
- func (t *MacAddress) SetLength(length uint16) DataType
- func (t *MacAddress) SetValue(v any) DataType
- func (t *MacAddress) String() string
- func (*MacAddress) Type() string
- func (t *MacAddress) UnmarshalJSON(in []byte) error
- func (t *MacAddress) Value() interface{}
- func (t *MacAddress) WithLength(length uint16) DataTypeConstructor
- type Message
- type OctetArray
- func (t *OctetArray) Clone() DataType
- func (t *OctetArray) Decode(in io.Reader) (n int, err error)
- func (*OctetArray) DefaultLength() uint16
- func (t *OctetArray) Encode(w io.Writer) (int, error)
- func (*OctetArray) IsReducedLength() bool
- func (t *OctetArray) Length() uint16
- func (t *OctetArray) MarshalJSON() ([]byte, error)
- func (t *OctetArray) SetLength(length uint16) DataType
- func (t *OctetArray) SetValue(v any) DataType
- func (t *OctetArray) String() string
- func (*OctetArray) Type() string
- func (t *OctetArray) UnmarshalJSON(in []byte) error
- func (t *OctetArray) Value() interface{}
- func (*OctetArray) WithLength(length uint16) DataTypeConstructor
- type OptionsTemplateRecord
- func (otr *OptionsTemplateRecord) Decode(r io.Reader) (n int, err error)
- func (otr *OptionsTemplateRecord) Encode(w io.Writer) (n int, err error)
- func (otr *OptionsTemplateRecord) Id() uint16
- func (otr *OptionsTemplateRecord) Length() uint16
- func (otr *OptionsTemplateRecord) MarshalJSON() ([]byte, error)
- func (otr *OptionsTemplateRecord) String() string
- func (otr *OptionsTemplateRecord) Type() string
- func (otr *OptionsTemplateRecord) UnmarshalJSON(in []byte) error
- type OptionsTemplateSet
- type PersistentCache
- func (t *PersistentCache) Add(ctx context.Context, key TemplateKey, template *Template) error
- func (t *PersistentCache) Close(context.Context) error
- func (t *PersistentCache) Delete(ctx context.Context, key TemplateKey) error
- func (t *PersistentCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
- func (t *PersistentCache) GetAll(ctx context.Context) map[TemplateKey]*Template
- func (t *PersistentCache) Initialize(ctx context.Context) error
- func (t *PersistentCache) MarshalJSON() ([]byte, error)
- func (t *PersistentCache) Name() string
- func (t *PersistentCache) Prepare() error
- func (t *PersistentCache) Start(ctx context.Context) error
- func (t *PersistentCache) Type() string
- type RawMessage
- type Set
- type SetHeader
- type Signed16
- func (t *Signed16) Clone() DataType
- func (t *Signed16) Decode(in io.Reader) (n int, err error)
- func (*Signed16) DefaultLength() uint16
- func (t *Signed16) Encode(w io.Writer) (int, error)
- func (t *Signed16) IsReducedLength() bool
- func (t *Signed16) Length() uint16
- func (t *Signed16) MarshalJSON() ([]byte, error)
- func (t *Signed16) SetLength(length uint16) DataType
- func (t *Signed16) SetValue(v any) DataType
- func (t *Signed16) String() string
- func (*Signed16) Type() string
- func (t *Signed16) UnmarshalJSON(in []byte) error
- func (t *Signed16) Value() interface{}
- func (t *Signed16) WithLength(length uint16) DataTypeConstructor
- type Signed32
- func (t *Signed32) Clone() DataType
- func (t *Signed32) Decode(in io.Reader) (n int, err error)
- func (*Signed32) DefaultLength() uint16
- func (t *Signed32) Encode(w io.Writer) (int, error)
- func (t *Signed32) IsReducedLength() bool
- func (t *Signed32) Length() uint16
- func (t *Signed32) MarshalJSON() ([]byte, error)
- func (t *Signed32) SetLength(length uint16) DataType
- func (t *Signed32) SetValue(v any) DataType
- func (t *Signed32) String() string
- func (*Signed32) Type() string
- func (t *Signed32) UnmarshalJSON(in []byte) error
- func (t *Signed32) Value() interface{}
- func (t *Signed32) WithLength(length uint16) DataTypeConstructor
- type Signed64
- func (t *Signed64) Clone() DataType
- func (t *Signed64) Decode(in io.Reader) (n int, err error)
- func (*Signed64) DefaultLength() uint16
- func (t *Signed64) Encode(w io.Writer) (int, error)
- func (t *Signed64) IsReducedLength() bool
- func (t *Signed64) Length() uint16
- func (t *Signed64) MarshalJSON() ([]byte, error)
- func (t *Signed64) SetLength(length uint16) DataType
- func (t *Signed64) SetValue(v any) DataType
- func (t *Signed64) String() string
- func (*Signed64) Type() string
- func (t *Signed64) UnmarshalJSON(in []byte) error
- func (t *Signed64) Value() interface{}
- func (t *Signed64) WithLength(length uint16) DataTypeConstructor
- type Signed8
- func (t *Signed8) Clone() DataType
- func (t *Signed8) Decode(in io.Reader) (n int, err error)
- func (t *Signed8) DefaultLength() uint16
- func (t *Signed8) Encode(w io.Writer) (int, error)
- func (*Signed8) IsReducedLength() bool
- func (t *Signed8) Length() uint16
- func (t *Signed8) MarshalJSON() ([]byte, error)
- func (t *Signed8) SetLength(length uint16) DataType
- func (t *Signed8) SetValue(v any) DataType
- func (t *Signed8) String() string
- func (*Signed8) Type() string
- func (t *Signed8) UnmarshalJSON(in []byte) error
- func (t *Signed8) Value() interface{}
- func (*Signed8) WithLength(length uint16) DataTypeConstructor
- type StatefulTemplateCache
- func NewDefaultEphemeralCache() StatefulTemplateCache
- func NewDefaultPersistentCache(file *os.File, fieldCache FieldCache, templateCache StatefulTemplateCache) StatefulTemplateCache
- func NewNamedEphemeralCache(name string) StatefulTemplateCache
- func NewNamedPersistentCache(name string, file *os.File, fieldCache FieldCache, ...) StatefulTemplateCache
- type String
- func (t *String) Clone() DataType
- func (t *String) Decode(in io.Reader) (n int, err error)
- func (*String) DefaultLength() uint16
- func (t *String) Encode(w io.Writer) (int, error)
- func (*String) IsReducedLength() bool
- func (t *String) Length() uint16
- func (t *String) MarshalJSON() ([]byte, error)
- func (t *String) SetLength(length uint16) DataType
- func (t *String) SetValue(v any) DataType
- func (s *String) String() string
- func (*String) Type() string
- func (t *String) UnmarshalJSON(in []byte) error
- func (t *String) Value() interface{}
- func (*String) WithLength(length uint16) DataTypeConstructor
- type SubTemplateList
- func (t *SubTemplateList) Clone() DataType
- func (t *SubTemplateList) Decode(r io.Reader) (n int, err error)
- func (*SubTemplateList) DefaultLength() uint16
- func (t *SubTemplateList) Elements() []DataRecord
- func (t *SubTemplateList) Encode(w io.Writer) (n int, err error)
- func (t *SubTemplateList) IsReducedLength() bool
- func (t *SubTemplateList) Length() uint16
- func (t *SubTemplateList) MarshalJSON() ([]byte, error)
- func (t *SubTemplateList) NewBuilder() templateListeTypeBuilder
- func (t *SubTemplateList) Semantic() ListSemantic
- func (t *SubTemplateList) SetLength(length uint16) DataType
- func (t *SubTemplateList) SetSemantic(semantic ListSemantic) *SubTemplateList
- func (t *SubTemplateList) SetValue(v any) DataType
- func (t *SubTemplateList) String() string
- func (t *SubTemplateList) TemplateID() uint16
- func (t *SubTemplateList) Type() string
- func (t *SubTemplateList) UnmarshalJSON(in []byte) error
- func (t *SubTemplateList) Value() interface{}
- func (t *SubTemplateList) WithLength(length uint16) DataTypeConstructor
- type SubTemplateMultiList
- func (t *SubTemplateMultiList) Clone() DataType
- func (t *SubTemplateMultiList) Decode(r io.Reader) (n int, err error)
- func (*SubTemplateMultiList) DefaultLength() uint16
- func (t *SubTemplateMultiList) Elements() []subTemplateListContent
- func (t *SubTemplateMultiList) Encode(w io.Writer) (n int, err error)
- func (*SubTemplateMultiList) IsReducedLength() bool
- func (t *SubTemplateMultiList) Length() uint16
- func (t *SubTemplateMultiList) MarshalJSON() ([]byte, error)
- func (t *SubTemplateMultiList) NewBuilder() templateListeTypeBuilder
- func (t *SubTemplateMultiList) Semantic() ListSemantic
- func (t *SubTemplateMultiList) SetLength(length uint16) DataType
- func (t *SubTemplateMultiList) SetSemantic(semantic ListSemantic) *SubTemplateMultiList
- func (t *SubTemplateMultiList) SetValue(v any) DataType
- func (t *SubTemplateMultiList) String() string
- func (t *SubTemplateMultiList) Type() string
- func (t *SubTemplateMultiList) UnmarshalJSON(in []byte) error
- func (t *SubTemplateMultiList) Value() interface{}
- func (t *SubTemplateMultiList) WithLength(length uint16) DataTypeConstructor
- type TCPListener
- type Template
- type TemplateCache
- type TemplateCacheDriver
- type TemplateCacheWithTimeout
- type TemplateKey
- type TemplateMetadata
- type TemplateRecord
- func (tr *TemplateRecord) Decode(r io.Reader) (n int, err error)
- func (tr *TemplateRecord) Encode(w io.Writer) (n int, err error)
- func (tr *TemplateRecord) Id() uint16
- func (tr *TemplateRecord) Length() uint16
- func (tr *TemplateRecord) MarshalJSON() ([]byte, error)
- func (tr *TemplateRecord) String() string
- func (tr *TemplateRecord) Type() string
- func (tr *TemplateRecord) UnmarshalJSON(in []byte) error
- type TemplateSet
- type UDPListener
- type Unsigned16
- func (t *Unsigned16) Clone() DataType
- func (t *Unsigned16) Decode(in io.Reader) (n int, err error)
- func (t *Unsigned16) DefaultLength() uint16
- func (t *Unsigned16) Encode(w io.Writer) (int, error)
- func (t *Unsigned16) IsReducedLength() bool
- func (t *Unsigned16) Length() uint16
- func (t *Unsigned16) MarshalJSON() ([]byte, error)
- func (t *Unsigned16) SetLength(length uint16) DataType
- func (t *Unsigned16) SetValue(v any) DataType
- func (t *Unsigned16) String() string
- func (t Unsigned16) Type() string
- func (t *Unsigned16) UnmarshalJSON(in []byte) error
- func (t *Unsigned16) Value() interface{}
- func (t *Unsigned16) WithLength(length uint16) DataTypeConstructor
- type Unsigned32
- func (t *Unsigned32) Clone() DataType
- func (t *Unsigned32) Decode(in io.Reader) (n int, err error)
- func (*Unsigned32) DefaultLength() uint16
- func (t *Unsigned32) Encode(w io.Writer) (int, error)
- func (t *Unsigned32) IsReducedLength() bool
- func (t *Unsigned32) Length() uint16
- func (t *Unsigned32) MarshalJSON() ([]byte, error)
- func (t *Unsigned32) SetLength(length uint16) DataType
- func (t *Unsigned32) SetValue(v any) DataType
- func (t *Unsigned32) String() string
- func (*Unsigned32) Type() string
- func (t *Unsigned32) UnmarshalJSON(in []byte) error
- func (t *Unsigned32) Value() interface{}
- func (t *Unsigned32) WithLength(length uint16) DataTypeConstructor
- type Unsigned64
- func (t *Unsigned64) Clone() DataType
- func (t *Unsigned64) Decode(in io.Reader) (n int, err error)
- func (*Unsigned64) DefaultLength() uint16
- func (t *Unsigned64) Encode(w io.Writer) (int, error)
- func (t *Unsigned64) IsReducedLength() bool
- func (t *Unsigned64) Length() uint16
- func (t *Unsigned64) MarshalJSON() ([]byte, error)
- func (t *Unsigned64) SetLength(length uint16) DataType
- func (t *Unsigned64) SetValue(v any) DataType
- func (t *Unsigned64) String() string
- func (*Unsigned64) Type() string
- func (t *Unsigned64) UnmarshalJSON(in []byte) error
- func (t *Unsigned64) Value() interface{}
- func (t *Unsigned64) WithLength(length uint16) DataTypeConstructor
- type Unsigned8
- func (t *Unsigned8) Clone() DataType
- func (t *Unsigned8) Decode(in io.Reader) (n int, err error)
- func (*Unsigned8) DefaultLength() uint16
- func (t *Unsigned8) Encode(w io.Writer) (int, error)
- func (*Unsigned8) IsReducedLength() bool
- func (t *Unsigned8) Length() uint16
- func (t *Unsigned8) MarshalJSON() ([]byte, error)
- func (t *Unsigned8) SetLength(length uint16) DataType
- func (t *Unsigned8) SetValue(v any) DataType
- func (t *Unsigned8) String() string
- func (*Unsigned8) Type() string
- func (t *Unsigned8) UnmarshalJSON(in []byte) error
- func (t *Unsigned8) Value() interface{}
- func (*Unsigned8) WithLength(length uint16) DataTypeConstructor
- type VariableLengthField
- func (f *VariableLengthField) Clone() Field
- func (f *VariableLengthField) Constructor() DataTypeConstructor
- func (f *VariableLengthField) Decode(r io.Reader) (int, error)
- func (f *VariableLengthField) Encode(w io.Writer) (int, error)
- func (f *VariableLengthField) Id() uint16
- func (f *VariableLengthField) IsScope() bool
- func (f *VariableLengthField) Length() uint16
- func (f *VariableLengthField) Lift() *VariableLengthField
- func (f *VariableLengthField) MarshalJSON() ([]byte, error)
- func (f *VariableLengthField) Name() string
- func (f *VariableLengthField) ObservationDomainId() uint32
- func (f *VariableLengthField) PEN() uint32
- func (f *VariableLengthField) Prototype() *InformationElement
- func (f *VariableLengthField) Reversed() bool
- func (f *VariableLengthField) Reversible() bool
- func (f *VariableLengthField) SetScoped() Field
- func (f *VariableLengthField) SetValue(v any) Field
- func (f *VariableLengthField) String() string
- func (f *VariableLengthField) Type() string
- func (f *VariableLengthField) UnmarshalJSON(in []byte) error
- func (f *VariableLengthField) Value() DataType
Examples ¶
Constants ¶
const ( // NFv9 is the NFv9 template id, kept for completeness and compatibility, the module // does not actually support Netflow 9 decoding out of the box. NFv9 uint16 = iota // NFv9Options is the NFv9 options template id, kept for completeness and compatibility, the module // does not actually support Netflow 9 decoding out of the box. NFv9Options // IPFIX is the id denoting a template set. IPFIX // IPFIXOptions is the id denoting an options template set. IPFIXOptions )
const ( KindDataSet string = "DataSet" KindTemplateSet string = "TemplateSet" KindOptionsTemplateSet string = "OptionsTemplateSet" )
The Kind* constants are used for unmarshalling of JSON records to denote the specific type into which the elements of a set should be unmarshalled in.
const ReversePEN uint32 = 29305
ReversePEN is the private enterprise number designated for signaling bidirectional flow information contained in the record. By default, if a field has this PEN, the prototype IE is taken from the IANA IEs (PEN == 0), but the semantic value of the field denotes that the information describes data "in the opposite" direction of the flow. The decoder handles this transparently, i.e., lookup of IEs from the FieldCache is adjusted for.
All IANA IEs are reversible, except for the ones listed in nonReversibleFields, which is used to check if the exporter illegaly used the reverse PEN on a field that is not reversible.
const ( // VariableLength is the constant used for denoting a field being variable-length encoded // in template records before the length is known. VariableLength uint16 = 0xFFFF )
Variables ¶
var ( // ErrTemplateNotFound is the base error used for indicating missing templates in caches. // It may be used in errors.Is() checks for error type, whereas compound errors constructed // with TemplateNotFound(...) cannot be compared with == due to including more information ErrTemplateNotFound error = errors.New("template not found") // ErrUnknownVersion indicates an illegal version number for IPFIX in the header of the message. ErrUnknownVersion error = errors.New("unknown version") // ErrUnknownFlowId is used for indicating usage of a set ID unassigned in IPFIX, which is specifically // the interval [5, 255], which is reserved. ErrUnknownFlowId error = errors.New("unknown flow id") // ErrIllegalDataTypeEncoding is used in Decode of certain data types that explicitly define illegal formats // such as boolean (1 and 2 encoding true and false and all other values being illegal) or strings // only allowing utf8 sequences. ErrIllegalDataTypeEncoding = errors.New("illegal data type encoding") )
var ( ErrUnknownProtocolVersion = errors.New("unknown protocol version in field manager") ErrUnknownEnterpriseNumber = errors.New("unknown enterprise number in field manager") )
var ( PacketsTotal = prometheus.NewCounter(prometheus.CounterOpts{ Name: "decoder_decoded_packets_total", Help: "Total number of decoded packets in decoder", }) ErrorsTotal = prometheus.NewCounter(prometheus.CounterOpts{ Name: "decoder_errors_total", Help: "Total number of errors in decoder", }) DurationMicroseconds = prometheus.NewHistogram(prometheus.HistogramOpts{ Name: "decoder_duration_microseconds", Help: "Duration of decoding per protocol in microseconds", Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 2500}, }) DecodedSets = prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "decoder_decoded_sets_total", Help: "Total number of decoded sets per type", }, []string{"type"}) DecodedRecords = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: "collector", Name: "decoder_decoded_records_total", Help: "Total number of decoded records per type", }, []string{"type"}) DroppedRecords = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: "collector", Name: "decoder_dropped_records_total", Help: "Total number of records dropped due to filters per type", }, []string{"type"}) )
var ( TCPActiveConnections = prometheus.NewGauge(prometheus.GaugeOpts{ Name: "tcp_listener_active_connections_total", Help: "Total number of active connections currently maintained by the TCP listener", }) TCPErrorsTotal = prometheus.NewCounter(prometheus.CounterOpts{ Name: "tcp_listener_errors_total", Help: "Total number of errors encountered in the TCP listener", }) TCPReceivedBytes = prometheus.NewCounter(prometheus.CounterOpts{ Name: "tcp_listener_received_bytes", Help: "Total number of bytes read in the TCP listener", }) )
var ( UDPPacketsTotal = prometheus.NewCounter(prometheus.CounterOpts{ Name: "udp_listener_packets_total", Help: "Total number of packets received via UDP listener", }) UDPErrorsTotal = prometheus.NewCounter(prometheus.CounterOpts{ Name: "udp_listener_errors_total", Help: "Total number of errors encountered in the UDP listener", }) UDPPacketBytes = prometheus.NewCounter(prometheus.CounterOpts{ Name: "udp_listener_packet_bytes", Help: "Total number of bytes read in the UDP listener", }) )
var ( DefaultDecoderOptions = DecoderOptions{ OmitRFC5610Records: false, } )
var (
Log = logr.New(rootLog)
)
var NonReversibleFields map[uint16]InformationElement = map[uint16]InformationElement{
10: {
Id: 10,
Name: "ingressInterface",
},
14: {
Id: 14,
Name: "egressInterface",
},
137: {
Id: 137,
Name: "commonPropertiesId",
},
138: {
Id: 138,
Name: "observationPointId",
},
141: {
Id: 141,
Name: "lineCardId",
},
142: {
Id: 142,
Name: "portId",
},
143: {
Id: 143,
Name: "meteringProcessId",
},
144: {
Id: 144,
Name: "exportingProcessId",
},
145: {
Id: 145,
Name: "templateId",
},
148: {
Id: 148,
Name: "flowId",
},
149: {
Id: 149,
Name: "observationDomainId",
},
130: {
Id: 130,
Name: "exporterIPv4Address",
},
131: {
Id: 131,
Name: "exporterIPv6Address",
},
217: {
Id: 217,
Name: "exporterTransportPort",
},
211: {
Id: 211,
Name: "collectorIPv4Address",
},
212: {
Id: 212,
Name: "collectorIPv6Address",
},
213: {
Id: 213,
Name: "exportInterface",
},
214: {
Id: 214,
Name: "exportProtocolVersion",
},
215: {
Id: 215,
Name: "exportTransportVersion",
},
216: {
Id: 216,
Name: "collecotrTransportPort",
},
173: {
Id: 173,
Name: "flowKeyIndicator",
},
41: {
Id: 41,
Name: "exportedMessageTotalCount",
},
40: {
Id: 40,
Name: "exportedOctetTotalCount",
},
42: {
Id: 42,
Name: "exportedFlowRecordTotalCount",
},
163: {
Id: 163,
Name: "observedFlowTotalCount",
},
164: {
Id: 164,
Name: "ignoredPacketTotalCount",
},
165: {
Id: 165,
Name: "ignoredOctetTotalCount",
},
166: {
Id: 166,
Name: "notSentFlowTotalCount",
},
167: {
Id: 167,
Name: "notSentPacketTotalCount",
},
168: {
Id: 168,
Name: "notSentOctetTotalCount",
},
210: {
Id: 210,
Name: "paddingOctets",
},
239: {
Id: 239,
Name: "biflowDirection",
},
}
NonReversibleFields is the lookup map for fields that are _not_ reversible as per RFC 5103. Authoritative information is found in https://datatracker.ietf.org/doc/html/rfc5103
var ( // UDP packet size is globally limited by the packet header length field of 2^16-1. // However, additionally, IP data path MTU can cause UDP packets larger than the MTU // to be fragmented. If fragments are lost due to packet loss, the UDP packet cannot be // reassembled and is therefore dropped entirely. Preventing fragmentation therefore is // a good measure to prevent losing to many packets // // These days, MTU is most of the times in the ball park of 1500 Bytes. Subtracting IP and UDP // packet header lengths, as well as headers of various encapsulation formats, this then yields // a maximum packet size for UDP packets of 1420 (At least that is what yaf assumes). // // We patched yaf to support larger UDP packets, and can therefore use more bytes here // udpPacketBufferSize int = 0xFFFF, which in turn allows us to use more verbose DPI // with UDP transport (this previously caused a lot of unrecoverable crashes) UDPPacketBufferSize int = 1500 )
Functions ¶
func MustReadCSV ¶
func MustReadCSV(r io.Reader) map[uint16]*InformationElement
func MustReadXML ¶
func MustReadXML(r io.Reader) map[uint16]InformationElement
func MustReadYAML ¶
func MustReadYAML(r io.Reader) map[uint16]*InformationElement
func MustWriteYAML ¶
func MustWriteYAML(w io.Writer, m map[uint16]*InformationElement)
func NewDataTypeBuilder ¶
func NewDataTypeBuilder(constructor DataTypeConstructor) *dataTypeBuilder
func NewIPFIXFileReader ¶ added in v0.3.0
func NewIPFIXFileReader(f io.ReadCloser) *ipfixFileReader
NewIPFIXFileReader creates a new reader from a file-like reader. It is intended to be used asynchronously using the message channel. from a parent context. Therefore, it needs to be started using Start(context.Context) inside a goroutine, because Start(context.Context) blocks until
Types ¶
type BasicList ¶
type BasicList struct {
// contains filtered or unexported fields
}
BasicList implements the basicList abstract data type as per RFC 6313. Note that the structured data types described in RFC 6313 require state: during decoding, the *Field ID* in the header of the data type is looked up to map the data semantics to other IPFIX abstract data types (read: what type of data the list contains).
For this, go-ipfix uses its abstraction of Caches, here a FieldCache, to look up this state during decoding. This abstraction allows you to add new field types and information elements during runtime. For more information, see the Cache documentation or the general workflow using go-ipfix.
func (*BasicList) DefaultLength ¶
DefaultLength implements DataType's DefaultLength method, which for basic lists, is statically 0.
func (*BasicList) Elements ¶
Elements does the same thing as Value(), returning t.value, which is a slice of DataType- implementors, but with a narrower type than Values(), which returns interface{}
func (*BasicList) IsReducedLength ¶
func (*BasicList) MarshalJSON ¶
func (*BasicList) NewBuilder ¶
func (t *BasicList) NewBuilder() listTypeBuilder
func (*BasicList) Semantic ¶
func (t *BasicList) Semantic() ListSemantic
func (*BasicList) SetFieldID ¶
func (*BasicList) SetSemantic ¶
func (t *BasicList) SetSemantic(s ListSemantic) *BasicList
func (*BasicList) SetValue ¶
NOTE that this allows for various types of list items, as long as they implement the DataType interface; in IPFIX, this is forbidden: basicList elements must all have the same type, encoded by the fieldId read in the "header" bytes of the list.
SetValue does not perform additional type checks as it is fine with variable types.
func (*BasicList) String ¶
String converts a basic list to a string similar to Go's formating of slices. A stringified basicList is "[ value_1, value_2, ... ]" where value_1 etc. are the string representations of the contents of the basic list.
func (*BasicList) UnmarshalJSON ¶
func (*BasicList) Value ¶
func (t *BasicList) Value() interface{}
Value returns the internal value of the BasicList object as a type-assertable interface. This is prominently used during JSON serialization.
func (*BasicList) WithLength ¶
func (t *BasicList) WithLength(length uint16) DataTypeConstructor
WithLength implements DataType's WithLength decorator method that injects a static length into the constructed BasicList object. This is used in FieldBuilder, which should be used by users, rather than instantiating the BasicList directly.
func (*BasicList) WithManager ¶
func (t *BasicList) WithManager(mgr FieldCache) DataTypeConstructor
WithManager is a decorator for a BasicList that returns a new constructor function for new basic lists with a given FieldCache injected into the BasicList object. This injection is prominently used in the FieldBuilder, which *should* be used by users instead of instantiating the BasicList directly.
type BidirectionalField ¶
type BidirectionalField interface { // IsReversible returns true if the field's underlying information element is // *not* contained in the list of irreversible information elements as per RFC // 5103. // // Note that this function is only practical for information elements described // in RFC 5103, i.e., IEs assigned by IANA, because only for those, the semantics // of reversal are well-defined by the RFC. Enterprise-specific IEs may choose // to implement their own reversal semantics, as it is the case with e.g. CERT IEs Reversible() bool // Reversed returns the field's state with regards to RFC 5103, i.e., if the // field is used to carry biflow information for the reverse direction. Note that // this is also used for returning the field's name, indicating that a field is // reversed by prepending "reversed" in front of the name. Reversed() bool }
type Boolean ¶
type Boolean struct {
// contains filtered or unexported fields
}
Boolean is the cannonic boolean data type in RFC 7011 describing boolean values. IPFIX encodes boolean as a single octet, where 0x01 equals true and 0x02 equal false. All other values should fail to decode the data type
func (*Boolean) Decode ¶
Decode takes a set of bytes (specifically, SHOULD just one) and decodes it to a boolean information element. If in contains more than one byte, Decode panics
func (*Boolean) DefaultLength ¶
func (*Boolean) IsReducedLength ¶
IsReducedLength for Booleans returns false, as booleans are not reduced-length-encodable
func (*Boolean) MarshalJSON ¶
func (*Boolean) UnmarshalJSON ¶
func (*Boolean) WithLength ¶
func (*Boolean) WithLength(length uint16) DataTypeConstructor
WithLength for Booleans returns the default constructor, as boolean abstract data types are not reduced-length encodable
type DataRecord ¶
type DataRecord struct { TemplateId uint16 `json:"template_id,omitempty"` FieldCount uint16 `json:"field_count,omitempty"` Fields []Field `json:"fields,omitempty"` // contains filtered or unexported fields }
func (*DataRecord) Clone ¶
func (d *DataRecord) Clone() DataRecord
func (*DataRecord) Length ¶
func (d *DataRecord) Length() uint16
func (*DataRecord) String ¶
func (dr *DataRecord) String() string
func (*DataRecord) UnmarshalJSON ¶
func (dr *DataRecord) UnmarshalJSON(in []byte) error
func (*DataRecord) With ¶
func (dr *DataRecord) With(t *Template) *DataRecord
type DataSet ¶
type DataSet struct { Records []DataRecord `json:"records,omitempty" yaml:"records,omitempty"` // contains filtered or unexported fields }
type DataType ¶
type DataType interface { json.Marshaler json.Unmarshaler fmt.Stringer // Type returns a string representation of the DataTypes type. // The name is used in Recover() to reconstruct a Field's DataType constructor Type() string // Length returns the actual length of the value captured by the DataType. This // is important because reduced-length and variable-length-encoded fields // will have different lengths than the spec prescribes Length() uint16 // DefaultLength returns the DataType's length as defined by the specification DefaultLength() uint16 // Decode reads a fixed number of bytes from the reader and decodes the value // data-type-dependant. Returns the first error that occurs during decoding. Decode(io.Reader) (int, error) // Encode writes a data type in IPFIX binary format to a given writer. // It returns the number of written bytes, and an error if an error occurred Encode(io.Writer) (int, error) // Value returns the internal value of the DataType. Marshalling the DataType will // only marshal the internal value to a marshallable type. Additional information // such as reduced-length encoding defined on the field and references to FieldManager // and TemplateManager are NOT marshalled. Value() interface{} // IsReducedLength indicates that a field has been constructed with // a custom, reduced-length encoding as per RFC 7011. // Therefore, when restoring a Field from a ConsolidatedField, if true, // Length() can be used to reconstruct the custom length parameter with a // FieldBuilder. IsReducedLength() bool // WithLength decorates the DataTypeConstructor with a defined length such that every // DataType created with the constructor has a predefined length. // // NOTE that WithLength does not copy any other internal properties of a DataType, e.g., // already defined list semantics for basicList, subTemplateList, or subTemplateMultiList. // The only (non-testing) usage of WithLength is currently in FieldBuilder.Complete(), which // creates fresh Fields anyways so any already decoded or otherwise initialized properties // do not apply here. All other references should probably use the provided setters instead // of decorators because of this non-copy nature of the function. WithLength(uint16) DataTypeConstructor SetLength(uint16) DataType // Clone copies all copyable values on the DataType into a fresh one for creating new instances // of DataTypes Clone() DataType // SetValue sets the internal value on the DataType. SetValue panics if v cannot be asserted to the // internal type (this type safety should be ensured in the conversion from ConsolidatedField to Field) SetValue(v any) DataType }
func NewBasicList ¶
func NewBasicList() DataType
func NewBoolean ¶
func NewBoolean() DataType
func NewDateTimeMicroseconds ¶
func NewDateTimeMicroseconds() DataType
func NewDateTimeMilliseconds ¶
func NewDateTimeMilliseconds() DataType
func NewDateTimeNanoseconds ¶
func NewDateTimeNanoseconds() DataType
func NewDateTimeSeconds ¶
func NewDateTimeSeconds() DataType
func NewDefaultSubTemplateList ¶
func NewDefaultSubTemplateList() DataType
func NewDefaultSubTemplateMultiList ¶
func NewDefaultSubTemplateMultiList() DataType
func NewFloat32 ¶
func NewFloat32() DataType
func NewFloat64 ¶
func NewFloat64() DataType
func NewIPv4Address ¶
func NewIPv4Address() DataType
func NewIPv6Address ¶
func NewIPv6Address() DataType
func NewMacAddress ¶
func NewMacAddress() DataType
func NewOctetArray ¶
func NewOctetArray() DataType
func NewSigned16 ¶
func NewSigned16() DataType
func NewSigned32 ¶
func NewSigned32() DataType
func NewSigned64 ¶
func NewSigned64() DataType
func NewSigned8 ¶
func NewSigned8() DataType
func NewUnsigned16 ¶
func NewUnsigned16() DataType
func NewUnsigned32 ¶
func NewUnsigned32() DataType
func NewUnsigned64 ¶
func NewUnsigned64() DataType
func NewUnsigned8 ¶
func NewUnsigned8() DataType
type DataTypeConstructor ¶
type DataTypeConstructor func() DataType
DataTypeConstructor is a type capturing the 0-adic constructor function for a new DataType. All DataTypes should, aside from their implementation of DataType's methods, also provide such a constructor function for unified instantiation.
Mechanisms of dependency injection can also lead DataTypes to implement decorators that return new DataTypeConstructor functions with parameters curried inside the constructor function's closure.
func DataTypeFromNumber ¶
func DataTypeFromNumber(id uint8) DataTypeConstructor
DataTypeFromNumber looks up the default constructor for each of the currently known IPFIX abstract data types (both in RFC 7011 and RFC 6313) by their IANA-assigned identifier. If an id is given that is NOT in the lookup table, DataTypeFromNumber panics. This behaviour is due to no better error handling mechanism currently existing in the call path of this function.
TODO(zoomoid): rethink if panicking is the best idea here.
func LookupConstructor ¶
func LookupConstructor(name string) DataTypeConstructor
LookupConstructor is an accessor to the private internal, but global map of currently known IPFIX abstract data types.
If no constructor is associated with the given name, LookupConstructor panics. This behavior is to be discussed and potentially amended.
func SupportedTypes ¶
func SupportedTypes() []DataTypeConstructor
SupportedTypes returns a slice containing all currently known DataType constructors.
type DateTimeMicroseconds ¶
type DateTimeMicroseconds struct {
// contains filtered or unexported fields
}
func (*DateTimeMicroseconds) Clone ¶
func (t *DateTimeMicroseconds) Clone() DataType
func (*DateTimeMicroseconds) Decode ¶
func (t *DateTimeMicroseconds) Decode(in io.Reader) (int, error)
func (*DateTimeMicroseconds) DefaultLength ¶
func (t *DateTimeMicroseconds) DefaultLength() uint16
func (*DateTimeMicroseconds) Encode ¶
func (t *DateTimeMicroseconds) Encode(w io.Writer) (int, error)
func (*DateTimeMicroseconds) IsReducedLength ¶
func (*DateTimeMicroseconds) IsReducedLength() bool
IsReducedLength for DateTimeMicroseconds returns false, as time abstract data types are not reduced-length-encodable
func (DateTimeMicroseconds) Length ¶
func (t DateTimeMicroseconds) Length() uint16
func (*DateTimeMicroseconds) MarshalJSON ¶
func (t *DateTimeMicroseconds) MarshalJSON() ([]byte, error)
func (*DateTimeMicroseconds) SetLength ¶
func (t *DateTimeMicroseconds) SetLength(length uint16) DataType
func (*DateTimeMicroseconds) SetValue ¶
func (t *DateTimeMicroseconds) SetValue(v any) DataType
func (*DateTimeMicroseconds) String ¶
func (t *DateTimeMicroseconds) String() string
func (DateTimeMicroseconds) Type ¶
func (t DateTimeMicroseconds) Type() string
func (*DateTimeMicroseconds) UnmarshalJSON ¶
func (t *DateTimeMicroseconds) UnmarshalJSON(in []byte) error
func (*DateTimeMicroseconds) Value ¶
func (t *DateTimeMicroseconds) Value() interface{}
func (*DateTimeMicroseconds) WithLength ¶
func (*DateTimeMicroseconds) WithLength(length uint16) DataTypeConstructor
WithLength for DateTimeMicroseconds returns the default constructor, as time abstract data types are not reduced-length-encodable
type DateTimeMilliseconds ¶
type DateTimeMilliseconds struct {
// contains filtered or unexported fields
}
func (*DateTimeMilliseconds) Clone ¶
func (t *DateTimeMilliseconds) Clone() DataType
func (*DateTimeMilliseconds) Decode ¶
func (t *DateTimeMilliseconds) Decode(in io.Reader) (int, error)
func (*DateTimeMilliseconds) DefaultLength ¶
func (t *DateTimeMilliseconds) DefaultLength() uint16
func (*DateTimeMilliseconds) Encode ¶
func (t *DateTimeMilliseconds) Encode(w io.Writer) (int, error)
func (*DateTimeMilliseconds) IsReducedLength ¶
func (*DateTimeMilliseconds) IsReducedLength() bool
IsReducedLength for DateTimeMilliseconds returns false, as time abstract data types are not reduced-length-encodable
func (DateTimeMilliseconds) Length ¶
func (t DateTimeMilliseconds) Length() uint16
func (*DateTimeMilliseconds) MarshalJSON ¶
func (t *DateTimeMilliseconds) MarshalJSON() ([]byte, error)
func (*DateTimeMilliseconds) SetLength ¶
func (t *DateTimeMilliseconds) SetLength(length uint16) DataType
func (*DateTimeMilliseconds) SetValue ¶
func (t *DateTimeMilliseconds) SetValue(v any) DataType
func (*DateTimeMilliseconds) String ¶
func (t *DateTimeMilliseconds) String() string
func (DateTimeMilliseconds) Type ¶
func (t DateTimeMilliseconds) Type() string
func (*DateTimeMilliseconds) UnmarshalJSON ¶
func (t *DateTimeMilliseconds) UnmarshalJSON(in []byte) error
func (*DateTimeMilliseconds) Value ¶
func (t *DateTimeMilliseconds) Value() interface{}
func (*DateTimeMilliseconds) WithLength ¶
func (*DateTimeMilliseconds) WithLength(length uint16) DataTypeConstructor
WithLength for DateTimeMilliseconds returns the default constructor, as time abstract data types are not reduced-length-encodable
type DateTimeNanoseconds ¶
type DateTimeNanoseconds struct {
// contains filtered or unexported fields
}
func (*DateTimeNanoseconds) Clone ¶
func (t *DateTimeNanoseconds) Clone() DataType
func (*DateTimeNanoseconds) Decode ¶
func (t *DateTimeNanoseconds) Decode(in io.Reader) (int, error)
func (*DateTimeNanoseconds) DefaultLength ¶
func (t *DateTimeNanoseconds) DefaultLength() uint16
func (*DateTimeNanoseconds) IsReducedLength ¶
func (*DateTimeNanoseconds) IsReducedLength() bool
IsReducedLength for DateTimeNanoseconds returns false, as time abstract data types are not reduced-length-encodable
func (DateTimeNanoseconds) Length ¶
func (t DateTimeNanoseconds) Length() uint16
func (*DateTimeNanoseconds) MarshalJSON ¶
func (t *DateTimeNanoseconds) MarshalJSON() ([]byte, error)
func (*DateTimeNanoseconds) SetLength ¶
func (t *DateTimeNanoseconds) SetLength(length uint16) DataType
func (*DateTimeNanoseconds) SetValue ¶
func (t *DateTimeNanoseconds) SetValue(v any) DataType
func (*DateTimeNanoseconds) String ¶
func (t *DateTimeNanoseconds) String() string
func (*DateTimeNanoseconds) Type ¶
func (t *DateTimeNanoseconds) Type() string
func (*DateTimeNanoseconds) UnmarshalJSON ¶
func (t *DateTimeNanoseconds) UnmarshalJSON(in []byte) error
func (*DateTimeNanoseconds) Value ¶
func (t *DateTimeNanoseconds) Value() interface{}
func (*DateTimeNanoseconds) WithLength ¶
func (*DateTimeNanoseconds) WithLength(length uint16) DataTypeConstructor
WithLength for DateTimeNanoseconds returns the default constructor, as time abstract data types are not reduced-length-encodable
type DateTimeSeconds ¶
type DateTimeSeconds struct {
// contains filtered or unexported fields
}
func (*DateTimeSeconds) Clone ¶
func (t *DateTimeSeconds) Clone() DataType
func (*DateTimeSeconds) DefaultLength ¶
func (t *DateTimeSeconds) DefaultLength() uint16
func (*DateTimeSeconds) IsReducedLength ¶
func (*DateTimeSeconds) IsReducedLength() bool
IsReducedLength for DateTimeSeconds returns false, as time abstract data types are not reduced-length-encodable
func (*DateTimeSeconds) Length ¶
func (t *DateTimeSeconds) Length() uint16
func (*DateTimeSeconds) MarshalJSON ¶
func (t *DateTimeSeconds) MarshalJSON() ([]byte, error)
func (*DateTimeSeconds) SetLength ¶
func (t *DateTimeSeconds) SetLength(length uint16) DataType
func (*DateTimeSeconds) SetValue ¶
func (t *DateTimeSeconds) SetValue(v any) DataType
func (*DateTimeSeconds) String ¶
func (t *DateTimeSeconds) String() string
func (*DateTimeSeconds) Type ¶
func (*DateTimeSeconds) Type() string
func (*DateTimeSeconds) UnmarshalJSON ¶
func (t *DateTimeSeconds) UnmarshalJSON(in []byte) error
func (*DateTimeSeconds) Value ¶
func (t *DateTimeSeconds) Value() interface{}
func (*DateTimeSeconds) WithLength ¶
func (*DateTimeSeconds) WithLength(length uint16) DataTypeConstructor
WithLength for DateTimeSeconds returns the default constructor, as time abstract data types are not reduced-length-encodable
type DecayingEphemeralCache ¶
type DecayingEphemeralCache struct {
// contains filtered or unexported fields
}
func (*DecayingEphemeralCache) Add ¶
func (ts *DecayingEphemeralCache) Add(ctx context.Context, key TemplateKey, template *Template) error
func (*DecayingEphemeralCache) Close ¶
func (ts *DecayingEphemeralCache) Close(context.Context) error
func (*DecayingEphemeralCache) Delete ¶
func (t *DecayingEphemeralCache) Delete(ctx context.Context, key TemplateKey) error
func (*DecayingEphemeralCache) Get ¶
func (ts *DecayingEphemeralCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
func (*DecayingEphemeralCache) GetAll ¶
func (ts *DecayingEphemeralCache) GetAll(ctx context.Context) map[TemplateKey]*Template
func (*DecayingEphemeralCache) MarshalJSON ¶
func (ts *DecayingEphemeralCache) MarshalJSON() ([]byte, error)
func (*DecayingEphemeralCache) Name ¶
func (ts *DecayingEphemeralCache) Name() string
func (*DecayingEphemeralCache) SetTimeout ¶
func (ts *DecayingEphemeralCache) SetTimeout(d time.Duration)
SetTimeout updates the internal duration stored for calculating deadlines on addition of (new) templates. NOTE that SetTimeout does not alter any existing deadlines, i.e., if the timeout duration is changed during runtime, and there are already templates in the cache, a longer or shorter timeout duration does NOT affect the deadline of those templates, only new ones
func (*DecayingEphemeralCache) Start ¶
func (ts *DecayingEphemeralCache) Start(ctx context.Context) error
func (*DecayingEphemeralCache) Type ¶
func (ts *DecayingEphemeralCache) Type() string
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder is instantiated with a fieldManager and a templateManager such that it can decode IPFIX packets into Records containing fields and additionally learn new fields and templates.
func NewDecoder ¶
func NewDecoder(templates TemplateCache, fields FieldCache, opts ...DecoderOptions) *Decoder
NewDecoder creates a new Decoder for a given template cache and field manager
func (*Decoder) Decode ¶
Decode takes payload as a buffer and consumes it to construct an IPFIX packet containing records containing decoded fields.
func (*Decoder) WithCompletionHook ¶
type DecoderOptions ¶
type DecoderOptions struct {
OmitRFC5610Records bool
}
func (*DecoderOptions) Merge ¶
func (o *DecoderOptions) Merge(opts ...DecoderOptions)
type EphemeralCache ¶
type EphemeralCache struct {
// contains filtered or unexported fields
}
EphemeralCache is the most basic of in-memory caches. It is memory-safe by using a Read-Write mutex on all accessing functions. It does not expire entries automatically and does not persist anything on disk, nor does it support recovery
func (*EphemeralCache) Add ¶
func (ts *EphemeralCache) Add(ctx context.Context, key TemplateKey, template *Template) error
func (*EphemeralCache) Delete ¶
func (ts *EphemeralCache) Delete(ctx context.Context, key TemplateKey) error
func (*EphemeralCache) Get ¶
func (ts *EphemeralCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
func (*EphemeralCache) GetAll ¶
func (ts *EphemeralCache) GetAll(ctx context.Context) map[TemplateKey]*Template
func (*EphemeralCache) Initialize ¶
func (ts *EphemeralCache) Initialize(context.Context) error
func (*EphemeralCache) MarshalJSON ¶
func (ts *EphemeralCache) MarshalJSON() ([]byte, error)
func (*EphemeralCache) Name ¶
func (ts *EphemeralCache) Name() string
func (*EphemeralCache) Prepare ¶
func (ts *EphemeralCache) Prepare() error
func (*EphemeralCache) Type ¶
func (ts *EphemeralCache) Type() string
type EphemeralFieldCache ¶
type EphemeralFieldCache struct {
// contains filtered or unexported fields
}
func (*EphemeralFieldCache) Add ¶
func (fm *EphemeralFieldCache) Add(ctx context.Context, element InformationElement) error
func (*EphemeralFieldCache) Delete ¶
func (fm *EphemeralFieldCache) Delete(ctx context.Context, key FieldKey) error
func (*EphemeralFieldCache) Get ¶
func (fm *EphemeralFieldCache) Get(ctx context.Context, key FieldKey) (*InformationElement, error)
func (*EphemeralFieldCache) GetAll ¶
func (fm *EphemeralFieldCache) GetAll(ctx context.Context) map[FieldKey]*InformationElement
func (*EphemeralFieldCache) GetAllBuilders ¶
func (fm *EphemeralFieldCache) GetAllBuilders(ctx context.Context) map[FieldKey]*FieldBuilder
func (*EphemeralFieldCache) GetBuilder ¶
func (fm *EphemeralFieldCache) GetBuilder(ctx context.Context, key FieldKey) (*FieldBuilder, error)
func (*EphemeralFieldCache) MarshalJSON ¶
func (fm *EphemeralFieldCache) MarshalJSON() ([]byte, error)
type Field ¶
type Field interface { // Id returns the field id as defined Id() uint16 // Name returns the name of the field Name() string // Value returns the underlying data type Value() DataType // SetValue sets the value on the internal DataType stored in the field SetValue(v any) Field // Type returns a string representation of the underlying DataType Type() string // PEN returns the private enterprise number of the field, if set, as a uint32 pointer, // and otherwise nil PEN() uint32 // Constructor returns the data type constructor defined for the field Constructor() DataTypeConstructor // Length returns the semantically-aware length of the field Length() uint16 // ObservationDomainId returns the ID bound to the field from the builder For // fields whose underlying data types are reliant on this ID, i.e., // SubTemplateList and SubTemplateMultiList, this is required or otherwise // decoding will not work correctly. ObservationDomainId() uint32 // Prototype returns the field's IE specification, i.e., the prototype of the field. // This can be used for cloning and copying of the field while preserving semantics Prototype() *InformationElement // Lift converts a field to a variable-length field, indicating this to the // data type constructor Lift() *VariableLengthField // Decode creates a DataType from the supplied constructor and decodes the // value from the Reader Decode(io.Reader) (int, error) // Encode writes a field in IPFIX binary format to a given writer. // It returns the number of written bytes, and an error if an error occurred Encode(io.Writer) (int, error) // Clone clones a field entirely by-value such that changing a Field in a data // record which is also used in a template record does not cause side effects. // The only thing copied by- reference are FieldManager and TemplateManager // instances. Clone() Field // SetScoped is the setter to be used when the field is a scope field. SetScoped() Field // IsScope returns true if the field is set to be a scope field. This is // useful when encoding to a map structure in e.g. encoding/libfds, as scope // fields with the same name would be lost due to map key collisions. // Therefore we use this boolean to compute a key that is resistent to the // collision. IsScope() bool BidirectionalField json.Marshaler json.Unmarshaler fmt.Stringer // contains filtered or unexported methods }
Field defines the interface for both types of IPFIX fields, either fixed-length or variable-length fields. They share most of their logic, except for encoding and decoding, for which the Encode and Decode methods transparently handle their underlying nature. Fixed-length fields are intuitively simpler, as their length and data type is defined by templates. Variable-length fields encode their data length in the first 1 or 3 bytes (short and long format).
The interface also declares methods for converting a fixed-length field to a variable-length field, using Lift(). Though this is practically never done in plain IPFIX, it is convenient to have such a function in user space for conversion between both underlying field types. Note that the reverse direction, converting a variable-length field to a fixed-length field is NOT possible.
type FieldBuilder ¶
type FieldBuilder struct {
// contains filtered or unexported fields
}
func NewFieldBuilder ¶
func NewFieldBuilder(ie *InformationElement) *FieldBuilder
func NewUnassignedFieldBuilder ¶
func NewUnassignedFieldBuilder(id uint16) *FieldBuilder
NewUnassignedFieldBuilder may be used to quickly create a FieldBuilder from only an ID. The resulting builder and the underlying IE have no further defined fields, with the name being "unassigned", the enterprise ID being 0, the data type being the default IPFIX data type for unknown IEs, octetArray, undefined IE semantics and undefined IE status.
func (*FieldBuilder) Complete ¶
func (b *FieldBuilder) Complete() Field
func (*FieldBuilder) GetIE ¶
func (b *FieldBuilder) GetIE() *InformationElement
func (*FieldBuilder) MarshalJSON ¶
func (b *FieldBuilder) MarshalJSON() ([]byte, error)
func (*FieldBuilder) SetFieldManager ¶
func (b *FieldBuilder) SetFieldManager(fieldManager FieldCache) *FieldBuilder
func (*FieldBuilder) SetLength ¶
func (b *FieldBuilder) SetLength(length uint16) *FieldBuilder
SetLength sets the field's length. This handles 0xFF as variable
func (*FieldBuilder) SetObservationDomain ¶
func (b *FieldBuilder) SetObservationDomain(id uint32) *FieldBuilder
func (*FieldBuilder) SetPEN ¶
func (b *FieldBuilder) SetPEN(pen uint32) *FieldBuilder
SetPEN sets the field's Private Enterprise Number
func (*FieldBuilder) SetReversed ¶
func (b *FieldBuilder) SetReversed(isReverse bool) *FieldBuilder
func (*FieldBuilder) SetTemplateManager ¶
func (b *FieldBuilder) SetTemplateManager(templateManager TemplateCache) *FieldBuilder
func (*FieldBuilder) UnmarshalJSON ¶
func (b *FieldBuilder) UnmarshalJSON(in []byte) error
type FieldCache ¶
type FieldCache interface { // GetBuilder retrieves a field builder instance from the cache for creating // fields during decoding. // // If the field is not found in the cache, a new UnassignedFieldBuilder is // returned with the information embedded in the FieldKey // // If an error occurs during retrieval of the field, an error is returned, // and the FieldBuilder pointer is nil GetBuilder(context.Context, FieldKey) (*FieldBuilder, error) // Add adds a new Information Element definition to the field cache. // // The canonic implementation of FieldCache immediately creates FieldBuilder // instances to return on Get(), however this is up to implementor. // // If adding the new IE fails, an error is returned. Add(context.Context, InformationElement) error // Delete removes a field identified by a FieldKey from the cache. // // The canonic implementation of FieldCache stores both information elements given during Add(), // and the instantiated FieldBuilder types, and cleans up both at once. Delete(context.Context, FieldKey) error // Get returns the information element that defines a field currently in the cache. // // Get returns an error if no element with the FieldKey is stored in the cache. // // Get returns errors that occur during retrieval of the information element. Get(context.Context, FieldKey) (*InformationElement, error) // GetAll returns a map of FieldBuilders for all fields currently stored in the cache. // If no fields are stored in the cache, the map is empty. GetAllBuilders(context.Context) map[FieldKey]*FieldBuilder // GetAll returns a map of InformationElements of all the fields stored in the cache. // If no information elements were added to the cache prior to the call, the map is empty. GetAll(context.Context) map[FieldKey]*InformationElement json.Marshaler }
FieldCache is the interface that all, both ephemeral and persistent field caches need to implement. By default, this does not include methods for handling stateful FieldCaches, those should be provided on the explicit types. See etcd.FieldCache for such an implementation.
func NewEphemeralFieldCache ¶
func NewEphemeralFieldCache(templateManager TemplateCache) FieldCache
func NewIANAFieldManager ¶ added in v0.3.0
func NewIANAFieldManager(templateManager TemplateCache) FieldCache
NewIANAFieldManager is a utility for creating field managers with initialized IANA fields quickly, e.g. for unit testing.
NewIANAFieldManager panics if failing to add an IE to the cache.
type FixedLengthField ¶
type FixedLengthField struct {
// contains filtered or unexported fields
}
func (*FixedLengthField) Clone ¶
func (f *FixedLengthField) Clone() Field
func (*FixedLengthField) Constructor ¶
func (f *FixedLengthField) Constructor() DataTypeConstructor
func (*FixedLengthField) Id ¶
func (f *FixedLengthField) Id() uint16
func (*FixedLengthField) IsScope ¶
func (f *FixedLengthField) IsScope() bool
func (*FixedLengthField) Length ¶
func (f *FixedLengthField) Length() uint16
func (*FixedLengthField) Lift ¶
func (f *FixedLengthField) Lift() *VariableLengthField
func (*FixedLengthField) MarshalJSON ¶
func (f *FixedLengthField) MarshalJSON() ([]byte, error)
func (*FixedLengthField) Name ¶
func (f *FixedLengthField) Name() string
func (*FixedLengthField) ObservationDomainId ¶
func (f *FixedLengthField) ObservationDomainId() uint32
func (*FixedLengthField) PEN ¶
func (f *FixedLengthField) PEN() uint32
func (*FixedLengthField) Prototype ¶
func (f *FixedLengthField) Prototype() *InformationElement
func (*FixedLengthField) Reversed ¶
func (f *FixedLengthField) Reversed() bool
func (*FixedLengthField) Reversible ¶
func (f *FixedLengthField) Reversible() bool
func (*FixedLengthField) SetScoped ¶
func (f *FixedLengthField) SetScoped() Field
func (*FixedLengthField) SetValue ¶
func (f *FixedLengthField) SetValue(v any) Field
func (*FixedLengthField) String ¶ added in v0.3.0
func (f *FixedLengthField) String() string
func (*FixedLengthField) Type ¶
func (f *FixedLengthField) Type() string
func (*FixedLengthField) UnmarshalJSON ¶
func (f *FixedLengthField) UnmarshalJSON(in []byte) error
func (*FixedLengthField) Value ¶
func (f *FixedLengthField) Value() DataType
Value returns the fields value. If value is nil, i.e., has not yet been assigned, Value returns the zero value of the DataType constructor.
type Float32 ¶
type Float32 struct {
// contains filtered or unexported fields
}
func (*Float32) DefaultLength ¶
func (*Float32) IsReducedLength ¶
func (*Float32) MarshalJSON ¶
func (*Float32) UnmarshalJSON ¶
func (*Float32) WithLength ¶
func (*Float32) WithLength(length uint16) DataTypeConstructor
type Float64 ¶
type Float64 struct {
// contains filtered or unexported fields
}
func (*Float64) DefaultLength ¶
func (*Float64) IsReducedLength ¶
func (*Float64) MarshalJSON ¶
func (*Float64) UnmarshalJSON ¶
func (*Float64) WithLength ¶
func (*Float64) WithLength(length uint16) DataTypeConstructor
type IPv4Address ¶
type IPv4Address struct {
// contains filtered or unexported fields
}
func (*IPv4Address) Clone ¶
func (t *IPv4Address) Clone() DataType
func (*IPv4Address) DefaultLength ¶
func (*IPv4Address) DefaultLength() uint16
func (*IPv4Address) IsReducedLength ¶
func (*IPv4Address) IsReducedLength() bool
func (*IPv4Address) Length ¶
func (t *IPv4Address) Length() uint16
func (*IPv4Address) MarshalJSON ¶
func (t *IPv4Address) MarshalJSON() ([]byte, error)
func (*IPv4Address) SetLength ¶
func (t *IPv4Address) SetLength(length uint16) DataType
func (*IPv4Address) SetValue ¶
func (t *IPv4Address) SetValue(v any) DataType
func (*IPv4Address) String ¶
func (t *IPv4Address) String() string
func (*IPv4Address) Type ¶
func (*IPv4Address) Type() string
func (*IPv4Address) UnmarshalJSON ¶
func (t *IPv4Address) UnmarshalJSON(in []byte) error
func (*IPv4Address) Value ¶
func (t *IPv4Address) Value() interface{}
func (*IPv4Address) WithLength ¶
func (*IPv4Address) WithLength(length uint16) DataTypeConstructor
type IPv6Address ¶
type IPv6Address struct {
// contains filtered or unexported fields
}
func (*IPv6Address) Clone ¶
func (t *IPv6Address) Clone() DataType
func (*IPv6Address) DefaultLength ¶
func (t *IPv6Address) DefaultLength() uint16
func (*IPv6Address) IsReducedLength ¶
func (*IPv6Address) IsReducedLength() bool
func (IPv6Address) Length ¶
func (t IPv6Address) Length() uint16
func (*IPv6Address) MarshalJSON ¶
func (t *IPv6Address) MarshalJSON() ([]byte, error)
func (*IPv6Address) SetLength ¶
func (t *IPv6Address) SetLength(length uint16) DataType
func (*IPv6Address) SetValue ¶
func (t *IPv6Address) SetValue(v any) DataType
func (*IPv6Address) String ¶
func (t *IPv6Address) String() string
func (IPv6Address) Type ¶
func (t IPv6Address) Type() string
func (*IPv6Address) UnmarshalJSON ¶
func (t *IPv6Address) UnmarshalJSON(in []byte) error
func (*IPv6Address) Value ¶
func (t *IPv6Address) Value() interface{}
func (*IPv6Address) WithLength ¶
func (t *IPv6Address) WithLength(length uint16) DataTypeConstructor
type InformationElement ¶
type InformationElement struct { Constructor DataTypeConstructor `json:"-" yaml:"-"` Id uint16 `json:"id,omitempty" yaml:"id,omitempty"` Name string `json:"name,omitempty" yaml:"name,omitempty"` EnterpriseId uint32 `json:"pen,omitempty" yaml:"pen,omitempty"` Semantics semantics.Semantic `json:"semantics,omitempty" yaml:"semantics,omitempty"` Status status.Status `json:"status,omitempty" yaml:"status,omitempty"` Type *string `json:"type,omitempty" yaml:"type,omitempty"` Description *string `json:"description,omitempty" yaml:"description,omitempty"` Units *string `json:"units,omitempty" yaml:"units,omitempty"` Range *InformationElementRange `json:"range,omitempty" yaml:"range,omitempty"` AdditionalInformation *string `json:"additional_information,omitempty" yaml:"additionalInformation,omitempty"` Reference *string `json:"reference,omitempty" yaml:"reference,omitempty"` Revision *int `json:"revision,omitempty" yaml:"revision,omitempty"` Date *string `json:"date,omitempty" yaml:"date,omitempty"` }
func (*InformationElement) Clone ¶
func (i *InformationElement) Clone() InformationElement
func (InformationElement) String ¶
func (i InformationElement) String() string
func (*InformationElement) UnmarshalJSON ¶
func (i *InformationElement) UnmarshalJSON(in []byte) error
type InformationElementRange ¶
type InformationElementRange struct { Low int `json:"low,omitempty" yaml:"low,omitempty"` High int `json:"high,omitempty" yaml:"high,omitempty"` }
func (*InformationElementRange) Clone ¶
func (i *InformationElementRange) Clone() *InformationElementRange
type ListSemantic ¶
type ListSemantic uint8
ListSemantic is the type capturing the IANA-assigned list semantics as defined by RFC 6313
const ( // The "noneOf" structured data type semantic specifies that none of the // elements are actual properties of the Data Record. // // For example, a mediator might want to report to a Collector that a // specific Flow is suspicious, but that it checked already that this // Flow does not belong to the attack type 1, attack type 2, or attack // type 3. So this Flow might need some further inspection. In such a // case, the mediator would report the Flow Record with a basicList // composed of (attack type 1, attack type 2, attack type 3) and the // respective structured data type semantic of "noneOf". SemanticNoneOf ListSemantic = 0 // The "exactlyOneOf" structured data type semantic specifies that only // a single element from the structured data is an actual property of // the Data Record. This is equivalent to a logical XOR operation. SemanticExactlyOneOf ListSemantic = 1 // The "oneOrMoreOf" structured data type semantic specifies that one or // more elements from the list in the structured data are actual // properties of the Data Record. This is equivalent to a logical OR // operation. SemanticOneOrMoreOf ListSemantic = 2 // The "allOf" structured data type semantic specifies that all of the // list elements from the structured data are actual properties of the // Data Record. // // For example, if a Record contains a basicList of outgoing interfaces // with the "allOf" semantic, then the observed Flow is typically a // multicast Flow where each packet in the Flow has been replicated to // each outgoing interface in the basicList. SemanticAllOf ListSemantic = 3 // The "ordered" structured data type semantic specifies that elements // from the list in the structured data are ordered. // // For example, an Exporter might want to export the AS10 AS20 AS30 AS40 // BGP AS-PATH. In such a case, the Exporter would report a basicList // composed of (AS10, AS20, AS30, AS40) and the respective structured // data type semantic of "ordered". SemanticOrdered ListSemantic = 4 // For example, a mediator that wants to translate IPFIX [RFC5101] into // the export of structured data according to the specifications in this // document doesn't know what the semantic is; it can only guess, as the // IPFIX specifications [RFC5101] does not contain any semantic. // Therefore, the mediator should use the "undefined" semantic. SemanticUndefined ListSemantic = 255 )
func (ListSemantic) MarshalText ¶
func (s ListSemantic) MarshalText() ([]byte, error)
MarshalText implements encoding.Marshaler to convert a ListSemantic instance into a string representation
func (ListSemantic) String ¶
func (s ListSemantic) String() string
func (*ListSemantic) UnmarshalText ¶
func (s *ListSemantic) UnmarshalText(in []byte) error
UnmarshalText implements encoding.Unmarshaler to convert the string representation of a ListSemantic into its proper type
type MacAddress ¶
type MacAddress struct {
// contains filtered or unexported fields
}
func (*MacAddress) Clone ¶
func (t *MacAddress) Clone() DataType
func (*MacAddress) DefaultLength ¶
func (*MacAddress) DefaultLength() uint16
func (*MacAddress) IsReducedLength ¶
func (*MacAddress) IsReducedLength() bool
func (*MacAddress) Length ¶
func (t *MacAddress) Length() uint16
func (*MacAddress) MarshalJSON ¶
func (t *MacAddress) MarshalJSON() ([]byte, error)
func (*MacAddress) SetLength ¶
func (t *MacAddress) SetLength(length uint16) DataType
func (*MacAddress) SetValue ¶
func (t *MacAddress) SetValue(v any) DataType
func (*MacAddress) String ¶
func (t *MacAddress) String() string
func (*MacAddress) Type ¶
func (*MacAddress) Type() string
func (*MacAddress) UnmarshalJSON ¶
func (t *MacAddress) UnmarshalJSON(in []byte) error
func (*MacAddress) Value ¶
func (t *MacAddress) Value() interface{}
func (*MacAddress) WithLength ¶
func (t *MacAddress) WithLength(length uint16) DataTypeConstructor
type Message ¶
type Message struct { Version uint16 `json:"version,omitempty" yaml:"version,omitempty"` Length uint16 `json:"length,omitempty" yaml:"length,omitempty"` ExportTime uint32 `json:"export_time,omitempty" yaml:"exportTime,omitempty"` SequenceNumber uint32 `json:"sequence_number,omitempty" yaml:"sequenceNumber,omitempty"` ObservationDomainId uint32 `json:"observation_domain_id,omitempty" yaml:"observationDomainId,omitempty"` Sets []Set `json:"sets,omitempty" yaml:"sets,omitempty"` }
type OctetArray ¶
type OctetArray struct {
// contains filtered or unexported fields
}
func (*OctetArray) Clone ¶
func (t *OctetArray) Clone() DataType
func (*OctetArray) DefaultLength ¶
func (*OctetArray) DefaultLength() uint16
func (*OctetArray) IsReducedLength ¶
func (*OctetArray) IsReducedLength() bool
IsReducedLength for OctetArray abstract data types returns false, as reduced-length encoding for arrays of bytes has no semantic value.
func (*OctetArray) Length ¶
func (t *OctetArray) Length() uint16
func (*OctetArray) MarshalJSON ¶
func (t *OctetArray) MarshalJSON() ([]byte, error)
func (*OctetArray) SetLength ¶
func (t *OctetArray) SetLength(length uint16) DataType
func (*OctetArray) SetValue ¶
func (t *OctetArray) SetValue(v any) DataType
func (*OctetArray) String ¶
func (t *OctetArray) String() string
func (*OctetArray) Type ¶
func (*OctetArray) Type() string
func (*OctetArray) UnmarshalJSON ¶
func (t *OctetArray) UnmarshalJSON(in []byte) error
This overwrites the canonic UnmarshalJSON implementation for byte slices
func (*OctetArray) Value ¶
func (t *OctetArray) Value() interface{}
func (*OctetArray) WithLength ¶
func (*OctetArray) WithLength(length uint16) DataTypeConstructor
WithLength returns a DataTypeConstructor function with a fixed, given length
type OptionsTemplateRecord ¶
type OptionsTemplateRecord struct { TemplateId uint16 `json:"templateId,omitempty" yaml:"templateId,omitempty"` FieldCount uint16 `json:"fieldCount,omitempty" yaml:"fieldCount,omitempty"` ScopeFieldCount uint16 `json:"scopeFieldCount,omitempty" yaml:"scopeFieldCount,omitempty"` Scopes []Field `json:"scopes,omitempty" yaml:"scopes,omitempty"` Options []Field `json:"options,omitempty" yaml:"options,omitempty"` // contains filtered or unexported fields }
func (*OptionsTemplateRecord) Decode ¶
func (otr *OptionsTemplateRecord) Decode(r io.Reader) (n int, err error)
func (*OptionsTemplateRecord) Encode ¶
func (otr *OptionsTemplateRecord) Encode(w io.Writer) (n int, err error)
func (*OptionsTemplateRecord) Id ¶
func (otr *OptionsTemplateRecord) Id() uint16
func (*OptionsTemplateRecord) Length ¶
func (otr *OptionsTemplateRecord) Length() uint16
func (*OptionsTemplateRecord) MarshalJSON ¶
func (otr *OptionsTemplateRecord) MarshalJSON() ([]byte, error)
func (*OptionsTemplateRecord) String ¶ added in v0.3.0
func (otr *OptionsTemplateRecord) String() string
func (*OptionsTemplateRecord) Type ¶
func (otr *OptionsTemplateRecord) Type() string
func (*OptionsTemplateRecord) UnmarshalJSON ¶
func (otr *OptionsTemplateRecord) UnmarshalJSON(in []byte) error
type OptionsTemplateSet ¶
type OptionsTemplateSet struct { Records []OptionsTemplateRecord `json:"records,omitempty" yaml:"records,omitempty"` // contains filtered or unexported fields }
func (*OptionsTemplateSet) Decode ¶
func (d *OptionsTemplateSet) Decode(r io.Reader) (n int, err error)
func (*OptionsTemplateSet) Encode ¶
func (d *OptionsTemplateSet) Encode(w io.Writer) (n int, err error)
func (*OptionsTemplateSet) Length ¶
func (d *OptionsTemplateSet) Length() int
func (*OptionsTemplateSet) String ¶ added in v0.3.0
func (d *OptionsTemplateSet) String() string
type PersistentCache ¶
type PersistentCache struct {
// contains filtered or unexported fields
}
func (*PersistentCache) Add ¶
func (t *PersistentCache) Add(ctx context.Context, key TemplateKey, template *Template) error
func (*PersistentCache) Delete ¶
func (t *PersistentCache) Delete(ctx context.Context, key TemplateKey) error
func (*PersistentCache) Get ¶
func (t *PersistentCache) Get(ctx context.Context, key TemplateKey) (*Template, error)
func (*PersistentCache) GetAll ¶
func (t *PersistentCache) GetAll(ctx context.Context) map[TemplateKey]*Template
func (*PersistentCache) Initialize ¶
func (t *PersistentCache) Initialize(ctx context.Context) error
func (*PersistentCache) MarshalJSON ¶
func (t *PersistentCache) MarshalJSON() ([]byte, error)
func (*PersistentCache) Name ¶
func (t *PersistentCache) Name() string
func (*PersistentCache) Prepare ¶
func (t *PersistentCache) Prepare() error
func (*PersistentCache) Start ¶
func (t *PersistentCache) Start(ctx context.Context) error
Start implements manager.Runnable, to handle the lifecycle of the persistent cache
func (*PersistentCache) Type ¶
func (t *PersistentCache) Type() string
type RawMessage ¶ added in v0.3.0
type RawMessage []byte
func ReadFull ¶ added in v0.3.0
func ReadFull(f io.Reader) ([]RawMessage, error)
ReadFull consumes an entire io.Reader of a file containing IPFIX File Format messages and returns those messages as byte slices sliced up in a wrapping slice. Different to the IPFIXFileReader implementation this is generally synchronous.
Notably, if an error occurs during reading of messages, where err != io.EOF, the returned []RawMessage slice is nil, while the error propagates the returned error. This is different to the behaviour of the IPFIXFileReader implementation. Additionally, io.EOF errors are NOT propagated, i.e., ReadFull just returns the []RawMessage slice on occurence of an EOF.
decoder := ipfix.NewDecoder(...) file, _ := os.Open("flow_records.ipfix") msgs, err := ipfix.ReadFull(file) if err != nil { log.Fatal(err) } for _, msg := range msgs { n, err := decoder.Decode(bytes.NewBuffer(msg)) if err != nil { log.Fatal(err) } // Do anything with the decoded message afterwards }
type Set ¶
type Set struct { SetHeader `json:",inline" yaml:",inline"` Kind string `json:"kind,omitempty" yaml:"kind,omitempty"` Set set `json:"flow_set,omitempty"` }
func (*Set) MarshalJSON ¶
func (*Set) UnmarshalJSON ¶
type SetHeader ¶
type Signed16 ¶
type Signed16 struct {
// contains filtered or unexported fields
}
func (*Signed16) DefaultLength ¶
func (*Signed16) IsReducedLength ¶
func (*Signed16) MarshalJSON ¶
func (*Signed16) UnmarshalJSON ¶
func (*Signed16) WithLength ¶
func (t *Signed16) WithLength(length uint16) DataTypeConstructor
type Signed32 ¶
type Signed32 struct {
// contains filtered or unexported fields
}
func (*Signed32) DefaultLength ¶
func (*Signed32) IsReducedLength ¶
func (*Signed32) MarshalJSON ¶
func (*Signed32) UnmarshalJSON ¶
func (*Signed32) WithLength ¶
func (t *Signed32) WithLength(length uint16) DataTypeConstructor
type Signed64 ¶
type Signed64 struct {
// contains filtered or unexported fields
}
func (*Signed64) DefaultLength ¶
func (*Signed64) IsReducedLength ¶
func (*Signed64) MarshalJSON ¶
func (*Signed64) UnmarshalJSON ¶
func (*Signed64) WithLength ¶
func (t *Signed64) WithLength(length uint16) DataTypeConstructor
type Signed8 ¶
type Signed8 struct {
// contains filtered or unexported fields
}
func (*Signed8) DefaultLength ¶
func (*Signed8) IsReducedLength ¶
func (*Signed8) MarshalJSON ¶
func (*Signed8) UnmarshalJSON ¶
func (*Signed8) WithLength ¶
func (*Signed8) WithLength(length uint16) DataTypeConstructor
type StatefulTemplateCache ¶
type StatefulTemplateCache interface { TemplateCache // Start starts a stateful template cache. This is for example used in caches requiring a stateful connection // to a database/KV store like the etcd addon. // // These start/stop semantics are "leftovers" from the asynchronous architecture of FlowPlane from which this // library was factored out from. They might be removed in future (breaking) updates, as state management is // generally not the task of this library and thus usage and surface of these methods should be little. // // Start's behavior is to block indefinitely during the lifecycle of the template cache, which means that it is // best used *deferred*, either directly via a goroutine or via a lifecycle management component that starts // objects implementing such Start hooks. This is useful if you have many moving (read: concurrent) parts and are // using channels to pass data betweeen those components. Examples for this are asynchronous Apache Kafka producers, // which is the original setup how the decoder is used in FlowPlane. Start(context.Context) error // Close tears down any stateful component of a template store. E.g., this is used in the persistent template // cache to write the templates to disk before shutting down. Close(context.Context) error }
func NewDefaultEphemeralCache ¶
func NewDefaultEphemeralCache() StatefulTemplateCache
NewBasicTemplateCache creates a new in-memory template cache that lives for the lifetime of the caller
func NewDefaultPersistentCache ¶
func NewDefaultPersistentCache(file *os.File, fieldCache FieldCache, templateCache StatefulTemplateCache) StatefulTemplateCache
func NewNamedEphemeralCache ¶
func NewNamedEphemeralCache(name string) StatefulTemplateCache
func NewNamedPersistentCache ¶
func NewNamedPersistentCache(name string, file *os.File, fieldCache FieldCache, templateCache StatefulTemplateCache) StatefulTemplateCache
type String ¶
type String struct {
// contains filtered or unexported fields
}
func (*String) DefaultLength ¶
func (*String) IsReducedLength ¶
func (*String) MarshalJSON ¶
func (*String) UnmarshalJSON ¶
func (*String) WithLength ¶
func (*String) WithLength(length uint16) DataTypeConstructor
type SubTemplateList ¶
type SubTemplateList struct {
// contains filtered or unexported fields
}
func (*SubTemplateList) Clone ¶
func (t *SubTemplateList) Clone() DataType
func (*SubTemplateList) DefaultLength ¶
func (*SubTemplateList) DefaultLength() uint16
func (*SubTemplateList) Elements ¶
func (t *SubTemplateList) Elements() []DataRecord
func (*SubTemplateList) IsReducedLength ¶
func (t *SubTemplateList) IsReducedLength() bool
func (*SubTemplateList) Length ¶
func (t *SubTemplateList) Length() uint16
func (*SubTemplateList) MarshalJSON ¶
func (t *SubTemplateList) MarshalJSON() ([]byte, error)
func (*SubTemplateList) NewBuilder ¶
func (t *SubTemplateList) NewBuilder() templateListeTypeBuilder
func (*SubTemplateList) Semantic ¶
func (t *SubTemplateList) Semantic() ListSemantic
func (*SubTemplateList) SetLength ¶
func (t *SubTemplateList) SetLength(length uint16) DataType
func (*SubTemplateList) SetSemantic ¶
func (t *SubTemplateList) SetSemantic(semantic ListSemantic) *SubTemplateList
func (*SubTemplateList) SetValue ¶
func (t *SubTemplateList) SetValue(v any) DataType
func (*SubTemplateList) String ¶
func (t *SubTemplateList) String() string
func (*SubTemplateList) TemplateID ¶
func (t *SubTemplateList) TemplateID() uint16
func (*SubTemplateList) Type ¶
func (t *SubTemplateList) Type() string
func (*SubTemplateList) UnmarshalJSON ¶
func (t *SubTemplateList) UnmarshalJSON(in []byte) error
func (*SubTemplateList) Value ¶
func (t *SubTemplateList) Value() interface{}
func (*SubTemplateList) WithLength ¶
func (t *SubTemplateList) WithLength(length uint16) DataTypeConstructor
type SubTemplateMultiList ¶
type SubTemplateMultiList struct {
// contains filtered or unexported fields
}
func (*SubTemplateMultiList) Clone ¶
func (t *SubTemplateMultiList) Clone() DataType
func (*SubTemplateMultiList) Decode ¶
func (t *SubTemplateMultiList) Decode(r io.Reader) (n int, err error)
func (*SubTemplateMultiList) DefaultLength ¶
func (*SubTemplateMultiList) DefaultLength() uint16
func (*SubTemplateMultiList) Elements ¶
func (t *SubTemplateMultiList) Elements() []subTemplateListContent
func (*SubTemplateMultiList) Encode ¶
func (t *SubTemplateMultiList) Encode(w io.Writer) (n int, err error)
func (*SubTemplateMultiList) IsReducedLength ¶
func (*SubTemplateMultiList) IsReducedLength() bool
func (*SubTemplateMultiList) Length ¶
func (t *SubTemplateMultiList) Length() uint16
func (*SubTemplateMultiList) MarshalJSON ¶
func (t *SubTemplateMultiList) MarshalJSON() ([]byte, error)
func (*SubTemplateMultiList) NewBuilder ¶
func (t *SubTemplateMultiList) NewBuilder() templateListeTypeBuilder
func (*SubTemplateMultiList) Semantic ¶
func (t *SubTemplateMultiList) Semantic() ListSemantic
func (*SubTemplateMultiList) SetLength ¶
func (t *SubTemplateMultiList) SetLength(length uint16) DataType
TODO(zoomoid): check if this is safely done with just the length of the elements, or if we need to include "headers" of each subtemplate as well
func (*SubTemplateMultiList) SetSemantic ¶
func (t *SubTemplateMultiList) SetSemantic(semantic ListSemantic) *SubTemplateMultiList
func (*SubTemplateMultiList) SetValue ¶
func (t *SubTemplateMultiList) SetValue(v any) DataType
func (*SubTemplateMultiList) String ¶
func (t *SubTemplateMultiList) String() string
func (*SubTemplateMultiList) Type ¶
func (t *SubTemplateMultiList) Type() string
func (*SubTemplateMultiList) UnmarshalJSON ¶
func (t *SubTemplateMultiList) UnmarshalJSON(in []byte) error
func (*SubTemplateMultiList) Value ¶
func (t *SubTemplateMultiList) Value() interface{}
func (*SubTemplateMultiList) WithLength ¶
func (t *SubTemplateMultiList) WithLength(length uint16) DataTypeConstructor
type TCPListener ¶
type TCPListener struct {
// contains filtered or unexported fields
}
func NewTCPListener ¶ added in v0.3.0
func NewTCPListener(bindAddr string) *TCPListener
func (*TCPListener) Messages ¶
func (l *TCPListener) Messages() <-chan []byte
type Template ¶
type Template struct { *TemplateMetadata `json:"metadata,omitempty"` Record templateRecord // contains filtered or unexported fields }
func (Template) MarshalJSON ¶
func (*Template) UnmarshalJSON ¶
func (*Template) WithFieldCache ¶
func (tr *Template) WithFieldCache(f FieldCache) *Template
func (*Template) WithTemplateCache ¶
func (tr *Template) WithTemplateCache(f TemplateCache) *Template
type TemplateCache ¶
type TemplateCache interface { // GetAll returns the map of all templates currently stored in the cache GetAll(ctx context.Context) map[TemplateKey]*Template // Get returns the template stored at a given key, or an error if not found Get(ctx context.Context, key TemplateKey) (*Template, error) // Add adds a template at a given key into the cache. It may return an error if // anything bad happened during addition Add(ctx context.Context, key TemplateKey, template *Template) error Delete(ctx context.Context, key TemplateKey) error // Name returns the name of the cache set at construction Name() string // Type returns the constant type of the Cache as string Type() string // Caches implement json.Marshaler to be serializable json.Marshaler }
TemplateCache stores templates observed in an IPFIX/Netflow stream of flow packets
Caches have to implement a function to - add a template defined by its version and observation domain ID, - retrieve a template by its version, its observation domain ID, and its ID, and - get all templates currently stored in the cache as a map
Caches do not have to perform active expiry, for this, use TemplateCacheWithTimeout.
func NewDefaultDecayingEphemeralCache ¶
func NewDefaultDecayingEphemeralCache() TemplateCache
func NewNamedDecayingEphemeralCache ¶
func NewNamedDecayingEphemeralCache(name string) TemplateCache
type TemplateCacheDriver ¶
type TemplateCacheDriver interface { StatefulTemplateCache // Prepare is a validator for implementors of Driver that can return an error Prepare() error // Initialize is used for running context-dependent pre-checks such as connecting to KV databases, or opening file handles Initialize(context.Context) error // Close is used for destructing the cache's resources, e.g., closing file handles, disconnecting from databases etc. Close(context.Context) error }
TemplateCacheDriver is the interface to be provided by TemplateCaches that have side effects, such as persistent caches that write to files. Here, the TemplateCacheDriver interface provides functionality to e.g. close file handles or read from files, effectively a hook system that can be used to e.g. restore and dump templates.
type TemplateCacheWithTimeout ¶
type TemplateCacheWithTimeout interface { TemplateCache // SetTimeout should update the internal timeout duration after which templates expire. // Implementing caches MAY update existing template deadlines, but MUST calculate new deadlines // using the latest duration SetTimeout(time.Duration) }
CachesWithTimeout is the interface to be implemented by caches that periodically expire templates which is according to the IPFIX spec (but seemingly never implemented in any of the FOSS collectors)
type TemplateKey ¶
func NewKey ¶
func NewKey(observationDomainId uint32, templateId uint16) TemplateKey
func (*TemplateKey) MarshalText ¶
func (k *TemplateKey) MarshalText() (text []byte, err error)
func (*TemplateKey) String ¶
func (k *TemplateKey) String() string
func (*TemplateKey) Unmarshal ¶
func (k *TemplateKey) Unmarshal(text string) (err error)
func (*TemplateKey) UnmarshalText ¶
func (k *TemplateKey) UnmarshalText(text []byte) (err error)
type TemplateMetadata ¶
type TemplateMetadata struct { Name string `json:"name,omitempty"` TemplateId uint16 `json:"template_id,omitempty"` ObservationDomainId uint32 `json:"observation_domain_id,omitempty"` CreationTimestamp time.Time `json:"created"` Labels map[string]string `json:"labels,omitempty"` Annotations map[string]string `json:"annotations,omitempty"` }
type TemplateRecord ¶
type TemplateRecord struct { TemplateId uint16 `json:"template_id,omitempty"` FieldCount uint16 `json:"field_count,omitempty"` Fields []Field `json:"fields,omitempty"` // contains filtered or unexported fields }
func (*TemplateRecord) Id ¶
func (tr *TemplateRecord) Id() uint16
func (*TemplateRecord) Length ¶
func (tr *TemplateRecord) Length() uint16
func (*TemplateRecord) MarshalJSON ¶
func (tr *TemplateRecord) MarshalJSON() ([]byte, error)
func (*TemplateRecord) String ¶ added in v0.3.0
func (tr *TemplateRecord) String() string
func (*TemplateRecord) Type ¶
func (tr *TemplateRecord) Type() string
func (*TemplateRecord) UnmarshalJSON ¶
func (tr *TemplateRecord) UnmarshalJSON(in []byte) error
type TemplateSet ¶
type TemplateSet struct { Records []TemplateRecord `json:"records,omitempty" yaml:"records,omitempty"` // contains filtered or unexported fields }
func (*TemplateSet) Length ¶
func (d *TemplateSet) Length() int
func (*TemplateSet) String ¶ added in v0.3.0
func (d *TemplateSet) String() string
type UDPListener ¶
type UDPListener struct {
// contains filtered or unexported fields
}
func NewUDPListener ¶
func NewUDPListener(bindAddr string) *UDPListener
func (*UDPListener) Messages ¶
func (l *UDPListener) Messages() <-chan []byte
type Unsigned16 ¶
type Unsigned16 struct {
// contains filtered or unexported fields
}
func (*Unsigned16) Clone ¶
func (t *Unsigned16) Clone() DataType
func (*Unsigned16) DefaultLength ¶
func (t *Unsigned16) DefaultLength() uint16
func (*Unsigned16) IsReducedLength ¶
func (t *Unsigned16) IsReducedLength() bool
func (*Unsigned16) Length ¶
func (t *Unsigned16) Length() uint16
func (*Unsigned16) MarshalJSON ¶
func (t *Unsigned16) MarshalJSON() ([]byte, error)
func (*Unsigned16) SetLength ¶
func (t *Unsigned16) SetLength(length uint16) DataType
func (*Unsigned16) SetValue ¶
func (t *Unsigned16) SetValue(v any) DataType
func (*Unsigned16) String ¶
func (t *Unsigned16) String() string
func (Unsigned16) Type ¶
func (t Unsigned16) Type() string
func (*Unsigned16) UnmarshalJSON ¶
func (t *Unsigned16) UnmarshalJSON(in []byte) error
func (*Unsigned16) Value ¶
func (t *Unsigned16) Value() interface{}
func (*Unsigned16) WithLength ¶
func (t *Unsigned16) WithLength(length uint16) DataTypeConstructor
type Unsigned32 ¶
type Unsigned32 struct {
// contains filtered or unexported fields
}
func (*Unsigned32) Clone ¶
func (t *Unsigned32) Clone() DataType
func (*Unsigned32) DefaultLength ¶
func (*Unsigned32) DefaultLength() uint16
func (*Unsigned32) IsReducedLength ¶
func (t *Unsigned32) IsReducedLength() bool
func (*Unsigned32) Length ¶
func (t *Unsigned32) Length() uint16
func (*Unsigned32) MarshalJSON ¶
func (t *Unsigned32) MarshalJSON() ([]byte, error)
func (*Unsigned32) SetLength ¶
func (t *Unsigned32) SetLength(length uint16) DataType
func (*Unsigned32) SetValue ¶
func (t *Unsigned32) SetValue(v any) DataType
func (*Unsigned32) String ¶
func (t *Unsigned32) String() string
func (*Unsigned32) Type ¶
func (*Unsigned32) Type() string
func (*Unsigned32) UnmarshalJSON ¶
func (t *Unsigned32) UnmarshalJSON(in []byte) error
func (*Unsigned32) Value ¶
func (t *Unsigned32) Value() interface{}
func (*Unsigned32) WithLength ¶
func (t *Unsigned32) WithLength(length uint16) DataTypeConstructor
type Unsigned64 ¶
type Unsigned64 struct {
// contains filtered or unexported fields
}
func (*Unsigned64) Clone ¶
func (t *Unsigned64) Clone() DataType
func (*Unsigned64) DefaultLength ¶
func (*Unsigned64) DefaultLength() uint16
func (*Unsigned64) IsReducedLength ¶
func (t *Unsigned64) IsReducedLength() bool
func (*Unsigned64) Length ¶
func (t *Unsigned64) Length() uint16
func (*Unsigned64) MarshalJSON ¶
func (t *Unsigned64) MarshalJSON() ([]byte, error)
func (*Unsigned64) SetLength ¶
func (t *Unsigned64) SetLength(length uint16) DataType
func (*Unsigned64) SetValue ¶
func (t *Unsigned64) SetValue(v any) DataType
func (*Unsigned64) String ¶
func (t *Unsigned64) String() string
func (*Unsigned64) Type ¶
func (*Unsigned64) Type() string
func (*Unsigned64) UnmarshalJSON ¶
func (t *Unsigned64) UnmarshalJSON(in []byte) error
func (*Unsigned64) Value ¶
func (t *Unsigned64) Value() interface{}
func (*Unsigned64) WithLength ¶
func (t *Unsigned64) WithLength(length uint16) DataTypeConstructor
type Unsigned8 ¶
type Unsigned8 struct {
// contains filtered or unexported fields
}
func (*Unsigned8) DefaultLength ¶
func (*Unsigned8) IsReducedLength ¶
func (*Unsigned8) MarshalJSON ¶
func (*Unsigned8) UnmarshalJSON ¶
func (*Unsigned8) WithLength ¶
func (*Unsigned8) WithLength(length uint16) DataTypeConstructor
type VariableLengthField ¶
type VariableLengthField struct {
// contains filtered or unexported fields
}
func (*VariableLengthField) Clone ¶
func (f *VariableLengthField) Clone() Field
func (*VariableLengthField) Constructor ¶
func (f *VariableLengthField) Constructor() DataTypeConstructor
func (*VariableLengthField) Id ¶
func (f *VariableLengthField) Id() uint16
func (*VariableLengthField) IsScope ¶
func (f *VariableLengthField) IsScope() bool
func (*VariableLengthField) Length ¶
func (f *VariableLengthField) Length() uint16
func (*VariableLengthField) Lift ¶
func (f *VariableLengthField) Lift() *VariableLengthField
Variable-length fields are already encoded as such, just return the field
func (*VariableLengthField) MarshalJSON ¶
func (f *VariableLengthField) MarshalJSON() ([]byte, error)
func (*VariableLengthField) Name ¶
func (f *VariableLengthField) Name() string
func (*VariableLengthField) ObservationDomainId ¶
func (f *VariableLengthField) ObservationDomainId() uint32
func (*VariableLengthField) PEN ¶
func (f *VariableLengthField) PEN() uint32
func (*VariableLengthField) Prototype ¶
func (f *VariableLengthField) Prototype() *InformationElement
func (*VariableLengthField) Reversed ¶
func (f *VariableLengthField) Reversed() bool
func (*VariableLengthField) Reversible ¶
func (f *VariableLengthField) Reversible() bool
func (*VariableLengthField) SetScoped ¶
func (f *VariableLengthField) SetScoped() Field
func (*VariableLengthField) SetValue ¶
func (f *VariableLengthField) SetValue(v any) Field
func (*VariableLengthField) String ¶ added in v0.3.0
func (f *VariableLengthField) String() string
func (*VariableLengthField) Type ¶
func (f *VariableLengthField) Type() string
func (*VariableLengthField) UnmarshalJSON ¶
func (f *VariableLengthField) UnmarshalJSON(in []byte) error
func (*VariableLengthField) Value ¶
func (f *VariableLengthField) Value() DataType
Source Files ¶
- basic_list.go
- boolean.go
- constants.go
- csv.go
- data_record.go
- data_types.go
- date_time_microseconds.go
- date_time_milliseconds.go
- date_time_nanoseconds.go
- date_time_seconds.go
- decaying_cache.go
- decode.go
- doc.go
- ephemeral_cache.go
- errors.go
- field.go
- field_builder.go
- field_cache.go
- fixed.go
- float32.go
- float64.go
- header.go
- information_element.go
- ipfix_file_format.go
- ipv4_address.go
- ipv6_address.go
- list.go
- list_semantics.go
- logger.go
- mac_address.go
- message.go
- metrics.go
- octet_array.go
- options_template_record.go
- persistent.go
- rfc5103.go
- rfc5610.go
- sets.go
- signed16.go
- signed32.go
- signed64.go
- signed8.go
- string.go
- sub_template_list.go
- sub_template_multi_list.go
- tcp.go
- template.go
- template_cache.go
- template_record.go
- udp.go
- unassigned.go
- unsigned16.go
- unsigned32.go
- unsigned64.go
- unsigned8.go
- util.go
- variable.go
- xml.go
- yaml.go