goDB

package
v0.0.0-...-c20b0bb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2018 License: GPL-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source
const (
	EPOCH_DAY         int64 = 86400 // one day in seconds
	DB_WRITE_INTERVAL int64 = 300   // write out interval of capture probe
)
View Source
const (
	BUF_SIZE = 4096         // 512 * 64bit
	N_ELEM   = BUF_SIZE / 8 // 512
)
View Source
const (
	// First the attribute columns...
	SIP_COLIDX, _ columnIndex = iota, iota
	DIP_COLIDX, _
	PROTO_COLIDX, _
	DPORT_COLIDX, _
	// ... and then the columns we aggregate
	BYTESRCVD_COLIDX, COLIDX_ATTRIBUTE_COUNT
	BYTESSENT_COLIDX, _
	PKTSRCVD_COLIDX, _
	PKTSSENT_COLIDX, _
	COLIDX_COUNT, _
)

Indizes for all column types

View Source
const (
	SIP_SIZEOF       int = 16
	DIP_SIZEOF       int = 16
	PROTO_SIZEOF     int = 1
	DPORT_SIZEOF     int = 2
	BYTESRCVD_SIZEOF int = 8
	BYTESSENT_SIZEOF int = 8
	PKTSRCVD_SIZEOF  int = 8
	PKTSSENT_SIZEOF  int = 8
)

Sizeof (entry) for all column types

View Source
const (
	SUMMARY_FILE_NAME      = "summary.json"
	SUMMARY_LOCK_FILE_NAME = "summary.lock"
)
View Source
const (
	// Used for compression applied by GPFile.
	COMPRESSION_LEVEL = 512

	METADATA_FILE_NAME = "meta.json"
)
View Source
const SLOG_ADDR = "127.0.0.1"
View Source
const SLOG_PORT = "514"
View Source
const (
	SOCKET_PATH = "/var/run/goprobe.sock"
)

Variables

View Source
var SysLog *syslog.Writer
View Source
var TimeFormats []string = []string{
	time.ANSIC,
	time.RubyDate,
	time.RFC822Z,
	time.RFC1123Z,
	time.RFC3339,

	"2006-01-02 15:04:05 -0700",
	"2006-01-02 15:04:05",
	"2006-01-02 15:04 -0700",
	"2006-01-02 15:04",
	"02.01.2006 15:04",
	"02.01.2006 15:04 -0700",
	"02.01.06 15:04",
	"02.01.06 15:04 -0700",
	"2.1.06 15:04:05",
	"2.1.06 15:04:05 -0700",
	"2.1.06 15:04",
	"2.1.06 15:04 -0700",
	"2.1.2006 15:04:05",
	"2.1.2006 15:04:05 -0700",
	"2.1.2006 15:04",
	"2.1.2006 15:04 -0700",
	"02.1.2006 15:04:05",
	"02.1.2006 15:04:05 -0700",
	"02.1.2006 15:04",
	"02.1.2006 15:04 -0700",
	"2.01.2006 15:04:05",
	"2.01.2006 15:04:05 -0700",
	"2.01.2006 15:04",
	"2.01.2006 15:04 -0700",
	"02.1.06 15:04:05",
	"02.1.06 15:04:05 -0700",
	"02.1.06 15:04",
	"02.1.06 15:04 -0700",
	"2.01.06 15:04:05",
	"2.01.06 15:04:05 -0700",
	"2.01.06 15:04",
	"2.01.06 15:04 -0700"}

Utility variables and functions for time parsing -----------------------------

Functions

func DayTimestamp

func DayTimestamp(timestamp int64) int64

DayTimestamp returns timestamp rounded down to the nearest day

func HasDNSAttributes

func HasDNSAttributes(attributes []Attribute) bool

Find out if any of the attributes are usable for a reverse DNS lookup (e.g. check for IP attributes)

func IPStringToBytes

func IPStringToBytes(ip string) ([]byte, error)

Condition conversion utility functions ------------------------------------------------

func InitDBLog

func InitDBLog() error

func LockDBSummary

func LockDBSummary(dbpath string) (acquired bool, err error)

LockDBSummary tries to acquire a lockfile for the database summary. Its return values indicate whether it successfully acquired the lock and whether a file system error occurred.

func ModifyDBSummary

func ModifyDBSummary(dbpath string, timeout time.Duration, modify func(*DBSummary) (*DBSummary, error)) (modErr error)

Safely modifies the database summary when there are multiple processes accessing it.

If no lock can be acquired after (roughly) timeout time, returns an error.

modify is expected to obey the following contract:

  • The input summary is nil if no summary file is present.
  • modify returns the summary to be written (must be non-nil) and an error.
  • Since the summary is locked while modify is running, modify shouldn't take longer than roughly half a second.

func ParseTimeArgument

func ParseTimeArgument(timeString string) (int64, error)

Entry point for external calls -------------------------------------------------

func SanitizeUserInput

func SanitizeUserInput(conditional string) (string, error)

SanitizeUserInput sanitizes a conditional string provided by the user. Its main purpose is to convert other forms of precedence and logical operators to the condition grammar used. For example, some people may prefer a more verbose forms such as "dport=443 or dport=8080" or exotic forms such as "{dport=443 || dport=8080}". These should be caught and converted to the grammar-conforming expression "(dport=443|dport=8080)"

Input:

conditional: string containing the conditional specified in "user grammar"

Output:

string:  conditional string in the condition grammar. Note that this may still
         include syntactical errors or malspecified conditions. These will be caught
         at a latter stage
error:   any error from golang's regex module

NOTE: the current implementation of GPDPIProtocols.go has to make sure that the map keys

of "proto" to numbers are all lower case

func TokenizeConditional

func TokenizeConditional(condExpression string) ([]string, error)

TokenizeConditional tokenizes the given conditional. Note that the tokenization is "loose": All valid conditionals will be correctly tokenized, but there are invalid conditionals that will also be tokenized. Its the parser's job to catch those. Whitespace in conditionals is only useful for tokenization and not needed afterwards. TokenizeConditional doesn't emit any whitespace tokens.

Limitations: Only ASCII is supported. May give strange results on fancy Unicode strings.

func UnlockDBSummary

func UnlockDBSummary(dbpath string) (err error)

LockDBSummary removes the lockfile for the database summary. Its return values indicates whether a file system error occurred.

func WriteDBSummary

func WriteDBSummary(dbpath string, summ *DBSummary) error

Writes a new summary for the given database. If multiple processes might be operating on the summary simultaneously, you should lock it first.

func WriteMetadata

func WriteMetadata(path string, meta *Metadata) error

Types

type AggFlowMap

type AggFlowMap map[Key]*Val

type Attribute

type Attribute interface {
	Name() string
	// Some attributes use more than a single output column.
	ExtraColumns() []string
	// ExtractStrings() extracts a list of records representing the
	// attribute from a given key.
	// You may assume that the length of the returned list is always
	// the length of ExtraColumns() + 1.
	ExtractStrings(key *ExtraKey) []string
	// contains filtered or unexported methods
}

Interface for attributes This interface is not meant to be implemented by structs outside this package

func NewAttribute

func NewAttribute(name string) (Attribute, error)

Returns an Attribute for the given name. If no such attribute exists, an error is returned.

func ParseQueryType

func ParseQueryType(queryType string) (attributes []Attribute, hasAttrTime, hasAttrIface bool, err error)

Parses the given query type into a list of attributes. The returned list is guaranteed to have no duplicates. A valid query type can either be a comma-separated list of attribute names (e.g. "sip,dip,dport") or something like "talk_conv". The return variable hasAttrTime indicates whether the special time attribute is present. (time is never a part of the returned attribute list.) The time attribute is present for the query type 'raw', or if it is explicitly mentioned in a list of attribute names.

type BlockMetadata

type BlockMetadata struct {
	Timestamp            int64 `json:"timestamp"`
	PcapPacketsReceived  int   `json:"pcap_packets_received"`
	PcapPacketsDropped   int   `json:"pcap_packets_dropped"`
	PcapPacketsIfDropped int   `json:"pcap_packets_if_dropped"`
	PacketsLogged        int   `json:"packets_logged"`

	// As in Summary
	FlowCount uint64 `json:"flowcount"`
	Traffic   uint64 `json:"traffic"`
}

Represents metadata for one database block.

type BytesRecStringParser

type BytesRecStringParser struct{}

value parsers

func (*BytesRecStringParser) ParseVal

func (b *BytesRecStringParser) ParseVal(element string, val *Val) error

type BytesSentStringParser

type BytesSentStringParser struct{}

func (*BytesSentStringParser) ParseVal

func (b *BytesSentStringParser) ParseVal(element string, val *Val) error

type DBData

type DBData struct {
	// counters
	Bytes_rcvd []byte
	Bytes_sent []byte
	Pkts_rcvd  []byte
	Pkts_sent  []byte

	// attributes
	Dip   []byte
	Sip   []byte
	Dport []byte
	Proto []byte

	// metadata (important for folder naming)
	Tstamp int64
	Iface  string
}

func NewDBData

func NewDBData(br []byte, bs []byte, pr []byte, ps []byte, dip []byte, sip []byte, dport []byte, proto []byte, tstamp int64, iface string) DBData

constructor for the DBData struct in case it needs to be set from an external go program that included goProbe

type DBLog

type DBLog struct {
	Log *syslog.Writer
}

type DBSummary

type DBSummary struct {
	Interfaces map[string]InterfaceSummary `json:"interfaces"`
}

Summary for an entire database

func NewDBSummary

func NewDBSummary() *DBSummary

func ReadDBSummary

func ReadDBSummary(dbpath string) (*DBSummary, error)

Reads the summary of the given database. If multiple processes might be operating on the summary simultaneously, you should lock it first.

func (*DBSummary) Update

func (s *DBSummary) Update(u InterfaceSummaryUpdate)

type DBWorkManager

type DBWorkManager struct {
	// contains filtered or unexported fields
}

func NewDBWorkManager

func NewDBWorkManager(dbpath string, iface string, numProcessingUnits int) (*DBWorkManager, error)

func (*DBWorkManager) CreateWorkerJobs

func (w *DBWorkManager) CreateWorkerJobs(tfirst int64, tlast int64, query *Query) (nonempty bool, err error)

func (*DBWorkManager) ExecuteWorkerReadJobs

func (w *DBWorkManager) ExecuteWorkerReadJobs(mapChan chan map[ExtraKey]Val)

Spawning of processing units and pushing of workload onto factory channel -----------

func (*DBWorkManager) GetCoveredTimeInterval

func (w *DBWorkManager) GetCoveredTimeInterval() (time.Time, time.Time)

used to determine the time span actually covered by the query

func (*DBWorkManager) GetNumWorkers

func (w *DBWorkManager) GetNumWorkers() int

make number of workloads available to the outside world for loop bounds etc.

type DBWorkload

type DBWorkload struct {
	// contains filtered or unexported fields
}

type DBWriter

type DBWriter struct {
	// contains filtered or unexported fields
}

func NewDBWriter

func NewDBWriter(dbpath string, iface string) (w *DBWriter)

func (*DBWriter) Write

func (w *DBWriter) Write(flowmap AggFlowMap, meta BlockMetadata, timestamp int64) (InterfaceSummaryUpdate, error)

type DipAttribute

type DipAttribute struct{}

func (DipAttribute) ExtraColumns

func (_ DipAttribute) ExtraColumns() []string

func (DipAttribute) ExtractStrings

func (_ DipAttribute) ExtractStrings(key *ExtraKey) []string

func (DipAttribute) Name

func (_ DipAttribute) Name() string

type DipStringParser

type DipStringParser struct{}

func (*DipStringParser) ParseKey

func (d *DipStringParser) ParseKey(element string, key *ExtraKey) error

type DportAttribute

type DportAttribute struct{}

func (DportAttribute) ExtraColumns

func (_ DportAttribute) ExtraColumns() []string

func (DportAttribute) ExtractStrings

func (_ DportAttribute) ExtractStrings(key *ExtraKey) []string

func (DportAttribute) Name

func (_ DportAttribute) Name() string

type DportStringParser

type DportStringParser struct{}

func (*DportStringParser) ParseKey

func (d *DportStringParser) ParseKey(element string, key *ExtraKey) error

type ExtraKey

type ExtraKey struct {
	Time  int64
	Iface string
	Key
}

ExtraKey is a key with extra information

type GPFile

type GPFile struct {
	// contains filtered or unexported fields
}

func NewGPFile

func NewGPFile(p string) (*GPFile, error)

func (*GPFile) BlocksUsed

func (f *GPFile) BlocksUsed() (int, error)

func (*GPFile) Close

func (f *GPFile) Close() error

func (*GPFile) GetBlocks

func (f *GPFile) GetBlocks() []int64

func (*GPFile) GetTimestamps

func (f *GPFile) GetTimestamps() []int64

func (*GPFile) ReadBlock

func (f *GPFile) ReadBlock(block int) ([]byte, error)

func (*GPFile) ReadTimedBlock

func (f *GPFile) ReadTimedBlock(timestamp int64) ([]byte, error)

func (*GPFile) WriteTimedBlock

func (f *GPFile) WriteTimedBlock(timestamp int64, data []byte, comp int) error

type GPFiler

type GPFiler interface {
	BlocksUsed() (int, error)
	WriteTimedBlock(timestamp int64, data []byte) error
	ReadTimedBlock(timestamp int64) ([]byte, error)
	ReadBlock(block int) ([]byte, error)
	GetBlocks() ([]int64, error)
	GetTimestamps() ([]int64, error)
	Close() error
}

type IfaceStringParser

type IfaceStringParser struct{}

func (*IfaceStringParser) ParseKey

func (i *IfaceStringParser) ParseKey(element string, key *ExtraKey) error

type InterfaceSummary

type InterfaceSummary struct {
	// Number of flows
	FlowCount uint64 `json:"flowcount"`
	// Total traffic volume in byte
	Traffic uint64 `json:"traffic"`
	Begin   int64  `json:"begin"`
	End     int64  `json:"end"`
}

Summary for a single interface

type InterfaceSummaryUpdate

type InterfaceSummaryUpdate struct {
	// Name of the interface. For example, "eth0".
	Interface string
	// Number of flows
	FlowCount uint64
	// Traffic volume in bytes
	Traffic   uint64
	Timestamp time.Time
}

type Key

type Key struct {
	Sip      [16]byte
	Dip      [16]byte
	Dport    [2]byte
	Protocol byte
}

func (Key) String

func (k Key) String() string

print the key as a comma separated attribute list

type Metadata

type Metadata struct {
	Blocks []BlockMetadata `json:"blocks"`
}

Metadata for a collection of database blocks. By convention all blocks belong the same day.

func NewMetadata

func NewMetadata() *Metadata

func ReadMetadata

func ReadMetadata(path string) (*Metadata, error)

Reads the given metadata file.

func TryReadMetadata

func TryReadMetadata(path string) *Metadata

Tries to read the given metadata file. If an error occurs, a fresh Metadata struct is returned.

type NOPStringParser

type NOPStringParser struct{}

func (*NOPStringParser) ParseKey

func (n *NOPStringParser) ParseKey(element string, key *ExtraKey) error

The NOP parser doesn't do anything and just lets everything through which is not understandable by the other attribute parsers (e.g. the % field or any other field not mentioned above)

func (*NOPStringParser) ParseVal

func (n *NOPStringParser) ParseVal(element string, val *Val) error

type Node

type Node interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

An AST node for the conditional grammar This interface is not meant to be implemented by structs outside of this package.

func ParseAndInstrumentConditional

func ParseAndInstrumentConditional(conditional string, dnsTimeout time.Duration) (Node, error)

Parses and instruments the given conditional string for evaluation. This is the main external function related to conditionals.

type PacketsRecStringParser

type PacketsRecStringParser struct{}

func (*PacketsRecStringParser) ParseVal

func (p *PacketsRecStringParser) ParseVal(element string, val *Val) error

type PacketsSentStringParser

type PacketsSentStringParser struct{}

func (*PacketsSentStringParser) ParseVal

func (p *PacketsSentStringParser) ParseVal(element string, val *Val) error

type ProtoAttribute

type ProtoAttribute struct{}

func (ProtoAttribute) ExtraColumns

func (_ ProtoAttribute) ExtraColumns() []string

func (ProtoAttribute) ExtractStrings

func (_ ProtoAttribute) ExtractStrings(key *ExtraKey) []string

func (ProtoAttribute) Name

func (_ ProtoAttribute) Name() string

type ProtoStringParser

type ProtoStringParser struct{}

func (*ProtoStringParser) ParseKey

func (p *ProtoStringParser) ParseKey(element string, key *ExtraKey) error

type Query

type Query struct {
	// list of attributes that will be compared, e.g. "dip" "sip"
	// in a "talk_conv" query
	Attributes  []Attribute
	Conditional Node
	// contains filtered or unexported fields
}

func NewQuery

func NewQuery(attributes []Attribute, conditional Node, hasAttrTime, hasAttrIface bool) *Query

type SipAttribute

type SipAttribute struct{}

func (SipAttribute) ExtraColumns

func (_ SipAttribute) ExtraColumns() []string

func (SipAttribute) ExtractStrings

func (_ SipAttribute) ExtractStrings(key *ExtraKey) []string

func (SipAttribute) Name

func (_ SipAttribute) Name() string

type SipStringParser

type SipStringParser struct{}

attribute parsers

func (*SipStringParser) ParseKey

func (s *SipStringParser) ParseKey(element string, key *ExtraKey) error

type StringKeyParser

type StringKeyParser interface {
	ParseKey(element string, key *ExtraKey) error
}

func NewStringKeyParser

func NewStringKeyParser(kind string) StringKeyParser

type StringValParser

type StringValParser interface {
	ParseVal(element string, val *Val) error
}

func NewStringValParser

func NewStringValParser(kind string) StringValParser

type SyslogDBWriter

type SyslogDBWriter struct {
	// contains filtered or unexported fields
}

func NewSyslogDBWriter

func NewSyslogDBWriter() (*SyslogDBWriter, error)

func (*SyslogDBWriter) Write

func (s *SyslogDBWriter) Write(flowmap AggFlowMap, iface string, timestamp int64)

type TimeStringParser

type TimeStringParser struct{}

extra attributes

func (*TimeStringParser) ParseKey

func (t *TimeStringParser) ParseKey(element string, key *ExtraKey) error

type Val

type Val struct {
	NBytesRcvd uint64
	NBytesSent uint64
	NPktsRcvd  uint64
	NPktsSent  uint64
}

func (*Val) String

func (v *Val) String() string

Directories

Path Synopsis
Package bigendian provides functionality to convert (u)ints encoded in big-endian to little-endian.
Package bigendian provides functionality to convert (u)ints encoded in big-endian to little-endian.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL