cabby

package module
v0.0.0-...-cd9fc01 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 25, 2023 License: MIT Imports: 13 Imported by: 0

README

Go Report Card Code Coverage Build Status Go Doc Go City Release

cabby

TAXII 2.1 server in Golang.

Dependencies

  • Golang 1.10 or 1.11
  • SQLite

Setup

make

Testing

To run all tests: make test

"Helper" functions are in test_helper_test.go. The goal with this file was to put repetitive code that make the tests verbose into a DRY'er format.

GoSec

Security checking tool: https://github.com/securego/gosec make sec to run security tests make sec pkg=<package> to run against a specific package. Example: make sec pkg=sqlite to run it against sqlite package

Building

Cabby uses sqlite3 as its data store. The library in golang being used requires C extensions/bindings and as such doesn't build on a Mac for a Ubuntu OS. Therefore vagrant vm is used to build a debian for ubuntu (i know...gross).

Building debian package for a vagrant VM running ubuntu: make build-debian

Building: Troubleshooting on the VM

Reference: https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs

# tail the log for cabby
sudo journalctl -u cabby -f

# pipe it to less
sudo journalctl -u cabby | less
Building: Metrics/Logs on the VM (TICK Stack)

I used this project to also explore the TICK stack: https://www.influxdata.com/time-series-platform/

To use/play with it, run make vagrant

Syslog input reference (the VM uses syslog to as an input to influx-db):

Building: References

Versioning

Release branches are associated to a TAXII spec version and a SemVer version down to the minor revision.

Patches are annotated by creating a tag off of the release branch.

Examples:

  1. First major release of a TAXII 2.0 server:
  2. branch: release/2.0/1.0
  3. tag: release/2.0/1.0.0
  4. A version of TAXII 2.0 spec server that has backward incompatible changes BUT is still a TAXII 2.0 server
  5. branch: release/2.0/2.0
  6. tag: release/2.0/2.0.0
  7. Initial branch for a 2.1 spec server:
  8. branch: release/2.1/0.0
  9. tag: release/2.1/0.0.1 (once a chunk of code worth tagging is ready)

Configuration

The make task will generate certs and a default config file. Edit the config/cabby.json file to adjust things like

  • port
  • data store file path
  • cert paths

DB Setup

Using Sqlite as a light-weight data store to run this in development mode. Goal is to move to some kind of JSON store (rethinkdb or elasticsearch) in the future. See below API examples for setup instructions.

API Examples with a test user

The examples below require

  • jq
  • sqlite

On a mac you can install via brew:

brew install sqlite
brew install jq

Set up the DB for dev/test: make dev-db The user set up with make dev-db is an admin (so it can do admin things like create/update/delete certain resources).

In another terminal, run a server: make run

View TAXII Discovery
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/taxii2/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/taxii2/' | jq .
View API Root
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/' | jq .
View Collections
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/' | jq .
# view 1 of N
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/?limit=1' && echo
View Collection
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/' | jq .
Add Objects

In the above example, new collections were added. Kill the server (CTRL+C) and make run again. The logs will show new routes are added.

Now post a envelope of STIX 2.0 data:

curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/malware_envelope.json | jq .
Check status

From the above POST, you get a status object. You can query it from the server

export STATUSID=<your status id>
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' "https://localhost:1234/cabby_test_root/status/$STATUSID/" | jq .
unset STATUSID
View Manifest
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/manifest/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/manifest/' | jq .
View Objects
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' | jq .

# view 1 of N
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?limit=1' && echo
# view 1 0f N parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?limit=1' | jq .
View Object Versions

Post an envelope with same objects but new versions to illustrate:

curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/versions_envelope.json | jq .
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f/versions/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f/versions/' | jq .
View Object
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.stix+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.stix+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad/' | jq .
Filter objects
# filter on types
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[type\]=indicator,malware' | jq .
# filter on id
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[id\]=indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f' | jq .

# add objects to filter on versions
# the below envelope has objects that already exist; status will have 3 failures
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/versions_envelope.json | jq .

# filter on latest versions (indicator will be 2018 provided data above has been added)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=last' | jq .
# filter on oldest versions (indicator will be 2016)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=first' | jq .
# filter on specific versions (indicator will be 2017)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=2017-01-01T12:15:12.123Z' | jq .
Delete objects

NOTE: These actions are destructive, use make dev-db to reset the dev db after deleting.

# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -X DELETE 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad' && echo
# OR parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -X DELETE 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad' | jq .

# view objects to verify
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' | jq .

Resources

Documentation

Index

Constants

View Source
const (

	// DefaultDevelopmentConfig is the path to the local dev config
	DefaultDevelopmentConfig = "config/cabby.json"
	// DefaultProductionConfig is the path to the packaged config file
	DefaultProductionConfig = "/etc/cabby/cabby.json"

	// StixContentType20 represents a stix 2.0 content type
	StixContentType20 = "application/vnd.oasis.stix+json;version=2.0"
	// StixContentType represents a stix 2 content type
	StixContentType = "application/vnd.oasis.stix+json"
	// TaxiiContentType21 represents a taxii 2.1 content type
	TaxiiContentType21 = "application/vnd.oasis.taxii+json;version=2.1"
	// TaxiiContentType represents a taxii 2 content type
	TaxiiContentType = "application/vnd.oasis.taxii+json"
	// TaxiiVersion notes the supported version of the server
	TaxiiVersion = "taxii-2.1"

	// UnsetUnixNano is the value returned from an unset time.Time{}.UnixNano() call
	UnsetUnixNano = -6795364578871345152
)

Variables

This section is empty.

Functions

func LogServiceEnd

func LogServiceEnd(ctx context.Context, resource, action string, start time.Time)

LogServiceEnd takes a resource and action being performed and a start time, and logs it and how long it took

func LogServiceStart

func LogServiceStart(ctx context.Context, resource, action string) time.Time

LogServiceStart takes a resource and action being performed and logs it

func TakeBytes

func TakeBytes(ctx context.Context) int

TakeBytes returns the bytes stored in a context

func TakeTransactionID

func TakeTransactionID(ctx context.Context) uuid.UUID

TakeTransactionID returns the transaction id stored in a context

func WithBytes

func WithBytes(ctx context.Context, bytes int) context.Context

WithBytes decorates a context with bytes written

func WithTransactionID

func WithTransactionID(ctx context.Context, transactionID uuid.UUID) context.Context

WithTransactionID decorates a context with a transaction id

func WithUser

func WithUser(ctx context.Context, u User) context.Context

WithUser decorates a context with values from the user struct

Types

type APIRoot

type APIRoot struct {
	Path             string   `json:"path,omitempty"`
	Title            string   `json:"title"`
	Description      string   `json:"description,omitempty"`
	Versions         []string `json:"versions"`
	MaxContentLength int64    `json:"max_content_length"`
}

APIRoot resource

func (*APIRoot) IncludesMinVersion

func (a *APIRoot) IncludesMinVersion(vs []string) bool

IncludesMinVersion checks if minimum taxii version is included in list

func (*APIRoot) Validate

func (a *APIRoot) Validate() error

Validate an API Root

type APIRootService

type APIRootService interface {
	APIRoot(ctx context.Context, path string) (APIRoot, error)
	APIRoots(ctx context.Context) ([]APIRoot, error)
	CreateAPIRoot(ctx context.Context, a APIRoot) error
	DeleteAPIRoot(ctx context.Context, path string) error
	UpdateAPIRoot(ctx context.Context, a APIRoot) error
}

APIRootService for interacting with APIRoots

type Collection

type Collection struct {
	APIRootPath string   `json:"api_root_path,omitempty"`
	ID          ID       `json:"id"`
	CanRead     bool     `json:"can_read"`
	CanWrite    bool     `json:"can_write"`
	Title       string   `json:"title"`
	Description string   `json:"description,omitempty"`
	MediaTypes  []string `json:"media_types,omitempty"`
}

Collection resource

func NewCollection

func NewCollection(id ...string) (Collection, error)

NewCollection returns a collection resource; it takes an optional id string

func (*Collection) Validate

func (c *Collection) Validate() (err error)

Validate a collection

type CollectionAccess

type CollectionAccess struct {
	ID       ID   `json:"id"`
	CanRead  bool `json:"can_read"`
	CanWrite bool `json:"can_write"`
}

CollectionAccess defines read/write access on a collection

type CollectionService

type CollectionService interface {
	Collection(ctx context.Context, apiRoot, collectionID string) (Collection, error)
	Collections(ctx context.Context, apiRoot string, cr *Page) (Collections, error)
	CollectionsInAPIRoot(ctx context.Context, apiRoot string) (CollectionsInAPIRoot, error)
	CreateCollection(ctx context.Context, c Collection) error
	DeleteCollection(ctx context.Context, collectionID string) error
	UpdateCollection(ctx context.Context, c Collection) error
}

CollectionService interface for interacting with data store

type Collections

type Collections struct {
	Collections []Collection `json:"collections"`
}

Collections resource

type CollectionsInAPIRoot

type CollectionsInAPIRoot struct {
	Path          string
	CollectionIDs []ID
}

CollectionsInAPIRoot associated a list of collection IDs that belong to a API Root Path

type Config

type Config struct {
	Host      string
	Port      int
	SSLCert   string            `json:"ssl_cert"`
	SSLKey    string            `json:"ssl_key"`
	DataStore map[string]string `json:"data_store"`
}

Config for a server

func (Config) Parse

func (c Config) Parse(file string) (initializedConfig Config)

Parse takes a path to a config file and converts to Configs

#nosec G304

type DataStore

type DataStore interface {
	APIRootService() APIRootService
	Close()
	CollectionService() CollectionService
	DiscoveryService() DiscoveryService
	ManifestService() ManifestService
	MigrationService() MigrationService
	ObjectService() ObjectService
	Open() error
	StatusService() StatusService
	UserService() UserService
	VersionsService() VersionsService
}

DataStore interface for backend implementations

type Discovery

type Discovery struct {
	Title       string   `json:"title"`
	Description string   `json:"description,omitempty"`
	Contact     string   `json:"contact,omitempty"`
	Default     string   `json:"default,omitempty"`
	APIRoots    []string `json:"api_roots,omitempty"`
}

Discovery resource

func (*Discovery) Validate

func (d *Discovery) Validate() error

Validate a discovery resource

type DiscoveryService

type DiscoveryService interface {
	CreateDiscovery(ctx context.Context, d Discovery) error
	DeleteDiscovery(ctx context.Context) error
	Discovery(ctx context.Context) (Discovery, error)
	UpdateDiscovery(ctx context.Context, d Discovery) error
}

DiscoveryService interface for interacting with Discovery resources

type Envelope

type Envelope struct {
	More    bool              `json:"more"`
	Objects []json.RawMessage `json:"objects"`
}

Envelope resource for transmitting stix objects

type Error

type Error struct {
	Title           string            `json:"title"`
	Description     string            `json:"description,omitempty"`
	ErrorID         string            `json:"error_id,omitempty"`
	ErrorCode       string            `json:"error_code,omitempty"`
	HTTPStatus      int               `json:"http_status,string,omitempty"`
	ExternalDetails string            `json:"external_details,omitempty"`
	Details         map[string]string `json:"details,omitempty"`
}

Error struct for TAXII 2 errors

type Filter

type Filter struct {
	AddedAfter stones.Timestamp
	IDs        string
	Types      string
	Versions   string
}

Filter for filtering results based on URL parameters

type ID

type ID struct {
	uuid.UUID
}

ID for taxii resources

func IDFromString

func IDFromString(s string) (ID, error)

IDFromString takes a uuid string and coerces to ID

func IDUsingString

func IDUsingString(s string) (ID, error)

IDUsingString creates a V5 UUID from the given string

func NewID

func NewID() (ID, error)

NewID returns a new ID which is a UUID v4

func (*ID) IsEmpty

func (id *ID) IsEmpty() bool

IsEmpty returns a boolean based on whether the UUID is not defined

IE: string representation 00000000-0000-0000-0000-000000000000 is undefined

type Key

type Key int

Key type for context ids; per context docuentation, use a Key type for context Keys

const (
	// KeyBytes stores the bytes written for a response
	KeyBytes Key = iota

	// KeyTransactionID to track a request from http request to response (including service calls)
	KeyTransactionID

	// KeyUser for storing a User struct
	KeyUser
)

type Manifest

type Manifest struct {
	Objects []ManifestEntry `json:"objects,omitempty"`
}

Manifest resource lists a summary of objects in a collection

type ManifestEntry

type ManifestEntry struct {
	ID         string           `json:"id"`
	DateAdded  stones.Timestamp `json:"date_added"`
	Version    stones.Timestamp `json:"version"`
	MediaTypes []string         `json:"media_types"`
}

ManifestEntry is a summary of an object in a manifest

type ManifestService

type ManifestService interface {
	Manifest(ctx context.Context, collectionID string, cr *Page, f Filter) (Manifest, error)
}

ManifestService provides manifest data

type MigrationService

type MigrationService interface {
	CurrentVersion() (int, error)
	Up() error
}

MigrationService for performing database migrations

type ObjectService

type ObjectService interface {
	CreateEnvelope(ctx context.Context, e Envelope, collectionID string, s Status, ss StatusService)
	CreateObject(ctx context.Context, collectionID string, o stones.Object) error
	DeleteObject(ctx context.Context, collectionID, objecteID string) error
	Object(ctx context.Context, collectionID, objectID string, f Filter) ([]stones.Object, error)
	Objects(ctx context.Context, collectionID string, cr *Page, f Filter) ([]stones.Object, error)
}

ObjectService provides Object data

type Page

type Page struct {
	Limit uint64
	// Used for setting X-TAXII-Date-Added-First
	MinimumAddedAfter stones.Timestamp
	// Used for setting X-TAXII-Date-Added-Last
	MaximumAddedAfter stones.Timestamp
	Total             uint64
}

Page is used for paginated requests to represent the requested data range

func NewPage

func NewPage(limit string) (p Page, err error)

NewPage returns a Page given a string from the 'Page' HTTP header string the Page HTTP Header is specified by the request with the syntax 'items X-Y'

func (*Page) AddedAfterFirst

func (p *Page) AddedAfterFirst() string

AddedAfterFirst returns the first added after as a string

func (*Page) AddedAfterLast

func (p *Page) AddedAfterLast() string

AddedAfterLast returns the last added after as a string

func (*Page) SetAddedAfters

func (p *Page) SetAddedAfters(date string)

SetAddedAfters only takes one date string and uses it to update the minimum and maximum added after fields

func (*Page) Valid

func (p *Page) Valid() bool

Valid returns whether the page is valid or not

type Status

type Status struct {
	ID               ID               `json:"id"`
	Status           string           `json:"status"`
	RequestTimestamp stones.Timestamp `json:"request_timestamp"`
	TotalCount       int64            `json:"total_count"`
	SuccessCount     int64            `json:"success_count"`
	Successes        []string         `json:"successes"`
	FailureCount     int64            `json:"failure_count"`
	Failures         []string         `json:"failures"`
	PendingCount     int64            `json:"pending_count"`
	Pendings         []string         `json:"pendings"`
}

Status represents a TAXII status object

func NewStatus

func NewStatus(objects int) (Status, error)

NewStatus returns a status struct

type StatusService

type StatusService interface {
	CreateStatus(ctx context.Context, s Status) error
	Status(ctx context.Context, statusID string) (Status, error)
	UpdateStatus(ctx context.Context, s Status) error
}

StatusService for status structs

type User

type User struct {
	Email                string `json:"email"`
	CanAdmin             bool   `json:"can_admin"`
	CollectionAccessList map[ID]CollectionAccess
}

User represents a cabby user should User and UserCollectionList be combined?

func TakeUser

func TakeUser(ctx context.Context) User

TakeUser returns the user stored in a context

func (*User) Defined

func (u *User) Defined() bool

Defined returns a bool indicating if a user is defined

func (*User) Validate

func (u *User) Validate() (err error)

Validate returns whether the object is valid or not

type UserCollectionList

type UserCollectionList struct {
	Email                string                  `json:"email"`
	CollectionAccessList map[ID]CollectionAccess `json:"collection_access_list"`
}

UserCollectionList holds a list of collections a user can access

type UserService

type UserService interface {
	CreateUser(ctx context.Context, u User, password string) error
	DeleteUser(ctx context.Context, u string) error
	UpdateUser(ctx context.Context, u User) error
	CreateUserCollection(ctx context.Context, u string, ca CollectionAccess) error
	DeleteUserCollection(ctx context.Context, u, id string) error
	UpdateUserCollection(ctx context.Context, u string, ca CollectionAccess) error
	User(ctx context.Context, user, password string) (User, error)
	UserCollections(ctx context.Context, user string) (UserCollectionList, error)
}

UserService provides Users behavior

type Versions

type Versions struct {
	Versions []string `json:"versions"`
}

Versions contains a list of versions for an object

type VersionsService

type VersionsService interface {
	Versions(c context.Context, cid, oid string, cr *Page, f Filter) (Versions, error)
}

VersionsService provides object versions

Directories

Path Synopsis
backends
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL