shunt

package
v0.3.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2019 License: GPL-3.0 Imports: 10 Imported by: 0

Documentation

Overview 

Package shunt is a general-purpose high-performance content pipeline/buffering system written in Golang. Its purpose is to provide a fast and reliable means of sending massive volumes of pre-encoded or raw data to "some destination".

At its core, shunt is really just a channel (think of a chute or conduit) circumscribed within a go routine (a lightweight thread). This channel deposits any received Items (submitted by the user or calling-application) into a buffer "evaluation process" based on each Item's "tag". If a buffer profile exists that matches the prescribed tag, said Item is subsequently placed into said buffer. Should no matching buffer profile exist, the Item is discarded. This occurs forever (the lifecycle of the calling application).

yotool/shunt also supports custom Dispatch functions; writing a function that honors the DispatchFunc type signature, one can register said function and use that for their ultimate message transmission plans/desires. For more information on writing DispatchFuncs, see the NewShunter, *Shunter.NewProfile, *Shunter.RegisterDispatchFunc, as well as the dispatch branch notes (https://godoc.org/github.com/subcon42/yotool/shunt/dispatch).

Credits

• Jesse Coretta (Main contributor/maintainer)

• Jason Mansfield (Additive log contrib, general review)

• Pawan Uberoy (original buffering prototype concept development)

See LICENSE file in the root of this repository.

Setup

To use shunt, some very basic setup is required. Let's first create our *Shunter instance: 

  package main

  import (
	"github.com/subcon42/yotool/shunt"
  )

  func main() {
	// You MUST initialize this way, but you can name it anything
	// you like, or  store the instance within a portable type of
	// your choosing.
	var S *shunt.Shunter = shunt.NewShunter()
	... other code ...
  }

For this demonstration, we'll use the built-in Stdout DispatchFunc type, which will simply index each buffer entry and print it to the console. 

  func main() {
        ... continued from above ...

	S.NewProfile(
		`LogToConsoleDemo`,	// Assign a **UNIQUE** name to your profile
		shunt.Stdout,		// The desired DispatchFunc for this profile (we'll use the built-in dispatcher here)
		shunt.Priority,		// Choose one of shunt.{Discard|Queue|Priority}
		map-or-nil,		// Optional user-authored map[string]interface{} for specialized DispatchFunc controls, etc; use nil for basic Stdout use
	)
  }

Now that we registered our profile, and aligned our profile with a known DispatchFunc, let's define the severity level(s) we're interested in.

Severity levels act like filtering mechanisms in that you can sort *Item instances based on their assigned severity, or "tone".

When registering a new profile, one must define at least one accepted *Item severity. In this example, we'll allow the shunt.Error severity. 

  func main() {
	... continued from above ...

        S.AddSeverity(`LogToConsoleDemo`, shunt.Error)
  }

Now we can craft *Item instances. *Item instances contain a payload; it could be a text message, or a sequence of JSON bytes. 

  func main() {
	... continued from above ...

        I := shunt.NewItem()
        I.SetProfile(`LogToConsoleDemo`)
        I.Errorf("We're in a lot of trouble")	// sets severity to Error *and* adds the message

        fmt.Printf("Sending 400 *Item instances\n")
        for i := 0; i <= 399; i++ {
                S.Shunt(I)
        }

  }

Thats it! 400 messages alleging some kind of trouble should have been printed to your terminal :).

Users can author their own DispatchFunc instances, register them and then enjoy yotool/shunt with the added benefits of interoperability with, perhaps, a specialized endpoint. SQL DB? LDAP DIT? Gnarly *log.Logger instance? Use your imagination! See the *Shunter.RegisterDispatchFunc() method documentation for details!

Severities

Severities mark the urgency, or "tone", of a particular *Item instance. If all of your *Item instances have the same tone (perhaps they simply contain dispassionate monitoring metrics as JSON, for example), then you may simply choose to use shunt.Payload or maybe shunt.Info, as the severity for such *Item instances.

There are built-in severities for use:

◉ Debug - set via *Item.SetDebug() -or- *Item.SetSeverity(Debug)

◉ Info - set via *Item.SetInfo() -or- *Item.SetSeverity(Info)

◉ Warning - set via *Item.SetWarning() -or- *Item.SetSeverity(Warning)

◉ Error - set via *Item.SetError() -or- *Item.SetSeverity(Error)

◉ Fatal - set via *Item.SetFatal() -or- *Item.SetSeverity(Fatal)

◉ Payload - set via *Item.SetPayload([]byte(...))

Priorities

For each profile created, it must be given a priority. A profile's priority will define how its buffer is handled (specifically, with what level of urgency).

If the priority of shunt.Priority is chosen, buffers are emptied as fast as possible. This results in many more, but smaller, buffer payloads.

However, if the priority of shunt.Queue is chosen, buffers are emptied when their maximum capacity is reached. This results in larger, but fewer, buffer payloads.

The priority of shunt.Discard is effectively a "/dev/null"; all *Item instance sent to a profile created under this priority will be immediately discarded and lost forever. This is really just useful for testing the entire "flow" of shunt, but without actually dispatching data anywhere.

Dispatch configurations

When dealing with your dispatch "driver" (e.g: your custom-written DispatchFunc that was, ostensibly, registered), you'll (usually) need to pass it some kind of information store; for this, we've used the map[string]interface{} type within the DispatchFunc signature.

This affords the user with the ability to "transport" virtually anything they need into their given DispatchFunc: A pointer instance to some DB API, an *http.Client instance, a *log.Logger instance ... pretty much anything that one can use as a value when appending a new entry to a map[string]interface{} object instance.

As a rule of thumb, only put what you absolutely need inside your configuration instance. If by some chance no configuration is needed, pass nil for the cfg Signature variable.

See dispatch/stdout.go for a (VERY basic) example of a user-authored DispatchFunc.

Item Management

The shunt-provided type, *Item, is that which you store your "data" inside; what we mean by "data", in this context, could be text messages, JSON payloads, etc. Pretty much anything capable of being expressed in bytes or as a string.

When implementing shunt into your code, ideally you'd want an "Item Factory" of sorts; this is particularly true if you're going to be sending *Item instances of varied severities and profile-destinations. Instead of doing the following every time you need an *Item instance ... 

i := shunt.NewItem()
i.SetProfile(`MyAwesomeHTTPProfile`)
i.SetSeverity(shunt.Payload)
i.SetPayload(myByteSeqVar)

... instead, you should *wrap* a similar block within a function, perhaps called makePayload(b []byte), and have it do what you need on an on-demand basis.

For further efficiency, you could even pre-seed the *Item except for the so-called "Payload" and then clone the item as needed. Its up to you and your situational needs.

Contributions

Contributes, in the form of Git Pull Requests, are welcome and encouraged. Please understand, however, that the goal of shunt was to let people write their own backend driver methods, thus we want to keep the shunt framework as generic as possible.

Example HTTP DispatchFunc

This is provided with absolutely no warranty whatsoever. This is a REFERENCE implementation of an *http.Client-capable DispatchFunc. Use/modify at your discretion. 

  // Splunk HTTP Event Collector data POSTer method for shunt
  func dispatchHTTP(sb dispatch.SmartBuffer, cfg map[string]interface{}) (bool, error) {

	var (
		client				*http.Client
		do_retry, do_newlines, ok 	bool
		err				error
		req      			*http.Request
		resp      			*http.Response
		retries				int = 5
		url, token, config		string
	)

	if len(sb) == 0 {
		return ok, fmt.Errorf("An empty buffer was sent to dispatchHTTP; ignoring")
	}

	// First make sure our configuration is healthy. We'll scan for any missing
	// parameters considered 'required' and throw an error should any be found.
	params := []string{`url`,`token`,`config`, `retry`, `newlines`, `httpClient`}
	for p := range params {
		if _, exists := cfg[params[p]]; !exists {
			err = fmt.Errorf("dispatchHTTP() method is missing a required `%s` parameter", params[p])
			return false, err
		}
		switch tv := cfg[params[p]].(type) {
		case string:
			if len(tv) == 0 {
				err = fmt.Errorf("The required parameter `%s` cannot be zero-length", params[p])
				return ok, err
			}
			if params[p] == `url` {
				url = tv
			} else if params[p] == `token` {
				token = `Splunk `+tv
			} else if params[p] == `config` {
				config = tv
			} else {
				err = fmt.Errorf("The parameter `%s` is not a string parameter", params[p])
				return ok, err
			}
		case bool:
			if params[p] == `retry` {
				do_retry = tv
			} else if params[p] == `newlines` {
				do_newlines = tv
			} else {
				err = fmt.Errorf("The parameter `%s` is not a boolean parameter", params[p])
				return ok, err
			}
		case *http.Client:
			if params[p] == `httpClient` {
				client = tv
			} else {
				err = fmt.Errorf("The parameter `%s` is not an *http.Client parameter", params[p])
				return ok, err
			}
		default:
			err = fmt.Errorf("The required parameter `%s` is an unsupported type (%T)", params[p], cfg[params[p]])
			return ok, err
		}
	}

	// Compose our HTTP Request
	req, err = http.NewRequest(`POST`, url, bytes.NewReader(sb.DumpBytes(do_newlines)))
	if err != nil {
		if logLevel.isWarning() {
			warningCLI("dispatchHTTP(%p) error: %s", dispatchHTTP, err.Error())
		}
		return ok, err
	}

	// Attempt to mitigate spurious EOF errors when
	// posting to busy/sluggish HTTP endpoints ...
	req.Close = true

	// Set our Token and Config for our headers
	req.Header.Add(`Authorization`, token)
	req.Header.Add(`X-Splunk-Request-Channel`, config)

	// Do at least one try, possibly more ...
	rc := 1
	for retries > 0 {
		resp, err = client.Do(req)
		if err != nil {
			if do_retry {
				retries -= 1 // Allow another attempt by decrementing by one due to Retry being true
				err = nil
			} else {
				retries = 0 // First failure is last failure because Retry is false
			}
			rc++
		}

		// Ditch the response body no matter what
		if resp != nil {
			if resp.StatusCode < 300 {
				ok = true
				io.Copy(ioutil.Discard, resp.Body)
				resp.Body.Close()
				break
			} else {
				err = fmt.Errorf("%s", resp.Status)
				io.Copy(ioutil.Discard, resp.Body)
				resp.Body.Close()
			}
		}

		// No more tries left, so clean up
		if retries == 0 {
			break
		}
	}

	return ok, err
  }

Index

Examples

Constants

View Source 
const (
	ProductName = `shunt`
	Version     = `0.3.8`
)

Core constants

View Source 
const (

	// Discard is akin to "/dev/null"
	Discard priorityId = iota

	// Queue waits until a buffer is full before dispatch
	Queue

	// Priority dispatches any buffer inactive after 2 seconds if non-empty
	Priority
)
View Source 
const (
	// Debug Severity loglevel
	Debug severity = 1 << iota

	// Info Severity loglevel
	Info

	// Warning Severity loglevel
	Warning

	// Error Severity loglevel
	Error

	// Fatal Severity loglevel
	Fatal

	// Payload Severity loglevel
	Payload
)
View Source 
const (

	// BytesRX represents the number of bytes received via *Shunter.Shunt()
	BytesRX = iota

	// BytesTX represents the number of bytes successfully sent to their prescribed destination
	BytesTX

	// BytesErr represents the number of bytes ultimately lost due to an error
	BytesErr

	// BytesAvail represents the number of bytes unused (wasted) during writes writes
	BytesAvail

	// WriteOK represents the number of singular (batched) writes successfully made
	WriteOK

	// WriteErr represents the number of singular (batched) writes that have failed
	WriteErr
)
View Source 
const (

	// DiscardNil represents the number of *Item instances sent to *Shunter.Shunt() that were discarded due to nilness
	DiscardNil = iota

	// DiscardEmpty represents the number of *Item instances sent to *Shunter.Shunt() that were discarded due to a zero-length payload
	DiscardEmpty

	// DiscardPriority represents the number of *Item instances sent to *Shunter.Shunt() that were discarded due to an invalid priorityId
	DiscardPriority

	// DiscardSeverity represents the number of *Item instances sent to *Shunter.Shunt() that were discarded due to an invalid severity
	DiscardSeverity

	// DiscardProfile represents the number of *Item instances sent to *Shunter.Shunt() that were discarded due to an invalid profile name
	DiscardProfile
)
View Source 
const BufferMaxLen = 500

Maximum number of payloads within a buffer

View Source 
const MaxBufferCapacity = BufferMaxLen * PayloadMaxLen

Maximum possible number of bytes that can fit into a payload

View Source 
const PayloadMaxLen = 5000

PayloadMaxLen is 5000 bytes (5KB)

View Source 
const (
	Stdout = `Stdout`
)

Built-in Dispatcher Constants

Variables

This section is empty.

Functions

This section is empty.

Types

type DispatchFunc 

type DispatchFunc func(sb dispatch.SmartBuffer, cfg map[string]interface{}) (bool, error)

The DispatchFunc type represents the required functional signature to be honored when dealing with user-crafted dispatch methods.

See the *Shunter.RegisterDispatchFunc documentation for more information on usage.

type Item 

type Item struct {
	// contains filtered or unexported fields
}

An Item is a single registered event, be it totally an error message or a data payload of some kind.

Items are created via the NewItem() method, and are then modifiable via myriad functions extended by this type.

Do note create an *Item instance via: 

var item *Item

... nor ... 

item := &Item{}

Instances of *Items must be properly initialized using the NewItem() method mentioned above. For example: 

var item *Item = shunt.NewItem()

... or ... 

item := shunt.NewItem()

Once this is done, you are free to use the extended methods as seen fit.

Items contain a payload. This is a sequence of bytes ([]byte). Such a sequence can be anything text or byte-based:

  • JSON
  • fmt.Sprintf()-type stuff
  • Arbitrary sequences

One can set a single payload per *Item instance. If the data is a string, use SetPayloadFromString(), if it is a sequence of bytes ([]byte), use SetPayloadFromBytes().

func NewItem 

func NewItem() *Item
NewItem creates a new instance of *Item which can subsequently be handled by a user as needed.

Once suitably tweaked, an *Item instance is then tossed into the *Shunter instance's so-called Item Channel for (eventual) handling.

Example 
ni := NewItem()
fmt.Printf("%T", ni)

Output:

*shunt.Item

func (*Item) Debugf 

func (I *Item) Debugf(text string, sp ...interface{})

Debugf accepts a text sequence and format specifies, which will become a payload. Additionally, the *Item instance will have its severity set to Debug.

Example (Severity) 
ni := NewItem()
ni.Debugf("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Debug

func (*Item) Error 

func (I *Item) Error() error
Error will return an *Item's payload data as an error type instance. If there was no data, nil is returned.

This method will not change the severity of the *Item instance in question; it only converts a payload to the built-in error type via the *Item.PayloadToString() method.

func (*Item) Errorf 

func (I *Item) Errorf(text string, sp ...interface{})
Errorf accepts a text sequence and format specifies, which will become a payload. Additionally, the *Item instance will have its severity set to Error.

This is essentially a convenience method for both *Item.Sprintf() and *Item.SetError() in one.

Example (Severity) 
ni := NewItem()
ni.Errorf("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Error

func (*Item) Fatalf 

func (I *Item) Fatalf(text string, sp ...interface{})
Fatalf accepts a text sequence and format specifies, which will become a payload.

Additionally, the *Item instance will have its severity set to Fatal.

Example (Severity) 
ni := NewItem()
ni.Fatalf("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Fatal

func (*Item) GetProfile 

func (I *Item) GetProfile() string

GetProfile returns the currently-set profile string name of an *Item instance, else an empty string if unset.

Example 
ni := NewItem()
ni.SetProfile(`MYPROFILENAME`)
fmt.Printf("%s", ni.GetProfile())

Output:

MYPROFILENAME

func (*Item) GetSeverity 

func (I *Item) GetSeverity() severity

GetSeverity returns the currently-set severity

Example 
ni := NewItem()
ni.SetSeverity(Debug)
fmt.Printf("%s", ni.GetSeverity())

Output:

Debug

func (*Item) Infof 

func (I *Item) Infof(text string, sp ...interface{})

Infof accepts a text sequence and format specifies, which will become a payload. Additionally, the *Item instance will have its severity set to Info.

Example (Severity) 
ni := NewItem()
ni.Infof("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Info

func (*Item) IsDebug 

func (I *Item) IsDebug() bool

IsDebug simply returns a bool indicating whether it is a debug-level message.

Example 
ni := NewItem()
ni.SetSeverity(Debug)
fmt.Printf("%t", ni.IsDebug())

Output:

true

func (*Item) IsError 

func (I *Item) IsError() bool

IsError simply returns a bool indicating whether it is an error.

Example 
ni := NewItem()
ni.SetSeverity(Error)
fmt.Printf("%t", ni.IsError())

Output:

true

func (*Item) IsFatal 

func (I *Item) IsFatal() bool

IsFatal simply returns a bool indicating whether it is considered a fatal error.

Example 
ni := NewItem()
ni.SetSeverity(Fatal)
fmt.Printf("%t", ni.IsFatal())

Output:

true

func (*Item) IsInfo 

func (I *Item) IsInfo() bool

IsInfo simply returns a bool indicating whether it is an informational message.

Example 
ni := NewItem()
ni.SetSeverity(Info)
fmt.Printf("%t", ni.IsInfo())

Output:

true

func (*Item) IsPayload 

func (I *Item) IsPayload() bool
IsPayload simply returns a bool indicating whether the *Item instance is a payload of neutral severity in nature.

This method shall only check severity status; it does not validate the presence (or absence) of a payload value.

Example 
ni := NewItem()
ni.SetSeverity(Payload)
fmt.Printf("%t", ni.IsPayload())

Output:

true

func (*Item) IsWarning 

func (I *Item) IsWarning() bool

IsWarning simply returns a bool indicating whether it is a warning.

Example 
ni := NewItem()
ni.SetSeverity(Warning)
fmt.Printf("%t", ni.IsWarning())

Output:

true

func (*Item) Len 

func (I *Item) Len() int

Len returns the calculated length of an enclosed payload.

Example 
ni := NewItem()
ni.Payloadf(`{"myReally": "longJSON!"}`)
fmt.Printf("%d characters long", ni.Len())

Output:

25 characters long

func (*Item) PayloadToBytes 

func (I *Item) PayloadToBytes(nl bool) []byte

PayloadToBytes will attempt to return the raw byte sequence stored as the payload. If not found, return an empty byte sequence.

Example 
ni := NewItem()
ni.Payloadf(`{"myReally": "longJSON!"}`)
fmt.Printf("%T", ni.PayloadToBytes(false))

Output:

[]uint8

func (*Item) PayloadToString 

func (I *Item) PayloadToString(nl bool) string

PayloadToString will attempt to return the string-form of a raw byte sequence stored as the payload. If not found, return an untyped empty string.

Example 
ni := NewItem()
ni.Payloadf(`{"myReally": "longJSON!"}`)
fmt.Printf("%T", ni.PayloadToString(false))

Output:

string

func (*Item) Payloadf 

func (I *Item) Payloadf(text string, sp ...interface{})
Payloadf accepts a text sequence and format specifies, which will become a payload. Additionally, the *Item instance will have its severity set to Payload.

This method is nearly identical to that of *Item.SetPayload(), except this method supports format specifiers.

Example (Severity) 
ni := NewItem()
ni.Payloadf("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Payload

func (*Item) Println 

func (I *Item) Println()
Println accepts no arguments and will print (if set) any byte sequence within an *Item.payload. Note this is neutral context (not meant for error conditions), rather in place of fmt.PrintX(...).

Because of the under-the-hood use of `fmt.Println()`, a newline will be added to all prints (thus your input text string(s) need not include newline characters).

func (*Item) SetPayload 

func (I *Item) SetPayload(seq interface{})
SetPayload sets the given string or byte sequence as the payload of said *Item, and will

set the *Item severity to Payload.

Example 
ni := NewItem()
ni.SetPayload([]byte(`l0t$_oF_g@RbAg3`))
fmt.Printf("%t", ni.IsPayload())

Output:

true

func (*Item) SetProfile 

func (I *Item) SetProfile(profile string)

SetProfile sets a given item with a profile name. This string is used (later) to determine where an instance of *Item ought to be buffered for handling. Note this will not validate the existence of the prescribed profile.

Example 
ni := NewItem()
ni.SetProfile(`MYPROFILENAME`)
fmt.Printf("%s", ni.GetProfile())

Output:

MYPROFILENAME

func (*Item) SetSeverity 

func (I *Item) SetSeverity(sev interface{})
SetSeverity accepts an int or a string as the assigned "severity" for this particular *Item instance.

No checks are conducted to determine if an assigned severity is registered within an instance of *Shunter until a given *Item instance is sent into the *Shunter.Shunt() channel method.

Example 
ni := NewItem()
ni.SetSeverity(Debug)
fmt.Printf("%s", ni.GetSeverity())

Output:

Debug

func (*Item) Warningf 

func (I *Item) Warningf(text string, sp ...interface{})

Warningf accepts a text sequence and format specifies, which will become a payload. Additionally, the *Item instance will have its severity set to Warning.

Example (Severity) 
ni := NewItem()
ni.Warningf("This is a really ugly debug string")
fmt.Printf("%s", ni.GetSeverity())

Output:

Warning

type Shunter 

type Shunter struct {
	// contains filtered or unexported fields
}

Shunter provides assorted functionalities offered by this library, as well as user-registered/ influenced contents. It **must be** produced by NewShunter() (and not via typical 'var' or ':=' declaration).

func NewShunter 

func NewShunter() *Shunter

NewShunter creates and returns an instance of *Shunter, initialized and ready for use.

Example 
s := NewShunter()
fmt.Printf("%T", s)

Output:

*shunt.Shunter

func (*Shunter) AddProfileSeverity 

func (S *Shunter) AddProfileSeverity(p string, s severity)

AddProfileSeverity will add the severity (s) to the specified profile (p)

Example 
s := NewShunter()
s.NewProfile(`COOL_PROFILE_BRO`, Stdout, Discard, nil)
s.AddProfileSeverity(`COOL_PROFILE_BRO`, Error)
fmt.Printf("%t", s.ProfileIsError(`COOL_PROFILE_BRO`))

Output:

true

func (*Shunter) DispatchFuncRegistered 

func (S *Shunter) DispatchFuncRegistered(n string) bool

DispatchFuncRegistered returns a bool indicating whether the specified DispatchFunc is registered in this instance of *Shunter.

func (*Shunter) DropProfileSeverity 

func (S *Shunter) DropProfileSeverity(p string, s severity)

DropProfileSeverity removes the severity (s) from the specified profile (p)

Example 
s := NewShunter()
s.NewProfile(`COOL_PROFILE_BRO`, Stdout, Discard, nil)
s.AddProfileSeverity(`COOL_PROFILE_BRO`, Error)
s.DropProfileSeverity(`COOL_PROFILE_BRO`, Error)
fmt.Printf("%t", s.ProfileIsError(`COOL_PROFILE_BRO`))

Output:

false

func (*Shunter) GetStats 

func (S *Shunter) GetStats() roStatisticalSet
GetStats will return, if not nil, the current populated read-only statistical set.  This will contain global totals of all buffer TX/RX bytes, number of successful/unsuccessful writes, etc.  This method does not return the individual "per-profile" statistical sets, but will contain information representing the collective total of such sets.

The statistical set design of this package is based on the awesome go-metrics package (https://github.com/rcrowley/go-metrics).

func (*Shunter) ItemOK 

func (S *Shunter) ItemOK(I *Item) (bool, error)
ItemOK will parse the given *Item instance for obvious errors.  Specifically:

◉ Whether the *Item instance is nil

◉ Whether a KNOWN severity is set

◉ If a payload is set, regardless of severity

If all of the above checks pass without incident, a boolean of true is returned with a nil error. Otherwise, false is returned alongside a non-nil error.

This will incur some performance penalties if run very often.

func (*Shunter) NewProfile 

func (S *Shunter) NewProfile(pname, dname string, priority priorityId, cfg map[string]interface{}) error

NewProfile creates and registers a new *shunt.profile instance.

The configuration arguments (cfg) in the form of a map[string]interface{} may be required and, if so, are extremely important. This configuration structure would contain items needed for successful dispatch of data, such as:

  • Credentials if talking to something that requires authentication
  • Any headers name/value pairs if talking to something like an HTTP server that requires particular HTTP headers be set
  • Any endpoint addresses, such as a log filename, an HTTP URL, Database DSN, etc ...
  • Whether newlines are required at the termination of each dispatched buffer entry or not
  • Whether your "driver" should attempt post/write retries or not
  • ... much much more ... use your imagination ...

As you can see, the configuration object is highly-flexible and completely at the discretion of the user. As a rule of thumb, however, only put in what you absolutely need. Remember, at this stage of the process, your data is packaged and ready-to-go. It's just waiting for your shipping label, so to speak.

In the rare cases where NO configuration object is needed, nil can be passed in lieu of an actual, populated map instance. The only active example of such a setup would be use of the built-in DispatchFunc known as dispatch.Stdout, which (when given such a nil configuration instance), would print out the entire buffer contents -without any newline terminators- to Stdout by way of the Go "fmt" package.

Example 
s := NewShunter()

// Craft our configuration (only put in what we really need
// or just send nil - it depends on the dispatcher) ...
cfg := make(map[string]interface{})
cfg[`reallyImportantThing`] = `whatever`

// We'll point this to Stdout (the built-in DispatchFunc),
// and we'll instruct shunt to treat data sent to this
// profile's buffer as HIGH PRIORITY.
s.NewProfile(`MY_COOL_PROFILE`, Stdout, Priority, cfg)
fmt.Printf("%t", s.ProfileExists(`MY_COOL_PROFILE`))

Output:

true

func (*Shunter) ProfileExists 

func (S *Shunter) ProfileExists(n string) bool

ProfileExists method returns a bool indicating whether the specified profile string name is registered in this instance of *Shunter.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
fmt.Printf("%t", s.ProfileExists(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsDebug 

func (S *Shunter) ProfileIsDebug(p string) bool

ProfileIsDebug returns a bool indicating whether submitted *Item instances bearing a severity of DEBUG will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Debug)
fmt.Printf("%t", s.ProfileIsDebug(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsDiscard 

func (S *Shunter) ProfileIsDiscard(p string) bool

ProfileIsDiscard returns a bool indicating whether the specified profile's priorityId is set to Discard (effectively a "/dev/null")

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
fmt.Printf("%t", s.ProfileIsDiscard(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsError 

func (S *Shunter) ProfileIsError(p string) bool

ProfileIsError returns a bool indicating whether submitted *Item instances bearing a severity of ERROR will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Error)
fmt.Printf("%t", s.ProfileIsError(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsFatal 

func (S *Shunter) ProfileIsFatal(p string) bool

ProfileIsFatal returns a bool indicating whether submitted *Item instances bearing a severity of FATAL will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Fatal)
fmt.Printf("%t", s.ProfileIsFatal(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsInfo 

func (S *Shunter) ProfileIsInfo(p string) bool

ProfileIsInfo returns a bool indicating whether submitted *Item instances bearing a severity of INFO will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Info)
fmt.Printf("%t", s.ProfileIsInfo(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsPayload 

func (S *Shunter) ProfileIsPayload(p string) bool

ProfileIsPayload returns a bool indicating whether submitted *Item instances bearing a severity of PAYLOAD will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Payload)
fmt.Printf("%t", s.ProfileIsPayload(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsPriority 

func (S *Shunter) ProfileIsPriority(p string) bool

ProfileIsPriority returns a bool indicating whether the specified profile's priorityId is set to Priority.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Priority, nil)
fmt.Printf("%t", s.ProfileIsPriority(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsQueue 

func (S *Shunter) ProfileIsQueue(p string) bool

ProfileIsQueue returns a bool indicating whether the specified profile's priorityId is set to Queue.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Queue, nil)
fmt.Printf("%t", s.ProfileIsQueue(`DEV_NULL`))

Output:

true

func (*Shunter) ProfileIsWarning 

func (S *Shunter) ProfileIsWarning(p string) bool

ProfileIsWarning returns a bool indicating whether submitted *Item instances bearing a severity of WARNING will be accepted or not by the specified profile name.

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Discard, nil)
s.AddProfileSeverity(`DEV_NULL`, Warning)
fmt.Printf("%t", s.ProfileIsWarning(`DEV_NULL`))

Output:

true

func (*Shunter) RegisterDispatchFunc 

func (S *Shunter) RegisterDispatchFunc(name string, function DispatchFunc) error

RegisterDispatchFunc accepts a unique name, and a DispatchFunc which uses the signature: 

func(sb dispatch.SmartBuffer, cfg map[string]interface{}) (bool, error)

This affords the user with the ability to write their own Dispatch methods. Such methods should be registered BEFORE one begins to craft and shunt *Item instances, and before *Shunter.NewProfile() is run.

To use a DispatchFunc, one first creates an instance of *Shunter as shown in the NewShunter() examples. With that said, once *Shunter is ready, one executes the following (assuming S as the variable name for a properly-instantiated instance of *Shunter): 

S.RegisterDispatchFunc(`<DESIRED-NAME>`, func(sb dispatch.SmartBuffer, cfg map[string]interface{}) (bool, error) {
      // ... Write your custom function code ... //
})

Your effective code should return a boolean of true (ok) if no errors, along with a nil error instance. If all is not well, a false boolean should be returned alongside a non-nil error.

All registered (and prescribed) instances of DispatchFunc are run at the figurative "end" of the *Item routing process before final purge and iterative restart.

See the "dispatch" branch of this package hierarchy for more details, and an example of Stdout use.

Example 
s := NewShunter()
s.RegisterDispatchFunc(`RAD_HTTPCLIENT`, func(sb dispatch.SmartBuffer, cfg map[string]interface{}) (bool, error) {
	// ... my cool code ... //
	return true, nil
})
fmt.Printf("%t", s.DispatchFuncRegistered(`RAD_HTTPCLIENT`))

Output:

true

func (*Shunter) SetDebugOutput 

func (S *Shunter) SetDebugOutput(b bool)

SetDebugOutput (when given a bool) will commence/cease debug-level output to the terminal. Colors can be toggled via *Shunter.UseColors(bool).

Example 
s := NewShunter()

// Enable it
s.SetDebugOutput(true)

// Disable it
s.SetDebugOutput(false)

Output:

func (*Shunter) SetInfoOutput 

func (S *Shunter) SetInfoOutput(b bool)

SetInfoOutput (when given a bool) will commence/cease informational output to the terminal.

Colors can be toggled via *Shunter.UseColors(bool).

Example 
s := NewShunter()

// Enable it
s.SetInfoOutput(true)

// Disable it
s.SetInfoOutput(false)

Output:

func (*Shunter) SetPayloadOutput 

func (S *Shunter) SetPayloadOutput(b bool)

SetPayloadOutput (when given a bool) will commence/cease successful payload-related output to the terminal.

Colors can be toggled via *Shunter.UseColors(bool).

Example 
s := NewShunter()

// Enable it
s.SetPayloadOutput(true)

// Disable it
s.SetPayloadOutput(false)

Output:

func (*Shunter) SetWarningOutput 

func (S *Shunter) SetWarningOutput(b bool)

SetWarningOutput (when given a bool) will commence/cease non-error output, pertaining to suboptimal conditions, to the terminal.

Colors can be toggled via *Shunter.UseColors(bool).

Example 
s := NewShunter()

// Enable it
s.SetWarningOutput(true)

// Disable it
s.SetWarningOutput(false)

Output:

func (*Shunter) Shunt 

func (S *Shunter) Shunt(I *Item)

Shunt will dispatch the supplied *Item instance into "The Channel". Beyond checking "nilness", this method does not discriminate nor filter *Item instances.

Additionally, for performance reasons, this method will not run an implicit *Shunter.ItemOK() run. If you pass a bunk *Item instance into the Shunt() method/channel, it will (eventually) get weeded out during the *Shunter siphoning stage (albeit at the cost of some performance), but will throw errors.

Given a properly configured *Shunter instance, this method shall "get your stuff where you want it".

Example 
s := NewShunter()
s.NewProfile(`DEV_NULL`, Stdout, Priority, nil)
s.AddProfileSeverity(`DEV_NULL`, Payload)

i := NewItem()
i.Payloadf(`{"blargetty": "blarg"}`)
s.Shunt(i)

Output:

func (*Shunter) UseColors 

func (S *Shunter) UseColors(u bool)

UseColors allows toggling of color-based terminal output via boolean value. Note that this edits a global var, which means for situations where multiple *Shunter instances are active, you'll need to make sure this feature isn't toggled against your wishes.

Source Files

View all Source files

Directories

Path Synopsis
Package dispatch simply contains any built-in instances of DispatcherFunc.
 Package dispatch simply contains any built-in instances of DispatcherFunc.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.