scotty

package module
v0.0.0-...-fe13f99 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 1, 2018 License: Apache-2.0 Imports: 7 Imported by: 9

README

Scotty

Scotty is a high performance, scalable collector for the tricorder metrics publishing library. Scotty provides a RESTful API to grab the latest polled metrics, and it can push metrics to various persistent stores.

Please see the design document and the online code documentation for more information.

Contributions

Prior to receiving information from any contributor, Symantec requires that all contributors complete, sign, and submit Symantec Personal Contributor Agreement (SPCA). The purpose of the SPCA is to clearly define the terms under which intellectual property has been contributed to the project and thereby allow Symantec to defend the project should there be a legal dispute regarding the software at some future time. A signed SPCA is required to be on file before an individual is given commit privileges to the Symantec open source project. Please note that the privilege to commit to the project is conditional and may be revoked by Symantec.

If you are employed by a corporation, a Symantec Corporate Contributor Agreement (SCCA) is also required before you may contribute to the project. If you are employed by a company, you may have signed an employment agreement that assigns intellectual property ownership in certain of your ideas or code to your company. We require a SCCA to make sure that the intellectual property in your contribution is clearly contributed to the Symantec open source project, even if that intellectual property had previously been assigned by you.

Please complete the SPCA and, if required, the SCCA and return to Symantec at:

Symantec Corporation Legal Department Attention: Product Legal Support Team 350 Ellis Street Mountain View, CA 94043

Please be sure to keep a signed copy for your records.

LICENSE

Copyright 2015 Symantec Corporation.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Building and Running Tests

When building scotty for the very first time, perform the following steps to install the correct dependencies.

go get github.com/Symantec/scotty
cd $GOPATH/src/github.com/Symantec/scotty
make getdeps

From the top level directory of the scotty project:

To run all the tests

go test -v ./...

To rebuild after doing code changes

go install ./...

Documentation

Overview

Package scotty collects endpoint health metrics asynchronously.

Package scotty collects endpoint health metrics asynchronously. The scotty code and the REST API that the scotty application provides are subject to change.

Fetching metrics from scotty using REST API

REST urls (always use GET) 

http://scottyhostname.com:6980/api/hosts/someHostName/someAppName
	Returns every metric from someAppName on someHostName
	along with values for the last hour.
http://scottyhostname.com:6980/api/hosts/someHostName/someAppName/a/path
	Returns every metric under /a/path from someAppName on
	someHostName along with values for the last hour.
	If no metrics match, returns an empty list.
http://scottyhostname.com:6980/api/latest/a/path
	Returns the latest values of all metrics under /a/path on all
	endpoints sorted by hostname first, appname second, and path third.
http://scottyhostname.com:6980/api/errors
	Returns a list of every endpoint that scotty cannot currently
	reach along with the timestamp of the last attempt and the
	error encountered. If no errors, returns an empty list.

Global optional REST Query Parameters 

format=text
	Adds indentation and line feeds to JSON to make it human
	readable.

Optional REST Query parameters for /api/hosts calls. 

singleton=true
	For the second api/hosts URL listed, returned metrics
	must match the specified metric path exactly. Use if you
	want a particular metric rather than all the metrics under
	a particular path.
history=123
	Shows values for the last 123 minutes. history=0 shows only
	the most recent values. If omitted, default is 60 minutes.

Sample JSON for api/hosts calls. 

[
	{
		"path": "/proc/args",
		"description": "Program args",
		"unit": "None",
		"kind": "string",
		"values": [
			{
				"timestamp": "1450810183.939569234",
				"value": "-r -t -n",
				"active": true
			}
		]
	},
	{
		"path": "/proc/cpu/sys",
		"description": "User CPU time used",
		"unit": "Seconds",
		"kind": "duration",
		"values": [
			{
				"timestamp": "1450812042.050224065",
				"value": "579.000432200"
				"active": true
			},
			{
				"timestamp": "1450811980.899730682",
				"value": "579.000167902"
				"active": true
			},
			{
				"timestamp": "1450811923.538924217",
				"value": "579.000008265"
				"active": true
			}
		}
	}
]

Sample JSON for api/error call 

[
	{
		"hostName": "b-imserv-r07e14-prod.ash1.symcpe.net*1037",
		"timestamp": "1450812819.791744615",
		"error": "dial tcp 10.119.150.73:7776: getsockopt: connection refused"
	},
	{
		"hostName": "b-imserv-r07e14-prod.ash1.symcpe.net*2037",
		"timestamp": "1450812819.807774730",
		"error": "dial tcp 10.119.150.73:7776: getsockopt: connection refused"
	}
]

Sample JSON for api/latest call 

[
	{
		"hostName": "first.net",
		"appName": "My App",
		"path": "/proc/args",
		"description": "Program args",
		"unit": "None",
		"kind": "string",
		"timestamp": "1476307345.592506885",
		"value": "-f --size=1000"
	},
	{
		"hostName": "second.net",
		"appName": "My App",
		"path": "/proc/args",
		"description": "Program args",
		"unit": "None",
		"kind": "string",
		"timestamp": "1476307345.554426431",
		"value": "--size=2000"
	},

	...
]

For more information on the json schema, see the scotty/messages package

Scotty GO RPC

Scotty GO RPC is available on the same port as the REST API, usually 6980. You can see these methods by visiting http://scottyhostname.com:6980/debug/rpc

Scotty.Latest

Gets the latest available metrics under a given path for all active endpoints. Pass the absolute path as the input. Empty string ("") means get all the latest available metrics. "/foo" means get the metric "/foo" from each endpoint or any metric underneath "/foo" from each endpoint. The output are the latest metrics as []*messages.LatestMetric. The output are sorted first by hostname then by application name and then by metric path.

GO RPC scotty.Latest Example code: 

import "fmt"
import "github.com/Symantec/scotty/messages"
import "log"
import "net/rpc"

client, _ := rpc.DialHTTP("tcp", "a.hostname.net:6980")
defer client.Close()
var latest []*messages.LatestMetric
if err := client.Call("Scotty.Latest", "/path/to/metrics", &latest); err != nil {
	log.Fatal(err)
}
for _, m := range latest {
	fmt.Println("Path: ", m.Path)
	fmt.Println("Timestamp: ", m.Timestamp)
	fmt.Println("Value: ", m.Value)
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConcurrentConnects 

func ConcurrentConnects() uint

ConcurrentConnects returns the maximum number of concurrent connects.

func ConcurrentPolls 

func ConcurrentPolls() uint

ConcurrentPolls returns the maximum number of concurrent polls. Default is 2 * number of CPUs. A return of 0 means no limit.

func SetConcurrentConnects 

func SetConcurrentConnects(x uint)

SetConcurrentConnects sets the maximum number of concurrent connects. Call SetConcurrentConnects at the beginning of the main() function before calling Endpoint.Poll

func SetConcurrentPolls 

func SetConcurrentPolls(x uint)

SetConcurrentPolls sets the maximum number of concurrent polls. Zero means no limit. Call SetConcurrentPolls at the beginning of the main() function before calling Endpoint.Poll

Types

type Endpoint 

type Endpoint struct {
	// contains filtered or unexported fields
}

Endpoint represents a particular endpoint with health metrics. Endpoint instances are safe to use with multiple goroutines.

func NewEndpointWithConnector 

func NewEndpointWithConnector(
	hostId *hostid.HostID, appName string, connector sources.Connector) *Endpoint

NewEndpointWithConnector creates a new endpoint for given host, port and connector.

func (*Endpoint) AppName 

func (e *Endpoint) AppName() string

AppName returns the app name of the endpoint.

func (*Endpoint) ConnectorName 

func (e *Endpoint) ConnectorName() string

ConnectorName returns the name of the underlying connector in this endpoint.

func (*Endpoint) HostName 

func (e *Endpoint) HostName() string

HostName returns the host name of the endpoint.

func (*Endpoint) IpAddress 

func (e *Endpoint) IpAddress() string

IpAddress returns the IP Address if known or the empty string if not known.

func (*Endpoint) Poll 

func (e *Endpoint) Poll(
	sweepStartTime time.Time, isTls bool, port uint, logger Logger)

Poll polls for metrics for this endpoint asynchronously. However, Poll may block while it waits to begin connecting if too many requests for metrics are in progress. Poll returns immediately if this instance is already in the process of collecting metrics. sweepStartTime is the start time of the current collection of metrics. isTls indicates whether or not TLS should be used port is the port to use to connect. logger logs collection events for this polling

type Logger 

type Logger interface {
	// Called when new metrics come in from a given endpoint. If
	// implementation returns a non-nil error, state goes to
	// FailedtoPoll. Otherwise, state goes to Synced.
	LogResponse(
		e *Endpoint, response metrics.List, timestamp time.Time) error
	// Called when error happens collecting metrics from a given
	// endpoint.
	// Also called when an error clears. In such a case both err and
	// state are nil.
	LogError(e *Endpoint, err error, state *State)
	// Called when collection status changes on a given endpoint
	LogStateChange(e *Endpoint, oldState, newState *State)
}

Logger is the interface for instances that log metric collection events. Endpoint instances call Logger methods immediately after updating themselves. Logger instances must be safe to use among multiple goroutines.

type State 

type State struct {
	// contains filtered or unexported fields
}

State represents the state of collecting metrics from a endpoint. State instances are immutable.

func (*State) ConnectorName 

func (s *State) ConnectorName() string

ConnectorName returns the name of the connector

func (*State) Status 

func (s *State) Status() Status

Status returns the status of the collection.

func (*State) SweepStartTime 

func (s *State) SweepStartTime() time.Time

SweepStartTime returns the start time of the sweep.

func (*State) TimeSpentConnecting 

func (s *State) TimeSpentConnecting() time.Duration

TimeSpentConnecting returns the time spent establishing a connection.

func (*State) TimeSpentPolling 

func (s *State) TimeSpentPolling() time.Duration

TimeSpentPolling returns the time spent downloading the metrics.

func (*State) TimeSpentWaitingToConnect 

func (s *State) TimeSpentWaitingToConnect() time.Duration

TimeSpentWaitingToConnect returns the time elapsed from the start of the sweep until establishing a connection first commences.

func (*State) TimeSpentWaitingToPoll 

func (s *State) TimeSpentWaitingToPoll() time.Duration

TimeSpentWaitingToPoll returns time elpased from when the connection was established to when downloading metrics first commences.

func (*State) Timestamp 

func (s *State) Timestamp() time.Time

Timestamp returns the timestamp of the last state change.

type Status 

type Status int

Status represents the status of metric collection for a particular endpoint. Failure statuses are negative, success statuses are positive. 

const (
	// failed to connect to endpoint
	FailedToConnect Status = -1 - iota
	// connected, but failed to collect metrics
	FailedToPoll
)
const (
	Unknown Status = iota
	WaitingToConnect
	Connecting
	WaitingToPoll
	Polling
	Synced
)

func (Status) String 

func (s Status) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package application manages individual applications for scotty.
 Package application manages individual applications for scotty.
apps
cloudhealthtestserver
cloudhealthtestserver This application is a fake cloudhealth endpoint.
 cloudhealthtestserver This application is a fake cloudhealth endpoint.
kafka
scotty
scotty/showallapps
scotty/splash
scottytool
Package awsinfo contains routines to extract from AWS metadata
 Package awsinfo contains routines to extract from AWS metadata
Package chpipeline manages collecting data for cloudhealth and other systems
 Package chpipeline manages collecting data for cloudhealth and other systems
The cis package handles sending data to CIS.
 The cis package handles sending data to CIS.
Package cloudhealth contains routines for writing to cloud health
 Package cloudhealth contains routines for writing to cloud health
cloudhealthlmm contains routines for writing cloudhealth data to lmm.
 cloudhealthlmm contains routines for writing cloudhealth data to lmm.
The cloudwatch package writes data to aws cloudwatch
 The cloudwatch package writes data to aws cloudwatch
Package consul integrates scotty with Consul.
 Package consul integrates scotty with Consul.
Package endpointdata contains a data structure for collecting data from endpoints.
 Package endpointdata contains a data structure for collecting data from endpoints.
Package hostid contains types to identify hosts
 Package hostid contains types to identify hosts
influx
qlutils
Package qlutils contains utilities for handling influx ql queries.
 Package qlutils contains utilities for handling influx ql queries.
responses
Package responses contains routines for handling influx ql result sets.
 Package responses contains routines for handling influx ql result sets.
lib
apiutil
Package apiutil contains routines for creating API endpoints.
 Package apiutil contains routines for creating API endpoints.
dynconfig
Package dynconfig provides routines for implementing dynamic configuration files.
 Package dynconfig provides routines for implementing dynamic configuration files.
gate
Package gate provides management of critical sections
 Package gate provides management of critical sections
httputil
Package httputil contains various http utilities.
 Package httputil contains various http utilities.
jsonutil
Package jsonutil provides utilities for reading and unmarshaling JSON including strict unmarshaling of JSON into go structs.
 Package jsonutil provides utilities for reading and unmarshaling JSON including strict unmarshaling of JSON into go structs.
keyedqueue
Package keyedqueue provides a queue with keyed elements allowing elements already on the queue to be easily replaced.
 Package keyedqueue provides a queue with keyed elements allowing elements already on the queue to be easily replaced.
pool
Package pool provides pools for io.Closer instances.
 Package pool provides pools for io.Closer instances.
preference
Package preference contains routines for maintaining order of preference for expensive processes such as RPC methods.
 Package preference contains routines for maintaining order of preference for expensive processes such as RPC methods.
queuesender
Package queuesender implements asynchronous http.
 Package queuesender implements asynchronous http.
retry
Package retry allows multiple goroutines to function until it succeeds.
 Package retry allows multiple goroutines to function until it succeeds.
synchttp
Package synchttp writes JSON to an endpoint and waits for a response.
 Package synchttp writes JSON to an endpoint and waits for a response.
trimetrics
Package trimetrics contains routines to facilitate exposing metrics in tricorder.
 Package trimetrics contains routines to facilitate exposing metrics in tricorder.
yamlutil
Package yamlutil provides utilities for reading and unmarshaling YAML including strict unmarshaling of YAML into go structs.
 Package yamlutil provides utilities for reading and unmarshaling YAML including strict unmarshaling of YAML into go structs.
Package machine keeps track of machines and applications for scotty.
 Package machine keeps track of machines and applications for scotty.
Package messages provides types for JSON and go rpc.
 Package messages provides types for JSON and go rpc.
Package metrics provides a view of metrics that is independent of source.
 Package metrics provides a view of metrics that is independent of source.
Package namesandports contains the NamesAndPorts datastructure.
 Package namesandports contains the NamesAndPorts datastructure.
Packge pstore and sub packages handle writing metrics to persistent storage.
 Packge pstore and sub packages handle writing metrics to persistent storage.
config
Package config includes utilities for handling configuration files.
 Package config includes utilities for handling configuration files.
config/influx
Package influx enables writing metric values to influxdb.
 Package influx enables writing metric values to influxdb.
config/kafka
Package kafka enables writing metric values to kafka.
 Package kafka enables writing metric values to kafka.
config/lmm
Package lmm enables writing metric values to lmm.
 Package lmm enables writing metric values to lmm.
config/mock
Package mock enables writing metric values to a mockdb for testing.
 Package mock enables writing metric values to a mockdb for testing.
config/tsdb
Package tsdb enables writing metric values to tsdb.
 Package tsdb enables writing metric values to tsdb.
Package sources provides the interfaces for metric sources.
 Package sources provides the interfaces for metric sources.
jsonsource
Package jsonsource connects to sources using tricorder json.
 Package jsonsource connects to sources using tricorder json.
loadsource
Package loadsource provides bogus metrics for load testing.
 Package loadsource provides bogus metrics for load testing.
selfsource
Package selfsource get metrics for current process
 Package selfsource get metrics for current process
snmpsource
Package snmpsource connects to sources using snmp
 Package snmpsource connects to sources using snmp
trisource
Package trisource connects to sources using tricorder.
 Package trisource connects to sources using tricorder.
Package store handles storing metric values in memory.
 Package store handles storing metric values in memory.
btreepq
Package btreepq implements a queue of pages for scotty based on btrees.
 Package btreepq implements a queue of pages for scotty based on btrees.
Package suggest contains routines to implement a suggest engine Package suggest contains routines to implement a suggest engine
 Package suggest contains routines to implement a suggest engine Package suggest contains routines to implement a suggest engine
Package sysmemory tells scotty how much memory to use.
 Package sysmemory tells scotty how much memory to use.
package tsdb contains the data structures for emulating the TSDB API.
 package tsdb contains the data structures for emulating the TSDB API.
aggregators
Package aggregators contains aggregator factory methods.
 Package aggregators contains aggregator factory methods.
Package tsdbexec is the top level package for serving tsdb requests.
 Package tsdbexec is the top level package for serving tsdb requests.
package tsdbimpl contains routines for internally fulfilling tsdb APi requests.
 package tsdbimpl contains routines for internally fulfilling tsdb APi requests.
Package tsdbjson handles tsdb JSON requests and responses.
 Package tsdbjson handles tsdb JSON requests and responses.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.