README
¶
Neptune agent
Neptune agent is a lightweight piece of software written in GO, which runs on your servers to run runbooks and report the results back to Neptune. We use an agent to accomplish this because it provides an easy way to run runbooks without exposing any open ports and without requiring SSH access to your servers.
Installation
Documentation is available at http://docs.neptune.io/docs/agent-installation
Administration
Please see http://docs.neptune.io/docs/agent-administration
Architecture white paper
This document has more details about threat model and different attack vectors. Please see https://s3.amazonaws.com/prod-neptuneio-downloads/security/agent/NeptuneArchitectureWhitePaper.pdf
Agent FAQs
Please see FAQs at http://docs.neptune.io/docs/agent-faqs
Architecture
Read more at http://docs.neptune.io/docs/agent-faqs#architecture-and-more
Network Security
- Traffic is always initiated by the agent to Neptune. No sessions are ever initiated from Neptune back to the agent
- All traffic is sent over SSL
- The destination for all agent data is: www.neptune.io It is a CNAME; its IP address is subject to change but belongs to the ranges:
23.20.0.0/14 (23.20.0.1 - 23.23.255.254)
50.16.0.0/15 (50.16.0.1 - 50.17.255.254)
50.19.0.0/16 (50.19.0.1 - 50.19.255.254)
52.0.0.0/15 (52.0.0.1 - 52.1.255.254)
52.2.0.0/15 (52.2.0.1 - 52.3.255.254)
52.20.0.0/14 (52.20.0.1 - 52.23.255.254)
52.4.0.0/14 (52.4.0.1 - 52.7.255.254)
52.70.0.0/15 (52.70.0.1 - 52.71.255.254)
52.72.0.0/15 (52.72.0.1 - 52.73.255.254)
52.86.0.0/15 (52.86.0.1 - 52.87.255.254)
52.90.0.0/15 (52.90.0.1 - 52.91.255.254)
52.95.245.0/24 (52.95.245.1 - 52.95.245.254)
52.95.255.80/28 (52.95.255.81 - 52.95.255.94)
54.144.0.0/14 (54.144.0.1 - 54.147.255.254)
54.152.0.0/16 (54.152.0.1 - 54.152.255.254)
54.156.0.0/14 (54.156.0.1 - 54.159.255.254)
54.160.0.0/13 (54.160.0.1 - 54.167.255.254)
54.172.0.0/15 (54.172.0.1 - 54.173.255.254)
54.174.0.0/15 (54.174.0.1 - 54.175.255.254)
54.196.0.0/15 (54.196.0.1 - 54.197.255.254)
54.198.0.0/16 (54.198.0.1 - 54.198.255.254)
54.204.0.0/15 (54.204.0.1 - 54.205.255.254)
54.208.0.0/15 (54.208.0.1 - 54.209.255.254)
54.210.0.0/15 (54.210.0.1 - 54.211.255.254)
54.221.0.0/16 (54.221.0.1 - 54.221.255.254)
54.224.0.0/15 (54.224.0.1 - 54.225.255.254)
54.226.0.0/15 (54.226.0.1 - 54.227.255.254)
54.234.0.0/15 (54.234.0.1 - 54.235.255.254)
54.236.0.0/15 (54.236.0.1 - 54.237.255.254)
54.242.0.0/15 (54.242.0.1 - 54.243.255.254)
54.80.0.0/13 (54.80.0.1 - 54.87.255.254)
54.88.0.0/14 (54.88.0.1 - 54.91.255.254)
54.92.128.0/17 (54.92.128.1 - 54.92.255.254)
67.202.0.0/18 (67.202.0.1 - 67.202.63.254)
72.44.32.0/19 (72.44.32.1 - 72.44.63.254)
75.101.128.0/17 (75.101.128.1 - 75.101.255.254)
107.20.0.0/14 (107.20.0.1 - 107.23.255.254)
174.129.0.0/16 (174.129.0.1 - 174.129.255.254)
184.72.128.0/17 (184.72.128.1 - 184.72.255.254)
184.72.64.0/18 (184.72.64.1 - 184.72.127.254)
184.73.0.0/16 (184.73.0.1 - 184.73.255.254)
204.236.192.0/18 (204.236.192.1 - 204.236.255.254)
216.182.224.0/20 (216.182.224.1 - 216.182.239.254)
To regenerate this list, you can run this small shell script located here.
You can also subscribe to AWS Public IP Address Changes via Amazon SNS.
Contributing
- Fork it
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create new Pull Request
Documentation
¶
Overview ¶
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package util contains the utility code used in Neptune.io agent.
Package config is responsible for parsing the agent config file, command-line params and coming up with the final config.
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package state contains logic to maintain an event store on Agent to remember the recently processed events. This helps in avoiding the processing of duplicate events, just in case an agent receives duplicate events (which shouldn't happen ideally).
Package executor is responsible for executing runbooks on agent machine. This will have logic to fetch Github runbooks if the agent is configured to use Github.
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package metadata contains all the meta data used by the agent in registration protocol ¶
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package worker is responsible for communicating with AWS SQS and handing over the events to executor for runbook execution if the message passes all the checks.
Package api is responsible for communication between agent and Neptune.io service. This includes the data structures and logic related to agent registration, heartbeating, uploading runbook execution results, uploading agent errors, etc.
Package security is responsible for verifying the integrity of received SQS messages before agent processes them. Currently, Neptune.io signs all the messages with a private key and Agent verifies the message signature with public key before processing the message.
Index ¶
- Constants
- Variables
- func Beat(configObj *NeptuneConfig, agentId string) error
- func DeleteMessage(regInfo *RegistrationInfo, receiptHandle *string) error
- func ExecuteAction(event *Event, regInfo *RegistrationInfo, ...) error
- func GetConfig(configFilePath string, cmdlineConfig NeptuneConfig, errorChannel chan error) (NeptuneConfig, AgentConfig, error)
- func HasProcessedEvent(eventId string) bool
- func InitializeEventsFile(dir string)
- func KillCommand(cmd *exec.Cmd)
- func PersistEvent(event *Event) error
- func ReportError(err string)
- func RunLoop(regInfo *RegistrationInfo, regInfoUpdatesCh <-chan string, ...)
- func SendActionOutput(configObj *NeptuneConfig, request *ActionOutputMessage) error
- func SetPGroup(cmd *exec.Cmd)
- func SetRegistrationInfo(reg *RegistrationInfo, metaData HostMetaData, nConfig NeptuneConfig)
- func UpdateStatus(newStatus Status)
- func UploadLogs(configObj *NeptuneConfig, filename string, agentId string) error
- func VerifyMessage(message, signature string) (bool, error)
- type ActionOutputMessage
- type AgentConfig
- type AgentError
- type ConcurrentMap
- func (m ConcurrentMap) Count() int
- func (m ConcurrentMap) Get(key string) (int64, bool)
- func (m ConcurrentMap) GetShard(key string) *ConcurrentMapShared
- func (m *ConcurrentMap) Has(key string) bool
- func (m *ConcurrentMap) IsEmpty() bool
- func (m ConcurrentMap) Iter() <-chan Tuple
- func (m ConcurrentMap) IterBuffered() <-chan Tuple
- func (m ConcurrentMap) MarshalJSON() ([]byte, error)
- func (m *ConcurrentMap) Remove(key string)
- func (m *ConcurrentMap) Set(key string, value int64)
- func (m *ConcurrentMap) UnmarshalJSON(b []byte) (err error)
- type ConcurrentMapShared
- type Config
- type Event
- type Heartbeat
- type HostMetaData
- type NeptuneConfig
- type RegistrationInfo
- type RegistrationRequest
- type Response
- type Status
- type Tuple
- type UploadLogsRequest
Constants ¶
const ( DefaultBaseURL = "www.neptune.io" DefaultConfigFileName = "neptune-agent.json" )
const ( // Agent's current version. // Note: We should improve the versioning logic to avoid hard-coding like this. AgentVersion = "1.1.4" )
Variables ¶
var ErrorsChannel = make(chan string, 10)
Channel to hold agent errors. All components of agents should push errors into this channel and a separate thread uploads them to Neptune.io service currently. In future, we can even log these to syslog so that customers will catch agent issues sooner.
var SHARD_COUNT = 32
Functions ¶
func Beat ¶
func Beat(configObj *NeptuneConfig, agentId string) error
Function to send a heartbeat to Neptune.io service.
func DeleteMessage ¶
func DeleteMessage(regInfo *RegistrationInfo, receiptHandle *string) error
Function to delete SQS message after the processing is done.
func ExecuteAction ¶
func ExecuteAction(event *Event, regInfo *RegistrationInfo, actionOutputs chan<- *ActionOutputMessage, githubKey string) error
Main function to execute runbook based on the given event.
This function does following checks before executing the runbook.
- Using persistent event store, it verifies that the newly received event is not a duplicate.
- Based on the timestamp on event, it checks if the event is not too old.
- If the agent is configured to execute only Github runbooks, it double checks that the event contains Github runbook link and agent configuration has the Github access key.
The event will be discarded and SQS message will be deleted if any of the above checks fail.
func GetConfig ¶
func GetConfig(configFilePath string, cmdlineConfig NeptuneConfig, errorChannel chan error) (NeptuneConfig, AgentConfig, error)
Function to return config objects based on the config file, command-line values, etc.
func HasProcessedEvent ¶
Function to check if the given event id was already processed by this agent or not.
func InitializeEventsFile ¶
func InitializeEventsFile(dir string)
func KillCommand ¶
func PersistEvent ¶
func ReportError ¶
func ReportError(err string)
Function to report an error happened on this agent. This function only pushes the error into a channel.
func RunLoop ¶
func RunLoop(regInfo *RegistrationInfo, regInfoUpdatesCh <-chan string, eventsChannel chan<- *Event, regChannel chan<- time.Time)
Main worker function which does the following things in an infinite loop.
- Poll for SQS messages using long polling technique.
- Check if the messages received are for this agent, by checking agent id. Release the messages not meant for this agent.
- Verify the signature of the message and delete the message immediately if signature isn't correct.
- Deserialize the event from SQS message.
- Re-verify the agent id (which is inside the payload) again just to double check that agent id attribute was not tampered. This guards against replaying old messages, etc.
- At this point, agent has decided to process the event. So, hide the SQS message for the action timeout and hand over the event to executor for runbook execution.
func SendActionOutput ¶
func SendActionOutput(configObj *NeptuneConfig, request *ActionOutputMessage) error
Function to upload runbook execution results to Neptune.io service.
func SetRegistrationInfo ¶
func SetRegistrationInfo(reg *RegistrationInfo, metaData HostMetaData, nConfig NeptuneConfig)
func UpdateStatus ¶
func UpdateStatus(newStatus Status)
Function to update the agent status with the given new status.
func UploadLogs ¶
func UploadLogs(configObj *NeptuneConfig, filename string, agentId string) error
Function to upload agent logs to Neptune.io service.
func VerifyMessage ¶
Function to verify the signature of given message and check if the received signature is same as computed one.
Types ¶
type ActionOutputMessage ¶
type ActionOutputMessage struct {
RuleName string `json:"ruleName"`
RuleId string `json:"ruleId"`
AgentId string `json:"agentId"`
EventId string `json:"eventId"`
Status string `json:"status"`
ActionOutput string `json:"actionOutput"`
FailureReason string `json:"failureReason"`
StatusCode int `json:"statusCode"`
InflightActionId string `json:"inflightActionId"`
IsTimeout bool `json:"isTimeout"`
HostName string `json:"hostName"`
ActionType string `json:"actionType"`
}
Message sent by Agent to Neptune.io service to report runbook execution results.
type AgentConfig ¶
type AgentConfig struct {
AssignedHostname string
LogFile string
DebugMode bool
GithubApiKey string
}
Agent (machine info) section of the config file.
type AgentError ¶
type AgentError struct {
ErrorMessage string
AgentId string
FullLogs bool
Hostname string
Status string
}
Data structure to hold an error that has happened on Agent. All agent errors are sent to Neptune.io service for quick identification of agent problems.
type ConcurrentMap ¶
type ConcurrentMap []*ConcurrentMapShared
A "thread" safe map of type string:int64. To avoid lock bottlenecks this map is dived to several (SHARD_COUNT) map shards.
func (ConcurrentMap) Count ¶
func (m ConcurrentMap) Count() int
Returns the number of elements within the map.
func (ConcurrentMap) Get ¶
func (m ConcurrentMap) Get(key string) (int64, bool)
Retrieves an element from map under given key.
func (ConcurrentMap) GetShard ¶
func (m ConcurrentMap) GetShard(key string) *ConcurrentMapShared
Returns shard under given key
func (*ConcurrentMap) Has ¶
func (m *ConcurrentMap) Has(key string) bool
Looks up an item under specified key
func (ConcurrentMap) Iter ¶
func (m ConcurrentMap) Iter() <-chan Tuple
Returns an iterator which could be used in a for range loop.
func (ConcurrentMap) IterBuffered ¶
func (m ConcurrentMap) IterBuffered() <-chan Tuple
Returns a buffered iterator which could be used in a for range loop.
func (ConcurrentMap) MarshalJSON ¶
func (m ConcurrentMap) MarshalJSON() ([]byte, error)
Reviles ConcurrentMap "private" variables to json marshal.
func (*ConcurrentMap) Remove ¶
func (m *ConcurrentMap) Remove(key string)
Removes an element from the map.
func (*ConcurrentMap) Set ¶
func (m *ConcurrentMap) Set(key string, value int64)
Sets the given value under the specified key.
func (*ConcurrentMap) UnmarshalJSON ¶
func (m *ConcurrentMap) UnmarshalJSON(b []byte) (err error)
type ConcurrentMapShared ¶
type ConcurrentMapShared struct {
// contains filtered or unexported fields
}
type Config ¶
type Config struct {
Neptune NeptuneConfig
Agent AgentConfig
}
Combine config object for agent. This contains sections for Neptune.io, agent machine data, etc.
type Event ¶
type Event struct {
Timestamp int64 `json:"timestamp"`
Source string `json:"source"`
Hostname string `json:"hostname"`
ActionType string `json:"actionType"`
EventId string `json:"eventId"`
AgentId string `json:"agentId"`
RuleId string `json:"ruleId"`
RuleName string `json:"ruleName"`
InflightActionId string `json:"inflightActionId"`
RunbookName string `json:"runbookName"`
RawCommand string `json:"rawCommand"`
Signature string `json:"signature"`
Timeout int32 `json:"timeout"`
GithubFilePath string `json:"githubFilePath"`
Environment map[string]string `json:"env"`
SQSMessageId string
ReceiptHandle string
}
Event is a type holding the data sent from Neptune.io as a single SQS message. Each event corresponds to one execute runbook request to agent.
type Heartbeat ¶
type Heartbeat struct {
Status string
}
Message sent by Agent to Neptune.io service as a heartbeat.
type HostMetaData ¶
type HostMetaData struct {
HostName string
AssignedHostname string
ProviderId string
ProviderType string
Platform string
PrivateIpAddress string
PrivateDnsName string
PublicIpAddress string
PublicDnsName string
Region string
}
Host meta data used by the agent in registration protocol.
func GetHostMetaData ¶
func GetHostMetaData(agentConfig *AgentConfig) (HostMetaData, error)
Function to get the complete meta data for the host on which agent is running. This method tries to get cloud specific meta data also, in case the machine is in a cloud.
type NeptuneConfig ¶
Neptune.io section of config file.
type RegistrationInfo ¶
type RegistrationInfo struct {
AgentId string
CreateTime int64
UpdateTime int64
ActionQueueEndpoint string
AWSAccessKey string
AWSSecretAccessKey string
AWSSecurityToken string
}
Message received from Neptune.io service when agent's registration has succeeded.
func RegisterAgent ¶
func RegisterAgent(data HostMetaData, configObj *NeptuneConfig) (*RegistrationInfo, error)
Function to register this agent with Neptune.io service.
type RegistrationRequest ¶
type RegistrationRequest struct {
AgentVersion string
Hostname string
AssignedHostname string
ProviderServerId string
ProviderServerType string
Platform string
PrivateIpAddress string
PrivateDnsName string
PublicIpAddress string
PublicDnsName string
Region string
StartTime int64
}
Message sent by Agent to Neptune.io service to register itself.
type Response ¶
type Response struct {
// contains filtered or unexported fields
}
Response sent by Neptune.io service when agent sends different requests.
type Status ¶
type Status int
Agent status type definition.
func CurrentStatus ¶
func CurrentStatus() Status
Function to get current status of the running agent.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd is responsible for bootstrapping the agent, starting GO routines for different tasks in Neptune.io agent.
|
Package cmd is responsible for bootstrapping the agent, starting GO routines for different tasks in Neptune.io agent. |
|
neptuneagent
Package cmd/main is the starting point for Neptune.io agent.
|
Package cmd/main is the starting point for Neptune.io agent. |
|
windows
Package cmd/main is the starting point for Neptune.io's windows agent, which runs as a service.
|
Package cmd/main is the starting point for Neptune.io's windows agent, which runs as a service. |
|
Package logging is the logging package used by Neptune.io agent.
|
Package logging is the logging package used by Neptune.io agent. |