cloudwatchmetricbeat

Cloudwatchmetricbeat

Welcome to Cloudwatchmetricbeat. Based on previous work on Cloudwatch Logs and Metrics

Installation

Download a binary, and put it in a good spot on your system. Or use the Dockerfile to build a Docker image.

To run this beat with a given configuration, run:

./cloudwatchmetricbeat -c cloudwatchmetricbeat.yml -e -d "*"

Credentials and permissions

The CloudWatch Exporter uses the AWS Go SDK, which offers a variety of ways to provide credentials. This includes the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

The cloudwatch:ListMetrics and cloudwatch:GetMetricStatistics IAM permissions are required.

Configuration

The configuration is in YAML, an example with common options:

---
aws_region: us-east-1
metrics:
 - aws_namespace: AWS/ELB
   aws_metric_name: RequestCount
   aws_dimensions: [AvailabilityZone, LoadBalancerName]
   aws_dimension_select:
     LoadBalancerName: [myLB]
   aws_statistics: [Sum]

CloudWatch has been observed to sometimes take minutes for reported values to converge. The default delay_seconds will result in data that is at least 5 minutes old being requested to mitigate this.

• Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution metrics and are available only for custom metrics that have been defined with a StorageResolution of 1.
• Data points with a period of 60 seconds (1-minute) are available for 15 days.
• Data points with a period of 300 seconds (5-minute) are available for 63 days.
• Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months).

Cost

Amazon charges for every API request, see the current charges.

Every metric retrieved requires one API request, which can include multiple statistics. In addition, when aws_dimensions is provided, the exporter needs to do API requests to determine what metrics to request. This should be negligible compared to the requests for the metrics themselves.

If you have 100 API requests every minute, with the price of USD$10 per million requests (as of Jan 2015), that is around $45 per month.

Getting Started with Cloudwatchmetricbeat

Requirements

• Golang >= 1.7
• Glide

Init Project

Clone this repository at the following location: ${GOPATH}/github.com/narmitech/cloudwatchmetricbeat.

To get running with Cloudwatchmetricbeat and also install the dependencies, run the following command:

make setup

It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.

To push Cloudwatchmetricbeat in the git repository, run the following commands:

git remote set-url origin https://github.com/narmitech/cloudwatchmetricbeat
git push origin master

For further development, check out the beat developer guide.

Build

To build the binary for Cloudwatchmetricbeat run the command below. This will generate a binary in the same directory with the name cloudwatchmetricbeat.

make

Run

To run Cloudwatchmetricbeat with debugging output enabled, run:

./cloudwatchmetricbeat -c cloudwatchmetricbeat.yml -e -d "*"

Test

To test Cloudwatchmetricbeat, run the following command:

make testsuite

alternatively:

make unit-tests
make system-tests
make integration-tests
make coverage-report

The test coverage is reported in the folder ./build/coverage/

Update

Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on etc/fields.yml. To generate etc/cloudwatchmetricbeat.template.json and etc/cloudwatchmetricbeat.asciidoc

make update

Cleanup

To clean Cloudwatchmetricbeat source code, run the following commands:

make fmt
make simplify

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Cloudwatchmetricbeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/github.com/narmitech/cloudwatchmetricbeat
cd ${GOPATH}/github.com/narmitech/cloudwatchmetricbeat
git clone https://github.com/narmitech/cloudwatchmetricbeat

For further development, check out the beat developer guide.

Packaging

The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:

make package

This will fetch and create all images required for the build process. The whole process to finish can take several minutes.