cumulus

cumulus

A library and command line tool for concurrent bulk investigation and occasional manipulation of multiple clouds.

TL;DR

Do you

• have multiple AWS accounts or multiple regions and need to list resources across them?
• get tired of the aws cli baroque command syntax?
• get annoyed by how slow and awkward the aws cli is?

This tool might be for you (at least eventually)

Futures: many more resource types, DigitalOcean, GCP, and Kubernetes

Development Status

Becoming Useful. Only handles AWS, only handles a few resource types, but handles many accounts/regions concurrently.

Installation

Pick one:

• MacOS: brew install deweysasser/tap/Cumulus
• Pre-built binaries from GitHub release page.
• Other platforms (deb, rpm) TBD as I get a curiosity bump or someone needs them.
• You can always go install github.com/deweysasser/cumulus@latest (currently the only thing you'll miss out on is the Makefile embedding version information in the binary.)

Running

Quick start

There's nothing to set up. The command will pick up all your profiles from ~/.aws/credentials, deduplicate them (i.e. you may have more than 1 pointing to an account -- it will take care of it), and use them.

If you use AWS creds in your environment -- file a bug and I'll make that work. It's not really important to me using multiple clouds.

If you have ~/.aws.credentials that you can use with the AWS CLI, try out cumulus instance list and go!

A quick tour

cumulus (account|instance|snapshot|machine-image|volume|dns zone|dns record) list -I <column to include> -X <column to exclude> -l filter_column=value

All include, exclude, filter columns and filter values are regular expressions.

For filters, using multiple filters on the command line (... -l one=two -l three=four) will select the UNION of objects matching the two filters, where using commas in a single filter expression (...-l one=two,three=four) will select the INTERSECTION of objects matching the individual filters.

Use --verbose or -v to show all available column and run statistics. Use --quiet or -q to suppress the output of the column header row.

Examples

List some instances

 $ time bin/cumulus instance list > /dev/null

real	0m5.585s
user	0m0.718s
sys	    0m0.141s

8:19PM INF discovered instance count=259
8:19PM INF timer average_latency=195.194586 count=47 name="AWS API calls" rate=8.566558814468866
8:19PM INF timer totals TotalCalls=47 TotalDuration=9174.145546 name="AWS API calls"
8:19PM INF run time duration=5484.798748

How many instances of each type are we using?

$ time ./bin/cumulus  instance list -I '^type$' -X . | sort | uniq -c | sort -rn 
  59 c5.2xlarge
   4 c5.xlarge
   3 c5.4xlarge
   1 c5.18xlarge
 ...

real	0m5.935s
user	0m0.738s
sys	0m0.150s

(Breakdown: List the instances, include the field matching regexp ^type$ in output, exclude all other fields. You can figure out the rest of the pipeline.)

Other possibilities

# Show the "environment" tag in a column
$ cumulus instance list -I tag:environment

# show only instances where the "environment" tag is "production"
$ cumulus instance list -l tag:environment=production

# show all instances where *any* column contains the word "test"
$ cumulus instance list -l .=test

# show all available fields for snapshots.  Warning:  there are a *LOT* of them.
$ cumulus snapshot list -v

# show snapshots tagged with "Release" containing "v1.0" (it's a regexp) *AND* "Status" containing "alpha"
$ cumulus snapshot list -l tag:Release=v1.0,tag:Status=alpha

It has similar performance for listing instances, zones...everything else (except, oddly, the API for S3 is pretty slow)

Command Overview

$ bin/cumulus  -h
Usage: cumulus <command>

Flags:
  -h, --help       Show context-sensitive help.
      --version    Show program version

Output
      --debug                   Show debugging information
      --output-format="auto"    How to show program output (auto|terminal|jsonl)
  -q, --quiet                   Be less verbose than usual
  -v, --verbose                 Be more verbose than usual

Profile
  --profile.cpu       profile the CPU
  --profile.memory    profile the Memory usage

Commands:
  account list

  instance list

  snapshot list

  machine-image list

  volume list

  dns zone list

  dns record list

More detailed Development Status

Becoming Useful. Only handles AWS, only handles a few resource types, but handles many accounts/regions concurrently.

Allows column selection and filtering.

Very fast

CLI might be awkward for now -- I haven't fully exercises all the options and made sure that everything behaves senibly all the time (e.g. when running the "--version" flag, you still have to give a subcommand)

Currently handles:

• AWS
  • listing
    • accounts
    • instances
    • snapshots
    • AMIs
    • EBS volumes
    • Route53 Zones
    • Route53 Recordsets

Plans and Intentions

• Codgen as much as possible
• Handle many more AWS types (all the ones I use regularly)
• Add kubernetes (discover clusters across accounts, then discover resources inside them)
• Add Digital Ocean
• Add GCP
• Add Azure? (I don't currently use Azure, but maybe by this time I'll find someone who does to add to it)

Why am I writing this?

The AWS command line is slow. S....l....o....w.

It's also...awkwardly laid out, in the format of "service - endpoint". GCP and Digital Ocean CLIs are much more intuitively designed in the form of "subject - object - verb - modifiers".

And, last, the AWS CLI works with a single account, and a single region, at a time. Even wehn I'm exclusively using AWS, this is not the right abstraction.

Oh, and very last...I have hit some non-linearity in the AWS command line, where large result sets are incredibly delayed. I'm not that patient.

So the goal of this tool is to produce a cloud library that's much better structured, faster, concurrent, and not tied strictly to AWS.

That last part is the most experimental. I have tried better structures for library and found them good. In this tool, I'm attempting a "common denominator" API approach to multiple clouds and...well...we're going to see how well that works.

Also, I'm using that library to create a better CLI.

Features and Bugs

Please use Github Issues for all bugs and feature requests.