cronner

cronner

cronner is a command line utility that wraps periodic (cron) jobs for statistics gathering and success monitoring. Metrics like the amount of time the command took to run, as well as the command's return code, are emitted as vanilla statsd metrics to port 8125. It also implements file-level locking for a very simple and dumb job semaphore. This utility doesn't concern itself with the scheduling of jobs, and instead expects that it will be invoked by something like cron, at, or manually by an operator.

The utility also supports emitting DogStatsD Events on:

• job start and job finish
• job finish if the job failed
• if the job is taking too long to finish running

While there are no reports of issues having come up, if your statsd agent isn't DogStatsD-compliant its behavior may be undefined if you try to emit events or attach tags to metrics.

For the finish DogStatsD event, the return code and output of the command are provided in the event body. If the output is too long, it is truncated. This output can optionally be saved to disk only if the job fails for later inspection.

Go 1.9+ Compatibility Notice

Version 1.0.0+ of cronner requires that the source be built against Go 1.9+. Go 1.9 released the transparent support for a monotonic time source, within the time.Time type. This change means that using time.Now() to keep track of how long something took is safe when leap seconds occur. This was not the case before Go 1.9. If you attempt to build against an older Go runtime you should experience the following error:

./cronner.go:146: undefined: cronnerRequiresAtleastGoVersion19

Project History

cronner was originally developed as an internal application at PagerDuty and was subsequently open-sourced. The original repository can be found here: PagerDuty/cronner. After that project became unmaintained, this fork was created to continue the development and support of cronner.

The PagerDuty repository was officially deprecated in-favor of this one, with the final step being that this repository was detached from the original so it no longer appears to be a fork.

License

cronner is released under the BSD 3-Clause License. See the LICENSE file for the full contents of the license.

Usage

Help Output

Usage:
  cronner [OPTIONS] -- command [arguments]...

Application Options:
  -d, --lock-dir=                  the directory where lock files will be placed (default: /var/lock)
  -e, --event                      emit a start and end datadog event
  -E, --event-fail                 only emit an event on failure
  -F, --log-fail                   when a command fails, log its full output (stdout/stderr) to the log directory using the UUID as the filename
  -g, --group=<group>              emit a cronner_group:<group> tag with statsd metrics
  -G, --event-group=<group>        emit a cronner_group:<group> tag with Datadog events, does not get sent with statsd metrics
  -H, --statsd-host=<host>         destination host to send datadog metrics
  -k, --lock                       lock based on label so that multiple commands with the same label can not run concurrently
  -l, --label=                     name for cron job to be used in statsd emissions and DogStatsd events. alphanumeric only; cronner will lowercase it
      --log-path=                  where to place the log files for command output (path for -F/--log-fail output) (default: /var/log/cronner)
  -L, --log-level=                 set the level at which to log at [none|error|info|debug] (default: error)
  -N, --namespace=                 namespace for statsd emissions, value is prepended to metric name by statsd client (default: cronner)
  -p, --passthru                   passthru stdout/stderr to controlling tty
  -P, --use-parent                 if cronner invocation is runner under cronner, emit the parental values as tags
  -s, --sensitive                  specify whether command output may contain sensitive details, this only avoids it being printed to stderr
  -t, --tag=                       additional tags to add to datadog events and metrics (can be used multiple times), either <key>:<value> or <string> format
  -V, --version                    print the version string and exit
  -w, --warn-after=N               emit a warning event every N seconds if the job hasn't finished, set to 0 to disable (default: 0)
  -W, --wait-secs=                 how long to wait for the file lock for (default: 0)

Help Options:
  -h, --help                       Show this help message

Running A Command

The label (-l, --label) flag is required.

To run the command /bin/sleep 10 and emit the stats as cronner.sleeptyime.time and cronner.sleepytime.exit_code you would run:

$ cronner -l sleepytime -- /bin/sleep 10

Note that -- in the command line arguments tells cronner to stop parsing CLI flags. It then grabs the rest of the arguments as the command to execute.

Environment Variables

The cronner process sets a few environment variables for subprocesses to consume if they wish. The CRONNER_PARENT_UUID environment variable is the canonical way for determining whether or not we are running under cronner.

If you invoke the cronner command with the -P/--use-parent flag it will look for these variables and tag the events and metrics emissions with their values. It lowercases the variable name before emitting the tag, so CRONNER_PARENT_GROUP becomes cronner_parent_group.

DogStatsd Emissions

If you were to have a UDP listener on port 8125 on localhost, the statsd emissions would look something like this:

cronner.sleepytime.time:10005.834649|ms
cronner.sleepytime.exit_code:0|g

It emits a timing metric for how long it took for the command to run, as well as the command's exit code.

Running A Command with a DogStatsD Event

If you want to run /bin/sleep 5 as sleepytime2 and emit a DogStatsD for when the job starts and finishes:

$ cronner -e -l sleepytime2 -- /bin/sleep 5

The UDP datagrams emitted would then look like this:

_e{35,12}:Cron sleepytime2 starting on rinzler|job starting|k:ab31f2f6-498e-468a-b572-ab990065e8d3|s:cronner|t:info
cronner.sleepytime2.time:5005.649979|ms
cronner.sleepytime2.exit_code:0|g
_e{55,22}:Cron sleepytime2 succeeded in 5.00565 seconds on rinzler|exit code: 0\\noutput:(none)|k:ab31f2f6-498e-468a-b572-ab990065e8d3|s:cronner|t:success

Chef Cookbook

To make cronner easier to install and use, there is a cronner Chef cookbook available for use. Not only does it allow you to install cronner, but it makes it easy to replace your cron_d resources with ones that will wrap the jobs with cronner.

Contributors

• Tim Heckman
• Thomas Dziedzic
• Alex Eftimie
• Anthony O'Brien
• Alex Hill
• Andre Cloutier
• R.T. Lechow

Development

• set up your workspace as per the instructions for standard Go development

• clone the cronner repository

  git clone git@github.com:theckman/cronner.git
  

• make your changes to the codebase, including adding relevant test cases

• run your tests to ensure all pass

  go test -v ./... -check.vv
  

• confirm that building cronner works

  go build