s3-kinesis-replay

s3-kinesis-replay

a cli for replaying historical kinesis messages via an s3 archive (e.g. s3 firehose delivery destination)

Installing

Download

Currently s3-kinesis-replay ships binaries for OSX and Linux 64bit systems. You can download the latest release from GitHub

$ wget https://github.com/cludden/s3-kinesis-replay/releases/download/{version}/{artifact}

Building From Source

$ make

Getting Started

View usage information:

$ s3-kinesis-replay --help
a cli for replaying historical kinesis messages via an s3 archive

Usage:
  s3-kinesis-replay [flags]

Flags:
      --bucket string                         s3 archive bucket name
      --delimiter string                      optional delimiter regexp
      --format string                         parser format
  -h, --help                                  help for s3-kinesis-replay
      --json-concurrency int                  json parser concurrency (default 4)
      --json-schema string                    json parser schema path
      --kinesis-backoff-interval string       kinesis backoff interval
      --kinesis-backoff-max-interval string   kinesis max backoff interval
      --kinesis-buffer-window string          kinesis buffer window size
      --kinesis-endpoint string               kinesis endpoint override
      --kinesis-region string                 kinesis region override
      --log-level string                      log verbosity level
      --partition-key string                  json parser parition key path
      --prefix string                         s3 archive prefix
      --replace string                        optional replace regexp
      --replace-with string                   optional replacement string
      --s3-concurrency int                    s3 download concurrency (default 4)
      --s3-region string                      s3 archive region
      --start-after string                    s3 archive start-after key
      --stop-at string                        s3 archive stop-at key
      --stream-name string                    target kinesis stream name

Basic usage with all required flags: (assumes aws environment is configured)

$ s3-kinesis-replay \
    --bucket my-bucket \
    --prefix 2018/01 \
    --stream-name my-stream \
    --format json \
    --partition-key path.to.partitionKey

Note: It is worthwile to use this application from within the same region as your s3 bucket & kinesis stream to avoid s3 egress charges

Configuration

s3-kinesis-replay uses viper for configuration, and accepts a configuration file, environment variables, cli flags, or a combination of all three.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes using conventional changelog standards (git commit -am 'feat: adds my new feature')
4. Push to the branch (git push origin my-new-feature)
5. Ensure linting/tests are all passing
6. Create new Pull Request

Testing

Prerequisities:

• Docker CE & Compose

Testing:

# run short tests
$ docker-compose run test make test-short

# run all tests
$ docker-compose run test make test-verbose

# run tests with coverage
$ docker-compose run test make test-coverage

Linting:

# run lint
$ docker-compose run test make lint

License

Licensed under the MIT License

Copyright (c) 2018 Chris Ludden