influxdb-athena-crawler

command module

v0.0.0-...-fe70aa2 Latest Latest Go to latest Published: Apr 2, 2024 License: Apache-2.0 Imports: 14 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/quortex/influxdb-athena-crawler

Links

Open Source Insights

README ¶

influxdb-athena-crawler

An AWS Athena crawler for InfluxDB.

Overview

This project is a utility designed to get AWS Athena results (CSV objects stored in AWS S3), parse them and write InfluxDB points.

Prerequisites

AWS

To be used with AWS and interact with the s3 bucket, an AWS account with the following permissions on s3 is required (note that s3:DeleteObject is only required if clean-objects is set):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "<BUCKET_NAME>"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListObjects", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "<BUCKET_NAME>/*"
    }
  ]
}

Installation

Helm (Kubernetes install)

Follow influxdb-athena-crawler documentation for Helm deployment here.

Configuration

Optional args

influxdb-athena-crawler takes as argument the parameters below.

Key	Description	Default
region	The AWS region.	`""`
bucket	The AWS bucket to watch.	`""`
prefix	The bucket prefix.	`""`
suffix	Filename suffix to restrict files processed on the bucket.	`""`
clean-objects	Whether to delete S3 objects after processing them.	`false`
timeout	The global timeout.	`"30s"`
influx-server	The InfluxDB server address.	`""`
influx-token	The InfluxDB token.	`""`
influx-org	The InfluxDB org to write to.	`""`
influx-bucket	The InfluxDB bucket write to.	`""`
measurement	A measurement acts as a container for tags, fields, and timestamps. Use a measurement name that describes your data.	`""`
timestamp-row	The timestamp row in CSV.	`"timestamp"`
timestamp-layout	The layout to parse timestamp.	`"2006-01-02T15:04:05.000Z"`
tag	Tags to add to InfluxDB point. Could be of the form `--tag=foo` if tag name matches CSV row or `--tag='foo={row:bar}'` to specify row.	`""`
field	Fields to add to InfluxDB point. Could be of the form `--field='foo={type:int,row:bar}'`, if not specified, CSV row matches field name. Type can be float, int, string or bool.	`""`