Sumo2S3 - Take Data from SumoLogic and put it into S3
export data from Sumo to S3
Build
go build
Run
# Create your cache folder
mkdir _cache_folder_
# edit your config with your favorite text editor
vi config
# create your initial statefile (we don't run without one)
echo "{}" > statefile
# run it
./sumo2s3
Configuration
{
"sumo_api_jobs": "https://api.sumologic.com/api/v1/search/jobs",
"open_jobs": 16,
"apiKeyConcurrency": 4,
"apiKeys": [
["123456789ABC", "ABC...123"]
],
"cachefolder": "./exports",
"start_date": "2021-10-09T00:00:00Z",
"end_date": "2021-12-09T00:00:10Z",
"increment":"60s",
"aws_region": "us-east-1",
"s3_bucket": "my-output-bucket",
"s3_tmp_prefix":"tmp",
"s3_prefix":"data",
"filters": [
{
"start_date": "1970-10-01T00:00:00Z",
"query": "*"
}
]
}
Todo/Missing Features
- No support for continuous mode. Sumo has a native feature for exporting data coming in to S3.
- internal state locking needs to be re-done