s3s

s3s

s3s is a go binary instead of vast-engineering/s3select.

Features

s3s query all files lower than S3 prefix.

Available below:

• Input JSON to Output JSON
• Input CSV to Output JSON
• Input Application Load Balancer Logs to Output JSON
• Input CloudFront Logs to Output JSON

Usage

$ s3s --help
NAME:
   s3s - Easy S3 select like searching in directories

USAGE:
   s3s [global options] command [command options] [arguments...]

VERSION:
   current

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug        erorr check for developer (default: false)
   --help, -h     show help
   --version, -v  print the version

   AWS:

   --max-retries value, -M value   max number of api requests to retry (default: 20)
   --region value                  region of target s3 bucket exist (default: ENV["AWS_REGION"])
   --thread-count value, -t value  max number of api requests to concurrently (default: 150)

   Input Format:

   --alb-logs, --alb_logs  (default: false)
   --cf-logs, --cf_logs    (default: false)
   --csv                   (default: false)

   Query:

   --count, -c              max number of results from each key to return (default: false)
   --limit value, -l value  max number of results from each key to return (default: 0)
   --query value, -q value  a query for S3 Select
   --where value, -w value  WHERE part of the query

   Run:

   --delve               like directory move before querying (default: false)
   --dry-run, --dry_run  pre request for s3 select (default: false)

   Target:

   --duration value  from current time if alb or cf (ex: "2h3m") (default: 0s)
   --since value     end at if alb or cf (ex: "2006-01-02 15:04:05")
   --until value     start at if alb or cf (ex: "2006-01-02 15:04:05")

s3s is execution S3 Select from json to json (default).

$ s3s s3://bucket/prefix
{"time":1654848930,"type":"speak"}
{"time":1654848969,"type":"sleep"}

// $ s3s s3://bucket/prefix_A s3://bucket/prefix_B s3://bucket/prefix_C

$ s3s -q 'SELECT * FROM S3Object s WHERE s.type = "speak"' s3://bucket/prefix
{"time":1654848930,"type":"speak"}

// alternate
// $ s3s -w 's.type = "speak"' s3://bucket/prefix

s3s can execute S3 Select from csv to json when --csv option enabled.

// 122, hello
$ s3s s3://bucket/prefix
{"_1":122,"_2":"hello"}

ALB and CF logs support

--alb-logs is a format for Application Load Balancer (ALB). --cf-logs is a format for CloudFront (CF).

Each options are tagging available instead of _1, _2, etc.

• Application Load Balancer Format
• CloudFront Format

And also, --where replace column names to column numbers. But --query does not replace columns for execution raw query.

// below query is same as $ s3s --alb-logs --query="'SELECT * FROM S3Object s WHERE s.`_2` = '2022-09-01T00:00:00.000000Z'" s3://prefix
$ s3s --alb-logs --where="s.`time` = '2022-09-01T00:00:00.000000Z'" s3://prefix

Support log range when alb and cf. time format is 2006-01-02 15:04:05 as UTC.

• --duration is a duration from now.
• --since is start time
• --until is end time

However, s3s stop when you target cloudfront and using --duration or --since only, because s3s hit too many keys.

-delve, like directory move before querying

search from prefix

$ s3s -delve s3://bucket/prefix

search from bucket list

$ s3s -delve

  bucket/prefix/C/
  bucket/prefix/B/
  bucket/prefix/A/        # delve more lower path than this prefix
  Query↵ (s3://bucket/prefix/) # choose and execute s3select this prefix
> ←Back upper path        # back to parent prefix
5/5
>

Querying after Enter.

{"time":1654848930,"type":"speak"}
{"time":1654848969,"type":"sleep"}

...

bucket/prefix/A/ (print path to stderr at end)