s3flatten

s3flatten

s3flatten copies a set of files in a nested directory on S3 to a flat directory. In the destination directory, /, which is used as the delimiter of the directory, is converted to - by default.

s3flatten --suffix=.txt s3://my-source/path1/ s3://my-destination/path2/

source path                      destination path
s3://my-source/path1/            s3://my-destination/path2/
    foo.txt            --COPY->         foo.txt
    bar.png
    abc/def.txt        --COPY->         abc-def.txt
    abc/def/ghi.txt    --COPY->         abc-def-ghi.txt

The source bucket and destination bucket can be the same. If the source bucket and destination bucket are the same, the destination key prefix must not be under the source key prefix.

Usage

Usage: s3flatten [-hv] [-c value] [-d value] [-s value] s3://src-bucket/path/to/src/ s3://dest-bucket/path/to/dest/
 -c, --concurrency=value
                Number of goroutine for COPY operation [128]
 -d, --delimiter=value
                Delimiter to replace '/' with to flatten path. [-]
 -h, --help     display help
 -s, --suffix=value
                Copy only objects which has this suffix in key
 -v, --verbose  verbose output