Documentation ¶
Overview ¶
tarball_divide takes a .tar.gz file of images and outputs one or more .tar.gz of images, where each contains images in a certain base directory. For example, the input tarball might contain three image files:
another_dir/dog/b.png Yet_another_dir/dog/c.png some_dir/0/a.png
The output includes two tarballs ¶
- 0.tar.gz includes one file
- some_dir/0/a.png
- dog.tar.gz inlcudes
- another_dir/dog/b.png
- Yet_another_dir/dog/c.png
tarball_merge can then merge the two tarballs into one, with an interleaved order of images:
another_dir/dog/b.png some_dir/0/a.png Yet_another_dir/dog/c.png
We need this pair of tools because the training of image classification models often needs to read images from a container file, the tarball, and we want each minibatch of successive images belong to different lables. In the convention, the base directory name is the label.
You can download the MNIST PNG dataset from https://github.com/myleott/mnist_png as /tmp/mnist_png.tar.gz and divide it using the following commands:
go install ./... tarball_divide -out=/tmp /tmp/mnist_png.tar.gz
The above command generates /tmp/[0-9].tar.gz, each of which contains only regular image files, no longer the directories.