jpegextract

jpegextract

Overview

jpegextract is a command-line utility to extract the embedded JPEGs from a camera RAW file. It's a GO language command based on RawParser. There are existing tools that perform this or similar functionality; however, the reasons for creating this tool:

1. I have many RAW files that are processed using commercial software, yet on occassion, I would like the camera-produced JPEG for comparison.
2. To utilize the concurrency model provided by the GO language to process multiple files without any explicit "traditional" locking (e.g, mutexes)
3. Experiment with GO's "C" package and interfacing with existing C libraries.

Dependencies

• Default build (GO image/jpeg)
  • GO 1.2 (maybe older GO 1.1.2? but not tested)
  • convert utility Required for rotating JPEGs
• Optional
  • libjpeg
  • ImageMagick
• Highly recommended
  • TurboJpeg
  • If you have many JPEGs to extract, TurboJpeg provides noticebly better performance. See performance observations.

Why not only use GO's image/jpeg package?

I originally planned on making the utility pure GO. In fact for small batches of images, image/jpeg does the job. However, when you have a lot of files to process, C-based image libraries are hard to beat with respect to processing time--even more, using TurboJpeg with its optimized processor instructions. See performance to further understand why.

Usage

• Obtain the utility. The following will retrieve jpegextract and any dependencies and will build the default executable. The default uses GO's image/jpeg package.

go get github.com/jeremytorres/jpegextract

• jpegextract is command-line executable: $GOPATH/bin/jpegextract

• To utilize libjpeg or turbojpeg:

cd $GOPATH/src/github.com/jeremytorres/jpegextract

# Build using libjpeg
# See jpeglibjpeg.go for CFLAGS and LDFLAGS and update for your environment.  # The defaults for Linux and Mac OS (10.9) should work.
go build -tags jpeg

# Build using turbojpeg
# See jpegturbojpeg.go for CFLAGS and LDFLAGS and update for your environment.  # The defaults for Linux and Mac OS (10.9) should work.
go build -tags turbojpeg

• Invoke jpegextract
  • Get help. Displays information on parameters and defaults.

./jpegextract -h

* Execute utility.

./jpgextract --raws "CR2,NEF" --src-dirs "/path_to/raw_files/dir1,/path_to/raw_file/dir" --dest-dir "/path_to/extract_dir" --num-routines 8 --quality 75

• Verify output. Extracted JPEGs will contain the source RAW file name with an "_extracted.jpg" extension.

--quality parameter

The quality parameter is the JPEG compressors "quaility" value. This is a value from 0-100, where 0 is the lowest compression quality. This value has a dramatic impact on overall processing time and the size of the resulting extracted JPEG. A good rule of thumb: 80 or less for really good JPEGs. If storage size is of concern, us a smaller value.

--num-routines parameter

This defines the max number of files to process concurrently (setting GOMAXPROCS internally). In the above example, 8 files would be processed simultaneously. It's recommended to set this parameter to the maximum number of logical cores of the host machine; however, your mileage will vary! Disk IO performance will drive the performance gains of this parameter, so experiment with this parameter to tweak for your environemt.

Current RAW file format support

• Nikon NEF
  • Nikon D700
• Canon CR2
  • Canon 5D MarkII
  • Canon 1D MarkIII

Performance

As is always the case, performance will very. Below are my observations based upon the aforementioned dependencies.

Performance Tests

• Mac Pro (Early 2008)
  • OS: Mac OS X 10.9.1 "Mavericks"
  • Processor: 2 x 2.8GHz Quad-Core Intel Xeon
  • Memory: 16 GB 800 MHz DDR2 memory
  • Disk (OS): 128 GB SSD
  • Disk (Data): 3 TB software RAID comprised of 3 1TB 7200RPM hard drives

• TurboJpeg library installed via homebrew
• CFLAGS for wrapper code: -O2
• jpegextract Utility Config
  • TurboJpeg defined in jpeg.go
  • LDFLAGS=-L/path_to/jpeg-turbo/lib -lturbojpeg
  • CFLGAS=-I/path_to/jpeg-turbo/include _ Inputs to jpegextract Utility
  • 1745 Canon CR2 files: a mixture of 10 and 21 mega pixel files.
  • Default JPEG compression value of 80.
  • GOMAXPROCS specified: 8
• Results
  • Time to Complete: 58.67 seconds (average per file: .03 seconds)

• libjpeg used is OS X default for 10.9
  • /usr/lib/libjpeg
• CFLAGS for wrapper code: -O2
• jpegextract Utility Config
  • libjpeg defined in jpeg.go
  • LDFLAGS=-ljpeg _ Inputs to jpegextract Utility
  • 1745 Canon CR2 files: a mixture of 10 and 21 mega pixel files.
  • Default JPEG compression value of 80.
  • GOMAXPROCS specified: 8
• Results
  • Time to Complete: 136.28 seconds (average per file: .08 seconds)

• GO version 1.2
• jpegextract Utility Config _ Inputs to jpegextract Utility
  • 1745 Canon CR2 files: a mixture of 10 and 21 mega pixel files.
  • Default JPEG compression value of 80.
  • GOMAXPROCS specified: 8
• Results
  • Time to Complete: 470.29 seconds (average per file: .27 seconds)

Current Development Status

• I consider the current status a beta version as there is a laundry list of this I will like to support:
  • Add performance benchmarks
  • Add additional camera RAW file support
  • Create a "better" parser interface