crc32c

command module
v0.0.0-...-063e46f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 10, 2021 License: MIT Imports: 11 Imported by: 0

README

`crc32c` is a trivial program that wraps Go's SSE4.2 accelerated CRC-32C (Castagnoli) implementation.  I needed a fast checksum for amd64 that could be plonked on a number of platforms with minimal fuss.  The most obvious solutions available in FreeBSD base, and its package repository, were surprisingly slow.

C/++ enthusiasts would probably be better served by https://github.com/google/crc32c[google/crc32c].

`crc32c` does not implement a _check_ or _verify_ mode.  This behaviour may be emulated by plumbing output into a `diff` operand.

Checksum computation is parallelised over logical CPUs when `crc32c` is invoked with multiple positional path arguments and `--parallel` set to some value greater than one.  Output from parallelised execution is written in non-deterministic order.  Pipe output through `sort -k2` for determinisitc order.

(The machine used in the following perf comparisons was optimised for low power draw.  Turbo clocks were disabled.)

----
# dmesg | grep ^CPU | head -n 1
CPU: Intel(R) Xeon(R) E-2224G CPU @ 3.50GHz (3504.34-MHz K8-class CPU)
# uname -srm
FreeBSD 12.2-RELEASE-p6 amd64

# time crc32c blob-1g
7ef14d22  blob-1g
crc32c blob-1g  0.06s user 0.14s system 100% cpu 0.200 total
----

rhash is slow:

----
# time rhash --crc32c blob-1g
7ef14d22  blob-1g
rhash --crc32c blob-1g  0.62s user 0.15s system 99% cpu 0.768 total
----

openssl is slow (SHA-1):

----
# time openssl sha1 blob-1g > /dev/null
openssl sha1 blob-1g > /dev/null  1.16s user 0.11s system 99% cpu 1.267 total
----

cksfv is slow (CRC-32):

----
# time cksfv blob-1g > /dev/null
cksfv blob-1g > /dev/null  2.13s user 0.28s system 99% cpu 2.415 total
----

cksum is slow (CRC-32?):

----
# time cksum blob-1g > /dev/null
cksum blob-1g > /dev/null  2.54s user 0.18s system 99% cpu 2.724 total
----

Documentation

The Go Gopher

There is no documentation for this package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL