README
¶
Warning This package has been deprecated. We have found a faster, simpler and more reliable way. See issue #12.
GenRawID
genrawid is a niche command line tool and/or Go package to generate a unique consistent number from the input.
$ genrawid --string "abcdefgh"
-2474118025671277174
$ # 'abcdefgh' --> The data to store in SQLite3
$ # -2474118025671277174 --> The primary key/rawid of stored data
$ echo -n 'abcdefgh' > ./sample.txt
$ genrawid ./sample.txt
-2474118025671277174
$ # 'abcdefgh' --> The data to store in SQLite3
$ # -2474118025671277174 --> The primary key/rawid of stored data
genrawid is similar to a hash function, but its value is a combination of a hash and a checksum.
The hash value is the digest of the input and the checksum is the CRC of the complete hash value.
By default, BLAKE3-512 is used for the hash algorithm and CRC-32C (CRC-32 with Castagnoli polynomial) for the checksum.
8 Bytes = The first 4 Bytes of the hash + 4 Bytes of the checksum of the hash
- See benchmark of BLAKE3 and CRC-32 comparing to other hash algorithms:
Why?
The main objective is to use SQLite3 as a fast KVS (Key-Value-Store) for CAS (Content-Addressable-Storage) usage.
genrawid generates a 64-bit/8-byte signed decimal number that can be used as rawid in SQLite3.
Searching for a record with a specific rowid, or for all records with rowids within a specified range is around twice as fast as a similar search made by specifying any other PRIMARY KEY or indexed value.
(From "ROWIDs and the INTEGER PRIMARY KEY" @ sqlite.org)
So, in theory, if you know the rawid of the content, you can find it twice as fast.
On the other hand, this command/package is of little use when using SQLite3 as RDB (Relational Database) or when dealing with mutable content.
Install
Download the binary for your operating system and architecture and place it as an executable in your PATH.
- Latest Releases (Windows, macOS, Linux, RaspberryPi)
For Homebrew/Linuxbrew users:
brew install KEINOS/apps/genrawid
Statuses
- Unit Tests/Code Coverage
- Secrurity/Vulnerability Check
CONTRIBUTING
- GolangCI Lint rules: .golangci-lint.yml
- To run tests in a container:
docker-compose --file ./.github/docker-compose.yml run v1_17- This will run:
go test -cover -race ./...golangci-lint rungolint ./...
- Branch to PR:
main- It is recommended that DraftPR be done first to avoid duplication of work.
License
- MIT, Copyright (c) KEINOS and the GenRawID contributors.
Documentation
¶
Overview ¶
Package genrawid provides functions to generate rawid.
For the sample implementation see: ./cmd/genrawid/main.go
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var IsModeFast = false
IsModeFast is the flag to use fast mode. It will use the last 16bit of the hash as the xor16 checksum.
var OsStdin = os.Stdin
OsStdin is a copy of os.Stdin to ease testing. Mock this variable during tests.
Functions ¶
func FromFile ¶
FromFile returns the rawid generated from the input file.
Example ¶
const pathFile = "./testdata/msg.txt" // msg.txt ==> "abcdefgh"
rawid, err := genrawid.FromFile(pathFile)
if err != nil {
log.Fatal(err)
}
fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())
Output: ddaa2ac39b79058a -2474118025671277174
Example (Fast_mode) ¶
oldMode := genrawid.IsModeFast
defer func() {
genrawid.IsModeFast = oldMode
}()
// Set to fast mode
genrawid.IsModeFast = true
pathFile := "./testdata/msg.txt" // msg.txt ==> "abcdefgh"
rawid, err := genrawid.FromFile(pathFile)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())
Output: ddaa2ac30a98963b -2474118028101904837
func FromStdin ¶
FromStdin returns the rawid generated from stdin as its input.
Example ¶
ExampleFromStdin
Since we can not receive input from stdin during the example run, we mock the OsStdin, the copy of os.Stdin in the package, by assigning a dummy file as its input.
// Backup and defer recover the stdin
oldStdin := genrawid.OsStdin
defer func() {
genrawid.OsStdin = oldStdin
}()
// Mock the stdin by assigning a file to stdin.
pathFile := "./testdata/msg.txt" // msg.txt ==> "abcdefgh"
osFile, err := os.Open(pathFile)
if err != nil {
fmt.Println(err.Error())
return
}
genrawid.OsStdin = osFile // <-- mock!
// Here is the actual example usage of FromStdin method to get the input from
// stdin and generate a rawid.
rawid, err := genrawid.FromStdin()
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())
Output: ddaa2ac39b79058a -2474118025671277174
func FromString ¶
FromString returns the rawid generated from the input string.
Example ¶
input := "abcdefgh"
rawid, err := genrawid.FromString(input)
if err != nil {
log.Fatal(err)
}
fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())
Output: ddaa2ac39b79058a -2474118025671277174
Types ¶
This section is empty.
Source Files
Directories