rid

package module
v1.1.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2024 License: MIT Imports: 7 Imported by: 0

README

godocTestCoverage

rid

Package rid provides a performant, goroutine-safe generator of short k-sortable unique IDs suitable for use where inter-process ID generation coordination is not required.

Using a non-standard character set (fewer vowels), IDs Base-32 encode as a 16-character URL-friendly, case-insensitive, representation like dfp7qt0v2pwt0v2x.

An ID is a:

  • 4-byte timestamp value representing seconds since the Unix epoch, plus a
  • 6-byte random value; see the Random Source discussion.

Built-in (de)serialization simplifies interacting with SQL databases and JSON. cmd/rid provides the rid utility to generate or inspect IDs. Thanks to fastrand[1], ID generation starts fast and remains so as cores are added. De-serialization has also been optimized. See Package Benchmarks.

Why rid as opposed to alternatives?

  • At 10 bytes binary, 16 bytes Base32 encoded, rid.IDs are case-insensitive and short, yet with 48 bits of uniqueness per second, are unique enough for many use cases.
  • IDs have a random component rather than potentially guessable monotonic counter.

Acknowledgement: This package borrows heavily from rs/xid (https://github.com/rs/xid), a zero-configuration globally-unique high-performance ID generator which itself levers ideas from MongoDB (https://docs.mongodb.com/manual/reference/method/ObjectId/).

Example:

id := rid.New()
fmt.Printf("%s\n", id.String())
// Output: dfp7qt97menfv8ll

id2, err := rid.FromString("dfp7qt97menfv8ll")
if err != nil {
	fmt.Println(err)
}
fmt.Printf("%s %d %v\n", id2.Time(), id2.Random(), id2.Bytes())
// Output: 2022-12-28 09:24:57 -0800 PST 43582827111027 [99 172 123 233 39 163 106 237 162 115]

CLI

Package rid also provides the rid tool for id generation and inspection.

$ rid 
dfpb18y8dg90hc74

$ rid -c 2
dfp9l9cgs05blztq
dfp9l9d80yxdf804

# produce 4 and inspect
$ rid `rid -c 4`
dfp9lmz9ksw87w48 ts:1672255955 rnd:256798116540552 2022-12-28 11:32:35 -0800 PST ID{ 0x63, 0xac, 0x99, 0xd3, 0xe9, 0x8e, 0x78, 0x83, 0xf0, 0x88 }
dfp9lmxefym2ht2f ts:1672255955 rnd:190729433933902 2022-12-28 11:32:35 -0800 PST ID{ 0x63, 0xac, 0x99, 0xd3, 0xad, 0x77, 0xa8, 0x28, 0x68, 0x4e }
dfp9lmt5zjy7km9n ts:1672255955 rnd: 76951796109621 2022-12-28 11:32:35 -0800 PST ID{ 0x63, 0xac, 0x99, 0xd3, 0x45, 0xfc, 0xbc, 0x78, 0xd1, 0x35 }
dfp9lmxt5sms80m7 ts:1672255955 rnd:204708502569607 2022-12-28 11:32:35 -0800 PST ID{ 0x63, 0xac, 0x99, 0xd3, 0xba, 0x2e, 0x69, 0x94,  0x2, 0x87 }

Random Source

Since cryptographically-secure IDs are not an objective for this package, other approaches could be considered. rid uses a Go internal runtime fastrand64 [1] which provides single and multi-core performance benefits.

You may enjoy reading Fast thread-safe randomness in Go.

[1] For more information on fastrand (wyrand) see: https://github.com/wangyi-fudan/wyhash and Go's sources for runtime/stubs.go.

To satisfy whether rid.IDs are unique enough for your use case, run eval/uniqcheck/main.go with various values for number of go routines and iterations, or, at the command line, produce 10,000,000 IDs and use OS utilities to check:

rid -c 10000000 | sort | uniq -d
// None output

Change Log

  • 2023-01-23 Replaced stdlib Base32 with unrolled version for decoding performance.
  • 2022-12-28 The "10byte" branch was merged to master; the "15byte-historical" branch will be left dormant. No major changes are now expected to this package; updates will focus on rounding out test coverage, addressing bugs, and clean up.

Contributing

Contributions are welcome.

Package Comparisons

Comparison table generated by eval/compare/main.go:

Package BLen ELen K-Sort Encoded ID and Next Method Components
mwyvr/rid 10 16 true dgckjfbkyxqkkbkj
dgckjfbe6mgr0r08		 fastrand 4 byte ts(sec) : 6 byte random
rs/xid 12 20 true cfbhieddt9dd5fv0uia0
cfbhieddt9dd5fv0uiag		 counter 4 byte ts(sec) : 2 byte mach ID : 2 byte pid : 3 byte monotonic counter
segmentio/ksuid 20 27 true 2L1aXRiLHHV9OmGbxn05RRbpn0N
2L1aXSnkoM8ojqpqPEEiWVI1FL6		 random 4 byte ts(sec) : 16 byte random
google/uuid 16 36 false 6cbe7d39-748f-4052-bb3d-898226c0486c
c6c24a2c-8005-4424-8960-489d6e386f2b		 crypt/rand v4: 16 bytes random with version & variant embedded
oklog/ulid 16 26 true 01GR03N2KVPZJYV4SZRG7QNVC4
01GR03N2KVN5XJFC0HBQSW6FXV		 crypt/rand 6 byte ts(ms) : 10 byte counter random init per ts(ms)
kjk/betterguid 17 20 true -NN-DcdvqJysHwukl1JR
-NN-DcdvqJysHwukl1JS		 counter 8 byte ts(ms) : 9 byte counter random init per ts(ms)

With only 48 bits of randomness per second, rid makes no attempt to weigh in as a globally-unique contender. rs/xid is a feature comparable solid alternative for such needs.

For a comparison of various Go-based unique ID solutions, see: https://blog.kowalczyk.info/article/JyRZ/generating-good-unique-ids-in-go.html

Package Benchmarks

A benchmark suite for the above noted packages can be found in eval/bench/bench_test.go. All runs were done with scaling_governor set to performance:

echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
AMD 8-core desktop, wired network connection
$ go test -cpu 1,2,4,16,32 -benchmem -bench .
goos: linux
goarch: amd64
pkg: github.com/mwyvr/rid/eval/bench
cpu: Intel(R) Core(TM) i9-14900K
BenchmarkRid              	47201202	        25.48 ns/op	       0 B/op	       0 allocs/op
BenchmarkRid-2            	78091510	        15.68 ns/op	       0 B/op	       0 allocs/op
BenchmarkRid-4            	77706714	        16.21 ns/op	       0 B/op	       0 allocs/op
BenchmarkRid-16           	69059413	        17.31 ns/op	       0 B/op	       0 allocs/op
BenchmarkRid-32           	76592799	        13.62 ns/op	       0 B/op	       0 allocs/op
BenchmarkXid              	41666000	        27.56 ns/op	       0 B/op	       0 allocs/op
BenchmarkXid-2            	39971817	        27.33 ns/op	       0 B/op	       0 allocs/op
BenchmarkXid-4            	37697936	        31.55 ns/op	       0 B/op	       0 allocs/op
BenchmarkXid-16           	36870097	        37.83 ns/op	       0 B/op	       0 allocs/op
BenchmarkXid-32           	47349218	        21.31 ns/op	       0 B/op	       0 allocs/op
BenchmarkKsuid            	 4847752	       240.4 ns/op	       0 B/op	       0 allocs/op
BenchmarkKsuid-2          	 4465123	       269.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkKsuid-4          	 4368691	       274.4 ns/op	       0 B/op	       0 allocs/op
BenchmarkKsuid-16         	 4267578	       279.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkKsuid-32         	 4209820	       279.6 ns/op	       0 B/op	       0 allocs/op
BenchmarkGoogleUuid       	 5501108	       212.9 ns/op	      16 B/op	       1 allocs/op
BenchmarkGoogleUuid-2     	 8988285	       133.3 ns/op	      16 B/op	       1 allocs/op
BenchmarkGoogleUuid-4     	16990833	        70.54 ns/op	      16 B/op	       1 allocs/op
BenchmarkGoogleUuid-16    	28899004	        39.21 ns/op	      16 B/op	       1 allocs/op
BenchmarkGoogleUuid-32    	39186010	        31.08 ns/op	      16 B/op	       1 allocs/op
BenchmarkUlid             	  202795	      5711 ns/op	    5440 B/op	       3 allocs/op
BenchmarkUlid-2           	  363986	      3064 ns/op	    5440 B/op	       3 allocs/op
BenchmarkUlid-4           	  674776	      1714 ns/op	    5440 B/op	       3 allocs/op
BenchmarkUlid-16          	  931116	      1122 ns/op	    5440 B/op	       3 allocs/op
BenchmarkUlid-32          	  916348	      1203 ns/op	    5440 B/op	       3 allocs/op
BenchmarkBetterguid       	24594817	        46.88 ns/op	      24 B/op	       1 allocs/op
BenchmarkBetterguid-2     	22549742	        51.16 ns/op	      24 B/op	       1 allocs/op
BenchmarkBetterguid-4     	18231304	        64.38 ns/op	      24 B/op	       1 allocs/op
BenchmarkBetterguid-16    	10490190	       107.5 ns/op	      24 B/op	       1 allocs/op
BenchmarkBetterguid-32    	 8507245	       139.2 ns/op	      24 B/op	       1 allocs/op

Documentation

Overview

Package rid provides a performant, k-sortable, scalable unique ID generator suitable for applications where ID generation coordination between machines or other processes is not required. ID generation is goroutine safe and scales well with CPU cores. Providing unique non-sequential keys for embeddable databases like SQLIte or BoltDB or key-value stores are typical use-cases.

Binary IDs Base-32 encode as a 16-character URL and human-friendly representation like dfp7qt0v2pwt0v2x.

The 10-byte binary representation of an ID is comprised of:

  • 4-byte timestamp value representing seconds since the Unix epoch
  • 6-byte random value; as of release v1.1.6 this package uses math/rand/v2 introduced with Go 1.22

Key features:

  • K-orderable in both binary and string representations
  • Encoded IDs are short (16 characters)
  • Automatic (de)serialization for SQL and JSON
  • Scalable performance as cores increase; ID generation is fast and remains so
  • URL and human friendly Base32 encoding using a custom character set to avoid unintended rude words if humans are to be exposed to IDs

Example usage: 

id := rid.New()
fmt.Printf("%s", id.String())
// Output: dfp7qt97menfv8ll
id, err := id.FromString("dfp7qt97menfv8ll")
// ID{0x63,0xac,0x7b,0xe9,0x27,0xa3,0x6a,0xed,0xa2,0x73}, nil

Acknowledgement: This source file is based on work in package github.com/rs/xid, a zero-configuration globally-unique ID generator. See LICENSE.rs-xid. The same API has been maintained.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (

	// ErrInvalidID represents errors returned when converting from invalid
	// []byte, string or json representations
	ErrInvalidID = errors.New("rid: invalid id")
)

Functions

func Sort 

func Sort(ids []ID)

Sort sorts an array of IDs in place.

Types

type ID 

type ID [rawLen]byte

ID represents a unique identifier

func FromBytes 

func FromBytes(b []byte) (ID, error)

FromBytes copies []bytes into an ID value. For validity, only a length-check is possible and performed.

func FromString 

func FromString(str string) (ID, error)

FromString decodes a Base32-encoded string to return an ID.

Example 
// dfp7emzzzzy30ey2 ts:1672246995 rnd:281474912761794 2022-12-28 09:03:15 -0800 PST ID{0x63,0xac,0x76,0xd3,0xff,0xff,0xfc,0x30,0x37,0xc2}
id, err := FromString("dfp7emzzzzy30ey2")
if err != nil {
	panic(err)
}
fmt.Println(id.Timestamp(), id.Random())

Output:

1672246995 281474912761794

func New 

func New() ID

New returns a new ID using the current time.

Example

examples 

id := New()
fmt.Printf(`ID:
    String()  %s
    Timestamp() %d
    Random()  %d 
    Time()    %v
    Bytes()   %3v\n`, id.String(), id.Timestamp(), id.Random(), id.Time().UTC(), id.Bytes())

Output:

func NewWithTime 

func NewWithTime(t time.Time) ID

NewWithTime returns a new ID using the supplied time.

The time value component of an ID is a Unix timestamp with seconds resolution; Go timestamp values reflect UTC and are not location aware.

Example 
id := NewWithTime(time.Date(2020, time.January, 1, 0, 0, 0, 0, time.UTC))
fmt.Printf(`ID: Timestamp() %d Time() %v`, id.Timestamp(), id.Time().UTC())

Output:

ID: Timestamp() 1577836800 Time() 2020-01-01 00:00:00 +0000 UTC

func NilID 

func NilID() ID

NilID returns a zero value for `rid.ID`.

func (ID) Bytes 

func (id ID) Bytes() []byte

Bytes returns the binary representation of ID.

func (ID) Compare 

func (id ID) Compare(other ID) int

Compare makes IDs k-sortable, returning an integer comparing only the first 4 bytes of two IDs.

Recall that an ID is comprized of a:

- 4-byte timestamp - 6-byte random value

Otherwise, it behaves just like `bytes.Compare(b1[:], b2[:])`.

The result will be 0 if two IDs are identical, -1 if current id is less than the other one, and 1 if current id is greater than the other.

func (ID) Encode 

func (id ID) Encode(dst []byte) []byte

Encode id, writing 16 bytes to dst and returning it.

func (ID) IsNil 

func (id ID) IsNil() bool

IsNil returns true if ID == nilID.

func (ID) IsZero 

func (id ID) IsZero() bool

IsZero is an alias of is IsNil.

func (ID) MarshalJSON 

func (id ID) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface. https://golang.org/pkg/encoding/json/#Marshaler

func (ID) MarshalText 

func (id ID) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler. https://golang.org/pkg/encoding/#TextMarshaler

func (ID) Random 

func (id ID) Random() uint64

Random returns the random component of the ID.

func (*ID) Scan 

func (id *ID) Scan(value interface{}) (err error)

Scan implements the sql.Scanner interface. https://golang.org/pkg/database/sql/#Scanner

func (ID) String 

func (id ID) String() string

String returns id as Base32 encoded string using a Crockford character set.

func (ID) Time 

func (id ID) Time() time.Time

Time returns the ID's timestamp as a Time value.

func (ID) Timestamp 

func (id ID) Timestamp() int64

Timestamp returns the ID's timestamp component as seconds since the Unix epoch.

func (*ID) UnmarshalJSON 

func (id *ID) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface. https://golang.org/pkg/encoding/json/#Unmarshaler

func (*ID) UnmarshalText 

func (id *ID) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler https://golang.org/pkg/encoding/#TextUnmarshaler All decoding is called from here.

func (ID) Value 

func (id ID) Value() (driver.Value, error)

Value implements package sql's driver.Valuer. https://golang.org/pkg/database/sql/driver/#Valuer

Source Files

View all Source files

Directories

Path Synopsis
cmd
rid
A utility to generate or inspect IDs.
 A utility to generate or inspect IDs.
eval
compare
Package main produces for documentation a markdown formatted table illustrating key differences between a number of unique ID packages.
 Package main produces for documentation a markdown formatted table illustrating key differences between a number of unique ID packages.
uniqcheck
Package main seeks to determine if the approach used delivers sufficiently unique IDs in go applications potentially running multiple goroutines.
 Package main seeks to determine if the approach used delivers sufficiently unique IDs in go applications potentially running multiple goroutines.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.