refid

refid

About

A refid (short for Reference Identifier) is a unique identifier, similar to UUIDv7, with a few difference.

There are two types of refids: TimePrefixed and RandomPrefixed.

TimePrefixed (type:0x00)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           unix_ts_ms                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           unix_ts_ms          |    rand_a   |t|      tag      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             rand_b                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             rand_b                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
unix_ts_ms:
    48 bits big-endian unsigned number of Unix epoch timestamp milliseconds.
    (2284 years worth... until about year 4200 or so)
rand_a:
    7 bits random pad. fill with crypto/rand random.
t:
    1 bit for type. TimePrefixed (type:0)
tag:
    8 bits tag. 255 separate tags (0 being untagged).
rand_b:
    64 bits random pad. fill with crypto/rand random.

RandomPrefixed (type:0x01)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             rand_a                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    rand_a                   |t|      tag      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             rand_b                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             rand_b                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
rand_a:
    55 bits random pad. fill with crypto/rand random.
t:
    1 bit for type. RandomPrefixed (type:1)
tag:
    8 bits tag. 255 separate tags (0 being untagged).
rand_b:
    64 bits random pad. fill with crypto/rand random.

Features

General:

• tagging (support for 255 distinct tags)
• supports go/sql scanner/valuer
• multiple encodings supported: native (base32), base64, base16 (hex)
• similar to UUIDv7, with different tradeoffs

TimePrefix:

• unix timestamp with millisecond precision
• sortable, db index friendly
• Compared to UUIDv7
  • tagging support
  • 48 bits of Unix timestamp milliseconds from epoch (similar to UUIDv7)
  • slightly smaller random section (71 vs 74 bits), though still good collision resistance
  • not a standard

RandomPrefix:

• not sortable, not db index friendly
• Compared to UUIDv7
  • tagging support
  • slightly smaller random section (119 vs 122 bits), though still good collision resistance
  • not a standard

Non-Features

• refids, like UUIDs, do not internally perform any signature verification. If the validity of the encoded timestamp and tag are required for any secure operations, the refid SHOULD be externally verified before parsing/decoding.
  An example of this could be a wrapping encoder/decoder doing hmac signing and verification.

Inspirations

• https://github.com/gofrs/uuid
  gofrs' UUIDv7 might be a better option if you need something standard.
• https://github.com/uuid6/uuid6-ietf-draft, https://github.com/ietf-wg-uuidrev/rfc4122bis
  UUIDv6/v7 drafts

Installation

go get -u github.com/dropwhile/refid

Usage

Simple

// generate a TimePrefixed refid
rID, err := refid.New()
// generate a TimePrefixed refid (or panic)
rID = refid.Must(refid.New())
// generate a RandomPrefixed refid (or panic)
rID = refid.Must(refid.NewRandom())

// encoding...
// encode to native encoding (base32 with Crockford alphabet)
s := rID.String() // "0r326xw2xbpga5tya7px89m7hw"
// encode to base64 encoding
s = rID.ToBase64String() // "BgYjd4Lq7QUXXlHt1CaHjw"
// encode to hex encoding
s = rID.ToHexString() // "0606237782eaed05175e51edd426878f"
// raw bytes
b := rID.Bytes()

// decoding...
// decode from native
rID, err := refid.Parse(s)
// decode from base64
rID, err = refid.FromBase64String(s)
// decode from hex
rID, err = refid.FromHexString(s)

// get the time out of a TimePrefixed refid (as a time.Time)
var ts time.Time = rID.Time()

Tagging

Simple tagging usage:

myTag := 2

// generate a refid with tag set to 1
rID = refid.Must(refid.NewTagged(1))
// you can also set it manually after generation
rID.SetTag(myTag)
// check if it is tagged
rID.Tagged() // true
// check if it has a specific tag
rID.HasTag(1) // false
rID.HasTag(2) // true


s := rID.String()
// require desired tag or fail parsing
r, err := refid.ParseTagged(1, s) // err != nil here, as refid was tagged 2
r, err = refid.ParseTagged(2, s) // err == nil here, as refid was tagged 2

What use is tagging?

Tag support ensures that a refid of a certain tag type can be made distinct from other refids -- those of a different tag type, or those with no tag type.

A hypothetical example is a refid url paramater for a type named "author", can be enforced as invalid when someone attempts to supply it as input for a different refid url parameter for a type named "book".

Making tagging usage easier with refid.IDTagger:

// AuthorID ensures it will only succesfully generate and parse tag=2 refids
AuthorIDT := refid.IDTagger(2)
// BookID ensures it will only succesfully generate and parse tag=3 refids
BookIDT := refid.IDTagger(3)

authorID := refid.Must(AuthorIDT.New()) // generated with a tag of 2
authorID.HasTag(2) // true
bookID := refid.Must(BookIDT.New()) // generated with a tag of 3
bookID.HasTag(3) // true

r, err := AuthorIDT.Parse(authorID.String()) // succeeds; err == nil
r, err = bookIDT.Parse(authorID.String()) // fails; err != nil

Tagging with Custom Types

A more complicated tagging example using embedded types:

import (
    "github.com/dropwhile/refid"
    "github.com/dropwhile/refid/reftag"
)

type NoteID struct {
    // the reftag.IDt* types are generated to provide a handy means to
    // wrap with your own struct type, to provide type constraints in function
    // parameters and struct members
    reftag.IDt8
}

// some helpful shortcut function aliases
var (
    // generate a new time-prefixed refid of tag type 8
    NewNoteID       = reftag.New[NoteID]
    // generate a new random-prefixed refid of tag type 8
    NewRandomNoteID = reftag.NewRandom[NoteID]
    // handy matcher for sql mock libraries (gomock, pgxmock, etc)
    NoteIDMatcher   = reftag.NewMatcher[NoteID]()
    // some handy parsing aliases
    NoteIDFromBytes = reftag.FromBytes[NoteID]
    ParseNoteID     = reftag.Parse[NoteID]
)

func ParseNote(ctx context.Context, db dbHandle, noteStr NoteID) (*NoteID, error) {
    noteID, err := ParseNoteID(noteStr)
    // error will be non-nil if the tag in the RefID does not match the expectation (`8`)
    ...
    return NoteID, err
}

func DBLookupNote(ctx context.Context, db dbHandle, noteID NoteID) (*DBNote, error) {
    // noteID is now a compile time checked type ensuring that RefIDs of a different
    // tag are not accidentally allowed.
    ...
}

refidtool CLI utility

Installation:

go install github.com/dropwhile/refid/cmd/refidtool@latest

# generate a refi.ID with a tag of 5
% refidtool generate -t 5
native enc:   0r326xw2xbpga5tya7px89m7hw
hex enc:      0606237782eaed05175e51edd426878f
base64 enc:   BgYjd4Lq7QUXXlHt1CaHjw
tag value:    5
type:         TimePrefixed
time(string): 2023-09-24T23:47:38.954477Z
time(millis): 1695599258954477

# generate a refid with a tag of 5, and only output the native(base32) encoding
% refidtool generate -t 5 -o native
0r34ky6h51r012an8skhbsvxt0

# generate a refid with a tag of 5, and only output the hex encoding
% refidtool generate -t 5 -o hex
060649f82794f10039169e91d0696763

# generate a random-prefixed refid with a tag of 4, and only output the base64 encoding
% refidtool generate -r -t 4 -o base64
DxBK5ksxywRfCMUUTIw2mw

# genrate a refid with a tag of 2, at a specific timestamp
% refidtool generate -t 2 -w "2023-01-01T00:00:11.123456Z"
native enc:   0qrjh15pzc004nzrkbpcp2v0wm
hex enc:      05f12884b6fb000257f89aeccb0b60e5
base64 enc:   BfEohLb7AAJX-Jrsywtg5Q
tag value:    2
type:         TimePrefixed
time(string): 2023-01-01T00:00:11.123456Z
time(millis): 1672531211123456

# decode a refid and display
% refidtool decode 0qrjh15pzc004nzrkbpcp2v0wm
native enc:   0qrjh15pzc004nzrkbpcp2v0wm
hex enc:      05f12884b6fb000257f89aeccb0b60e5
base64 enc:   BfEohLb7AAJX-Jrsywtg5Q
tag value:    2
type:         TimePrefixed
time(string): 2023-01-01T00:00:11.123456Z
time(millis): 1672531211123456

# here is what a random-prefixed refid looks like
# note the time is the zero value for time.Time
% refidtool parse DxBK5ksxywRfCMUUTIw2mw
native enc:   1w84nsjb675g8qr8rma4s31pkc
hex enc:      0f104ae64b31cb045f08c5144c8c369b
base64 enc:   DxBK5ksxywRfCMUUTIw2mw
tag value:    4
type:         RandomPrefixed
time(string): 1970-01-01T00:00:00Z
time(millis): 0