README
¶
redact
Utilities to redact Go strings for confidentiality
Some explanations about how this is used in e.g. CockroachDB:
https://wiki.crdb.io/wiki/spaces/CRDB/pages/1824817806/Log+and+error+redactability
Documentation
¶
Overview ¶
Package redact provides facilities for separating “safe” and “unsafe” pieces of data when logging and constructing error object.
An item is said to be “safe” if it is proven to not contain PII or otherwise confidential information that should not escape the boundaries of the current system, for example via telemetry or crash reporting. Conversely, data is considered “unsafe” until/unless it is known to be “safe”.
Example use:
redactable := redact.Sprintf("hello %s", "universe")
// At this point, 'redactable' contains "hello ‹universe›".
// This prints "hello universe":
fmt.Println(redactable.StripMarkers())
// This reports "hello ‹×›":
fmt.Println(redactable.Redact())
When defining your own custom types, you can define a SafeFormat method, implementing the redact.SafeFormatter interface in a way you'd otherwise implement fmt.Formatter. This is then recognized by this package's API automatically.
Alternatively:
- you can implement the SafeValue interface, which tells the redact package to always the default formatting of a type as safe and thus not included inside redaction markers.
- you can include a value within redact.Safe() and redact.Unsafe() in redact.Sprintf / redact.Fprintf calls, to force the omission or inclusion of redaction markers.
Example (Format) ¶
testCases := []struct {
format string
args []interface{}
}{
{"%d", []interface{}{123}},
{"%v", []interface{}{[]int{123, 456}}},
{"%v", []interface{}{[]RedactableString{"safe", "‹unsafe›"}}},
{"%v", []interface{}{[]safe{"safe", "safe2"}}},
{"%v", []interface{}{[]safestringer{{"safe"}, {"safe2"}}}},
{"%v", []interface{}{makeMixedInts()}},
{"%+v", []interface{}{makeMixedInts()}},
{"%v", []interface{}{[]mixedInts{makeMixedInts(), makeMixedInts()}}},
{"%+v", []interface{}{makeMixedSafe()}},
{"%+v", []interface{}{makeMixedSafeStringer()}},
{"%+v", []interface{}{makeMixedRedactableString()}},
{"%+v", []interface{}{makeMixedRedactableBytes()}},
}
type formatterFn func(format string, args ...interface{}) string
formatters := []struct {
name string
fn formatterFn
}{
{"fmt.sprint", fmt.Sprintf},
{"redact.sprint", func(format string, args ...interface{}) string { return string(Sprintf(format, args...)) }},
{"redact.printer", func(format string, args ...interface{}) string {
fn := func(w p) { w.Printf(format, args...) }
return string(Sprint(compose{fn: fn}))
}},
}
for _, tc := range testCases {
base := fmt.Sprintf("%q :: %T :: %+v", tc.format, tc.args[0], tc.args)
base = ptrRe.ReplaceAllString(base, "℘")
fmt.Println(base)
for _, f := range formatters {
result := f.fn(tc.format, tc.args...)
// Erase pointers.
result = ptrRe.ReplaceAllString(result, "℘")
fmt.Printf("%s:\t%s\n", f.name, result)
}
fmt.Println()
}
Output: "%d" :: int :: [123] fmt.sprint: 123 redact.sprint: ‹123› redact.printer: ‹123› "%v" :: []int :: [[123 456]] fmt.sprint: [123 456] redact.sprint: [‹123› ‹456›] redact.printer: [‹123› ‹456›] "%v" :: []markers.RedactableString :: [[safe ‹unsafe›]] fmt.sprint: [safe ‹unsafe›] redact.sprint: [safe ‹unsafe›] redact.printer: [safe ‹unsafe›] "%v" :: []redact.safe :: [[safe safe2]] fmt.sprint: [safe safe2] redact.sprint: [safe safe2] redact.printer: [safe safe2] "%v" :: []redact.safestringer :: [[{s:safe} {s:safe2}]] fmt.sprint: [{safe} {safe2}] redact.sprint: [{‹safe›} {‹safe2›}] redact.printer: [{‹safe›} {‹safe2›}] "%v" :: redact.mixedInts :: [{a:123 ap:℘ A:123 Ap:℘ Apn:<nil>}] fmt.sprint: {123 ℘ 123 ℘ <nil>} redact.sprint: {‹123› ‹℘› ‹123› ‹℘› ‹<nil>›} redact.printer: {‹123› ‹℘› ‹123› ‹℘› ‹<nil>›} "%+v" :: redact.mixedInts :: [{a:123 ap:℘ A:123 Ap:℘ Apn:<nil>}] fmt.sprint: {a:123 ap:℘ A:123 Ap:℘ Apn:<nil>} redact.sprint: {a:‹123› ap:‹℘› A:‹123› Ap:‹℘› Apn:‹<nil>›} redact.printer: {a:‹123› ap:‹℘› A:‹123› Ap:‹℘› Apn:‹<nil>›} "%v" :: []redact.mixedInts :: [[{a:123 ap:℘ A:123 Ap:℘ Apn:<nil>} {a:123 ap:℘ A:123 Ap:℘ Apn:<nil>}]] fmt.sprint: [{123 ℘ 123 ℘ <nil>} {123 ℘ 123 ℘ <nil>}] redact.sprint: [{‹123› ‹℘› ‹123› ‹℘› ‹<nil>›} {‹123› ‹℘› ‹123› ‹℘› ‹<nil>›}] redact.printer: [{‹123› ‹℘› ‹123› ‹℘› ‹<nil>›} {‹123› ‹℘› ‹123› ‹℘› ‹<nil>›}] "%+v" :: redact.mixedSafe :: [{s1:safe S1:safe s1p:℘ S1p:℘ S1pn:<nil>}] fmt.sprint: {s1:safe S1:safe s1p:℘ S1p:℘ S1pn:<nil>} redact.sprint: {s1:‹safe› S1:safe s1p:‹℘› S1p:℘ S1pn:<nil>} redact.printer: {s1:‹safe› S1:safe s1p:‹℘› S1p:℘ S1pn:<nil>} "%+v" :: redact.mixedSafeStringer :: [{s2:{s:safe} s2p:℘ S2:{s:safe} S2p:safe S2pn:<nil>}] fmt.sprint: {s2:{s:safe} s2p:℘ S2:{s:safe} S2p:safe S2pn:<nil>} redact.sprint: {s2:{s:‹safe›} s2p:‹℘› S2:{s:‹safe›} S2p:safe S2pn:<nil>} redact.printer: {s2:{s:‹safe›} s2p:‹℘› S2:{s:‹safe›} S2p:safe S2pn:<nil>} "%+v" :: redact.mixedRedactableString :: [{r:safe‹unsafe› rp:℘ R:safe‹unsafe› Rp:℘ Rpn:<nil>}] fmt.sprint: {r:safe‹unsafe› rp:℘ R:safe‹unsafe› Rp:℘ Rpn:<nil>} redact.sprint: {r:safe‹unsafe› rp:‹℘› R:safe‹unsafe› Rp:safe‹unsafe› Rpn:<nil>} redact.printer: {r:safe‹unsafe› rp:‹℘› R:safe‹unsafe› Rp:safe‹unsafe› Rpn:<nil>} "%+v" :: redact.mixedRedactableBytes :: [{r:[115 97 102 101 226 128 185 117 110 115 97 102 101 226 128 186] rp:℘ R:[115 97 102 101 226 128 185 117 110 115 97 102 101 226 128 186] Rp:℘ Rpn:<nil>}] fmt.sprint: {r:[115 97 102 101 226 128 185 117 110 115 97 102 101 226 128 186] rp:℘ R:[115 97 102 101 226 128 185 117 110 115 97 102 101 226 128 186] Rp:℘ Rpn:<nil>} redact.sprint: {r:safe‹unsafe› rp:‹℘› R:safe‹unsafe› Rp:safe‹unsafe› Rpn:<nil>} redact.printer: {r:safe‹unsafe› rp:‹℘› R:safe‹unsafe› Rp:safe‹unsafe› Rpn:<nil>}
Index ¶
- func EndMarker() []byte
- func EscapeMarkers(s []byte) []byte
- func Fprint(w io.Writer, args ...interface{}) (n int, err error)
- func Fprintf(w io.Writer, format string, args ...interface{}) (n int, err error)
- func JoinTo(w SafeWriter, delim RedactableString, values interface{})
- func MakeFormat(s fmt.State, verb rune) (justV bool, format string)
- func RedactedMarker() []byte
- func RegisterRedactErrorFn(fn func(err error, p i.SafePrinter, verb rune))
- func RegisterSafeType(t reflect.Type)
- func SortStrings(s []RedactableString)
- func StartMarker() []byte
- func StringWithoutMarkers(f SafeFormatter) string
- func Unsafe(a interface{}) interface{}
- type ManualBuffer
- type RedactableBytes
- type RedactableString
- func HelperForErrorf(format string, args ...interface{}) (RedactableString, error)
- func Join(delim RedactableString, s []RedactableString) RedactableString
- func Sprint(args ...interface{}) RedactableString
- func Sprintf(format string, args ...interface{}) RedactableString
- func Sprintfn(printer func(w SafePrinter)) RedactableString
- type SafeFloat
- type SafeFormatter
- type SafeInt
- type SafeMessager
- type SafePrinter
- type SafeRune
- type SafeString
- type SafeUint
- type SafeValue
- type SafeWriter
- type StringBuilder
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EscapeMarkers ¶
EscapeMarkers escapes the special delimiters from the provided byte slice.
func JoinTo ¶ added in v1.0.9
func JoinTo(w SafeWriter, delim RedactableString, values interface{})
JoinTo writes the given slice of values delimited by the provided delimiter to the given SafeWriter.
func MakeFormat ¶
MakeFormat is a helper for use by implementations of the SafeFormatter interface. It reproduces the format currently active in fmt.State and verb. This is provided because Go's standard fmt.State does not make the original format string available to us.
If the return value justV is true, then the current state was found to be %v exactly; in that case the caller can avoid a full-blown Printf call and use just Print instead to take a shortcut.
func RedactedMarker ¶
func RedactedMarker() []byte
RedactedMarker returns the special string used by Redact.
func RegisterRedactErrorFn ¶
func RegisterRedactErrorFn(fn func(err error, p i.SafePrinter, verb rune))
RegisterRedactErrorFn registers an error redaction function for use during automatic redaction by this package. Provided e.g. by cockroachdb/errors.
func RegisterSafeType ¶
RegisterSafeType registers a data type to always be considered safe during the production of redactable strings.
func SortStrings ¶ added in v1.0.9
func SortStrings(s []RedactableString)
SortStrings sorts the provided slice of redactable strings.
func StartMarker ¶
func StartMarker() []byte
StartMarker returns the start delimiter for an unsafe string.
func StringWithoutMarkers ¶
func StringWithoutMarkers(f SafeFormatter) string
StringWithoutMarkers formats the provided SafeFormatter and strips the redaction markers from the result. This is provided for convenience to facilitate the implementation of String() methods alongside SafeFormat() to avoid code duplication.
Note: if this function is ever found to be a performance bottleneck, one can consider using an alternate implementation of Sprint() which similarly calls the SafeFormat() methods but does not introduce markers and instead writes to a string buffer directly.
Types ¶
type ManualBuffer ¶ added in v1.1.0
ManualBuffer is a variable-sized buffer of bytes with Write methods. Writes are escaped in different ways depending on the output mode. Note: This type is only meant for "advanced" usage. For most common uses, consider using StringBuilder instead.
type RedactableBytes ¶
type RedactableBytes = m.RedactableBytes
RedactableBytes is like RedactableString but is a byte slice.
Instances of RedactableBytes should not be constructed directly; instead use the facilities from print.go (Sprint, Sprintf) or the methods below.
func EscapeBytes ¶
func EscapeBytes(s []byte) RedactableBytes
EscapeBytes escapes markers inside the given byte slice and encloses the entire byte slice between redaction markers. EscapeBytes escapes markers inside the given byte slice and encloses the entire byte slice between redaction markers.
type RedactableString ¶
type RedactableString = m.RedactableString
RedactableString is a string that contains a mix of safe and unsafe bits of data, but where it is known that unsafe bits are enclosed by redaction markers ‹ and ›, and occurrences of the markers inside the original data items have been escaped.
Instances of RedactableString should not be constructed directly; instead use the facilities from print.go (Sprint, Sprintf) or the methods below.
func HelperForErrorf ¶ added in v1.0.4
func HelperForErrorf(format string, args ...interface{}) (RedactableString, error)
HelperForErrorf is a helper to implement a redaction-aware fmt.Errorf-compatible function in a different package. It formats the string according to the given format and arguments in the same way as Sprintf, but in addition to this if the format contains %w and an error object in the proper argument position it also returns that error object.
Note: This function only works if an error redaction function has been injected with RegisterRedactErrorFn().
func Join ¶ added in v1.0.9
func Join(delim RedactableString, s []RedactableString) RedactableString
Join creates a redactable string from the given slice of redactable strings, adjoined with the provided delimiter.
func Sprint ¶
func Sprint(args ...interface{}) RedactableString
Sprint prints out the arguments and encloses unsafe bits between redaction markers. If either safe and unsafe bits of data contain the markers in their representation already, they are escaped first. If a RedactableString or RedactableBytes argument is passed, it is reproduced as-is without escaping.
func Sprintf ¶
func Sprintf(format string, args ...interface{}) RedactableString
Sprintf formats the arguments and encloses unsafe bits between redaction markers. If either safe and unsafe bits of data contain the markers in their representation already, they are escaped first. The format is always considered safe and the caller is responsible to ensure that the markers are not present in the format string.
func Sprintfn ¶
func Sprintfn(printer func(w SafePrinter)) RedactableString
Sprintfn produces a RedactableString using the provided SafeFormat-alike function.
type SafeFloat ¶ added in v1.1.0
SafeFloat represents a floating-point value that is not a sensitive value.
type SafeFormatter ¶
type SafeFormatter = i.SafeFormatter
SafeFormatter is implemented by object types that want to separate safe and non-safe information when printed out by a Printf-like formatter.
type SafeMessager ¶
type SafeMessager = i.SafeMessager
SafeMessager is an alternative to SafeFormatter used in previous versions of CockroachDB. NB: this interface is obsolete. Use SafeFormatter instead. TODO(knz): Remove this.
type SafePrinter ¶
type SafePrinter = i.SafePrinter
SafePrinter is a stateful helper that abstracts an output stream in the context of printf-like formatting, but with the ability to separate safe and unsafe bits of data.
This package provides one implementation of this using marker delimiters for unsafe data, see markers.go. We would like to aim for alternate implementations to generate more structured formats.
type SafeString ¶
type SafeString = i.SafeString
SafeString represents a string that is not a sensitive value.
type SafeValue ¶
SafeValue is a marker interface to be implemented by types that alias base Go types and whose natural representation via Printf is always safe for reporting.
This is recognized by the SafePrinter interface as an alternative to SafeFormatter.
It is provided to decorate "leaf" Go types, such as aliases to int.
Typically, a linter enforces that a type can only implement this interface if it aliases a base go type. More complex types should implement SafeFormatter instead.
It is advised to build an automatic process during builds to collect all the types that implement this interface, as well as all uses of this type, and produce a report. Changes to this report should receive maximal amount of scrutiny during code reviews.
func Safe ¶
func Safe(a interface{}) SafeValue
Safe turns any value into an object that is considered as safe by the formatter.
This is provided as an “escape hatch” for cases where the other interfaces and conventions fail. Increased usage of this mechanism should be taken as a signal that a new abstraction is missing. The implementation is also slow.
type SafeWriter ¶
type SafeWriter = i.SafeWriter
SafeWriter provides helper functions for use in implementations of SafeFormatter, to format mixes of safe and unsafe strings.
type StringBuilder ¶ added in v1.0.3
type StringBuilder = builder.StringBuilder
StringBuilder accumulates strings with optional redaction markers.
It implements io.Writer but marks direct writes as redactable. To distinguish safe and unsafe bits, it also implements the SafeWriter interface.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
redact
Package redact is a helper internal package that provides the Unsafe and Safe value wrappers.
|
Package redact is a helper internal package that provides the Unsafe and Safe value wrappers. |
|
rfmt/fmtsort
Package fmtsort provides a general stable ordering mechanism for maps, on behalf of the fmt and text/template packages.
|
Package fmtsort provides a general stable ordering mechanism for maps, on behalf of the fmt and text/template packages. |