scrubbing

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 29, 2023 License: MIT Imports: 14 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Disposition

type Disposition string

Disposition describes how to scrub a piece of data (in the abstract). It can be any of the following:

  • "erase": remove the data entirely from the output
  • "mask": scramble characters of the data
  • "generate(modelName)": create dummy replacement data using the given model

func (Disposition) Action

func (d Disposition) Action() string

func (Disposition) Parameter

func (d Disposition) Parameter() string

func (Disposition) String

func (d Disposition) String() string

type FieldNameRule

type FieldNameRule struct {
	// In is a field-name matching pattern to test whether this rule applied.
	In *regexp.Regexp
	// Out describes what to do when a value satisfies this rule.
	Out Disposition
}

FieldNameRule describes a scrubbing policy based on the name of a field. and irrespective of its value.

func (*FieldNameRule) MarshalJSON

func (r *FieldNameRule) MarshalJSON() ([]byte, error)

func (FieldNameRule) String

func (r FieldNameRule) String() string

func (*FieldNameRule) UnmarshalJSON

func (r *FieldNameRule) UnmarshalJSON(b []byte) error

type HeuristicRule

type HeuristicRule struct {
	// In is the name of a model that will be used to recognize values.
	In string

	P float64
	// Out describes what to do when a value satisfies this rule.
	Out Disposition
}

HeuristicRule describes a scrubbing policy based on a value irrespective of its field name.

func (HeuristicRule) String

func (r HeuristicRule) String() string

type Percentage

type Percentage float64

Percentage is a convenient way to print percentages with a single decimal place.

func (Percentage) MarshalYAML

func (p Percentage) MarshalYAML() (interface{}, error)

func (Percentage) String

func (p Percentage) String() string

type Policy

type Policy struct {
	// FieldName ensures that certain fields are always scrubbed based on their name.
	// Key: substring of a field name
	// Value: disposition when fields matching this substring are encountered
	FieldName []FieldNameRule `json:"fieldname"`
	// Heuristic applies selected models heuristically to all values to achieve
	// scrubbing based on the type, shape of pattern of the value itself.
	// Key: model name
	// Value: disposition when a value matches the model
	Heuristic []HeuristicRule `json:"heuristic"`
}

Policy reflects human decisionmaking about which values should be scrubbed based on their field name.

func DefaultPolicy

func DefaultPolicy() *Policy

DefaultPolicy returns a Policy with broadly-useful defaults that are suitable for a wide variety of use cases.

func (Policy) MatchFieldName

func (p Policy) MatchFieldName(names []string) (Disposition, int)

MatchFieldName returns a Disposition for the given field name if it matches any of the policy's field-name patterns. Otherwise it returns the empty string.

func (Policy) Validate

func (p Policy) Validate(models map[string]nlp.Model) []error

Validate checks that the policy is internally consistent.

type Report

type Report struct {
	// FieldName contains statistics about each field-name rule.
	FieldName []RuleReport
	// Heuristic contains statistics about each heuristic rule.
	Heuristic []RuleReport
	Summary   SummaryReport
}

Record provides statistics about how a policy was applied to a given input stream. It can be used to cross-check configuration and policy against real input data to make sure that sanitization is effective.

type RuleReport

type RuleReport struct {
	// Defn is the rule definition expressed as a string with compact, human-readable notation.
	Defn string
	// Fields is a list of distinct field names that were handled by a particular rule.
	Fields []string
	// Freq records the frequency (0-100%) with which this rule was applied.
	Freq Percentage
	// Safe determines the frequency of sanitized outputs that did not coincide with any input.
	//   100% => absolutely no overlap (output is perfectly sanitized)
	//     0% => complete overlap (output is effectively NOT sanitized, even if values are transposed; DANGER!)
	Safe Percentage
}

RuleReport describes how a rule was applied to a given input stream.

type Scrubber

type Scrubber struct {
	Verifier *Verifier
	// contains filtered or unexported fields
}

func NewScrubber

func NewScrubber(salt string, maskAll bool, policy *Policy, models map[string]nlp.Model) *Scrubber

func (*Scrubber) EraseString

func (sc *Scrubber) EraseString(s string, names []string) bool

EraseString signals to remove a string entirely from the input stream and replace it with a format-specific empty value.

It returns true for base64 encoded values since they are opaque and cannot be scrubbed; it's safest to remove them from the stream entirely.

It records hit statistics if a Verifier is provided, but does not record miss statistics under the assumption that the caller will always try to call ScrubString() if this returns false.

func (*Scrubber) ScrubData

func (sc *Scrubber) ScrubData(data any, names []string) any

ScrubData recursively scrubs maps and arrays in-place. It records no statistics with the Verifier.

func (*Scrubber) ScrubString

func (sc *Scrubber) ScrubString(s string, names []string) string

ScrubString applies rules to sanitize a string, preserving values that do not match any rule. It records statistics if a Verifier is provided.

type SummaryReport

type SummaryReport struct {
	Load Percentage
	Safe Percentage
}

type Verifier

type Verifier struct {
	// contains filtered or unexported fields
}

func NewVerifier

func NewVerifier(pol *Policy) *Verifier

NewVerifier creates a Scrubber linked to a Verifier. After all scrubbing is complete, call Report() to produce statistics.

func (*Verifier) Report

func (v *Verifier) Report() *Report

Report produces a YAML-printable summary of the Verifier's findings.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL