norm

package module

v0.0.0-...-9b058df Latest Latest Go to latest Published: Jul 10, 2023 License: MIT Imports: 8 Imported by: 2

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/alasdairforsythe/norm

Links

Open Source Insights

README ¶

norm

Basic text normalization.

Example:

norm := norm.NewNormalizer("nfd lines collapse trim")
output, err := norm.Normalize(input_slice)

Options

nfd lowercase accents quotemarks collapse trim leading-space lines

Documentation ¶

Index ¶

func Accents(b []byte) (output []byte, err error)
func AddLeadingSpace(b []byte) []byte
func Case(b []byte) (output []byte, err error)
func CaseAndAccents(b []byte) (output []byte, err error)
func Collapse(input []byte) []byte
func CollapseAndQuotemarks(input []byte) []byte
func CollapseAndQuotemarksAndUnixLines(input []byte) []byte
func CollapseAndUnixLines(input []byte) []byte
func NFD(input []byte) (output []byte, err error)
func NFDAndCase(b []byte) (output []byte, err error)
func Quotemarks(input []byte) []byte
func Trim(b []byte) []byte
func TrimAndAddLeadingSpace(b []byte) []byte
func UnixLines(input []byte) []byte
type Normalizer
- func NewNormalizer(s string) (Normalizer, error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func Accents ¶

func Accents(b []byte) (output []byte, err error)

Removes accents from UTF-8 text.

func AddLeadingSpace ¶

func AddLeadingSpace(b []byte) []byte

Adds a leading space if there isn't one already.

func Case ¶

func Case(b []byte) (output []byte, err error)

Lowercases UTF-8 text.

func CaseAndAccents ¶

func CaseAndAccents(b []byte) (output []byte, err error)

Lowercase and remove accents from UTF-8 text.

func Collapse ¶

func Collapse(input []byte) []byte

All sequences of 2 or more spaces are converted into single spaces.

func CollapseAndQuotemarks ¶

func CollapseAndQuotemarks(input []byte) []byte

All sequences of 2 or more spaces are converted into single spaces, and curly UTF-8 apostrophes and quotes are converted into ASCII.

func CollapseAndQuotemarksAndUnixLines ¶

func CollapseAndQuotemarksAndUnixLines(input []byte) []byte

Does all three in one loop.

func CollapseAndUnixLines ¶

func CollapseAndUnixLines(input []byte) []byte

Do both in one loop.

func NFD ¶

func NFD(input []byte) (output []byte, err error)

Performs UTF-8 NFD normalization.

func NFDAndCase ¶

func NFDAndCase(b []byte) (output []byte, err error)

Lowercases and performs UTF-8 NFD normalization.

func Quotemarks ¶

func Quotemarks(input []byte) []byte

Curly UTF-8 apostrophes and quotes are converted into ASCII.

func Trim ¶

func Trim(b []byte) []byte

Removes preceding and trailing whitespace (& some non-printable characters).

func TrimAndAddLeadingSpace ¶

func TrimAndAddLeadingSpace(b []byte) []byte

Removes preceding and trailing whitespace (& some non-printable characters), then adds space to the front.

func UnixLines ¶

func UnixLines(input []byte) []byte

Newlines converts /r/n to /n

Types ¶

type Normalizer ¶

type Normalizer struct {
	Flag uint8
}

func NewNormalizer ¶

func NewNormalizer(s string) (Normalizer, error)

func (Normalizer) Normalize ¶

func (n Normalizer) Normalize(data []byte) ([]byte, error)

func (Normalizer) SpecifiedAccents ¶

func (n Normalizer) SpecifiedAccents() bool

func (Normalizer) SpecifiedCollapse ¶

func (n Normalizer) SpecifiedCollapse() bool

func (Normalizer) SpecifiedLeadingSpace ¶

func (n Normalizer) SpecifiedLeadingSpace() bool

func (Normalizer) SpecifiedLowercase ¶

func (n Normalizer) SpecifiedLowercase() bool

func (Normalizer) SpecifiedNFD ¶

func (n Normalizer) SpecifiedNFD() bool

func (Normalizer) SpecifiedQuotemarks ¶

func (n Normalizer) SpecifiedQuotemarks() bool

func (Normalizer) SpecifiedTrim ¶

func (n Normalizer) SpecifiedTrim() bool

func (Normalizer) SpecifiedUnixLines ¶

func (n Normalizer) SpecifiedUnixLines() bool

func (Normalizer) String ¶

func (n Normalizer) String() string

Source Files ¶

View all Source files

norm.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL