uniseg

func FirstGraphemeCluster ¶ added in v0.3.0

func FirstGraphemeCluster(b []byte, state int) (cluster, rest []byte, width, newState int)

FirstGraphemeCluster returns the first grapheme cluster found in the given byte slice according to the rules of Unicode Standard Annex #29, Grapheme Cluster Boundaries. This function can be called continuously to extract all grapheme clusters from a byte slice, as illustrated in the example below.

If you don't know the current state, for example when calling the function for the first time, you must pass -1. For consecutive calls, pass the state and rest slice returned by the previous call.

The "rest" slice is the sub-slice of the original byte slice "b" starting after the last byte of the identified grapheme cluster. If the length of the "rest" slice is 0, the entire byte slice "b" has been processed. The "cluster" byte slice is the sub-slice of the input slice containing the identified grapheme cluster.

The returned width is the width of the grapheme cluster for most monospace fonts where a value of 1 represents one character cell.

Given an empty byte slice "b", the function returns nil values.

While slightly less convenient than using the Graphemes class, this function has much better performance and makes no allocations. It lends itself well to large byte slices.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("🇩🇪🏳️‍🌈!")
	state := -1
	var c []byte
	for len(b) > 0 {
		var width int
		c, b, width, state = uniseg.FirstGraphemeCluster(b, state)
		fmt.Println(string(c), width)
	}
}

Output:

🇩🇪 2
🏳️‍🌈 2
! 1

func FirstGraphemeClusterInString ¶ added in v0.3.0

func FirstGraphemeClusterInString(str string, state int) (cluster, rest string, width, newState int)

FirstGraphemeClusterInString is like FirstGraphemeCluster but its input and outputs are strings.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "🇩🇪🏳️‍🌈!"
	state := -1
	var c string
	for len(str) > 0 {
		var width int
		c, str, width, state = uniseg.FirstGraphemeClusterInString(str, state)
		fmt.Println(c, width)
	}
}

Output:

🇩🇪 2
🏳️‍🌈 2
! 1

func FirstLineSegment ¶ added in v0.3.0

func FirstLineSegment(b []byte, state int) (segment, rest []byte, mustBreak bool, newState int)

FirstLineSegment returns the prefix of the given byte slice after which a decision to break the string over to the next line can or must be made, according to the rules of Unicode Standard Annex #14. This is used to implement line breaking.

Line breaking, also known as word wrapping, is the process of breaking a section of text into lines such that it will fit in the available width of a page, window or other display area.

The returned "segment" may not be broken into smaller parts, unless no other breaking opportunities present themselves, in which case you may break by grapheme clusters (using the FirstGraphemeCluster function to determine the grapheme clusters).

The "mustBreak" flag indicates whether you MUST break the line after the given segment (true), for example after newline characters, or you MAY break the line after the given segment (false).

This function can be called continuously to extract all non-breaking sub-sets from a byte slice, as illustrated in the example below.

If you don't know the current state, for example when calling the function for the first time, you must pass -1. For consecutive calls, pass the state and rest slice returned by the previous call.

The "rest" slice is the sub-slice of the original byte slice "b" starting after the last byte of the identified line segment. If the length of the "rest" slice is 0, the entire byte slice "b" has been processed. The "segment" byte slice is the sub-slice of the input slice containing the identified line segment.

Given an empty byte slice "b", the function returns nil values.

Note that in accordance with UAX #14 LB3, the final segment will end with "mustBreak" set to true. You can choose to ignore this by checking if the length of the "rest" slice is 0 and calling HasTrailingLineBreak or HasTrailingLineBreakInString on the last rune.

Note also that this algorithm may break within grapheme clusters. This is addressed in Section 8.2 Example 6 of UAX #14. To avoid this, you can use the Step function instead.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("First line.\nSecond line.")
	state := -1
	var (
		c         []byte
		mustBreak bool
	)
	for len(b) > 0 {
		c, b, mustBreak, state = uniseg.FirstLineSegment(b, state)
		fmt.Printf("(%s)", string(c))
		if mustBreak {
			fmt.Print("!")
		}
	}
}

Output:

(First )(line.
)!(Second )(line.)!

func FirstLineSegmentInString ¶ added in v0.3.0

func FirstLineSegmentInString(str string, state int) (segment, rest string, mustBreak bool, newState int)

FirstLineSegmentInString is like FirstLineSegment but its input and outputs are strings.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "First line.\nSecond line."
	state := -1
	var (
		c         string
		mustBreak bool
	)
	for len(str) > 0 {
		c, str, mustBreak, state = uniseg.FirstLineSegmentInString(str, state)
		fmt.Printf("(%s)", c)
		if mustBreak {
			fmt.Println(" < must break")
		} else {
			fmt.Println(" < may break")
		}
	}
}

Output:

(First ) < may break
(line.
) < must break
(Second ) < may break
(line.) < must break

func FirstSentence ¶ added in v0.3.0

func FirstSentence(b []byte, state int) (sentence, rest []byte, newState int)

FirstSentence returns the first sentence found in the given byte slice according to the rules of Unicode Standard Annex #29, Sentence Boundaries. This function can be called continuously to extract all sentences from a byte slice, as illustrated in the example below.

If you don't know the current state, for example when calling the function for the first time, you must pass -1. For consecutive calls, pass the state and rest slice returned by the previous call.

The "rest" slice is the sub-slice of the original byte slice "b" starting after the last byte of the identified sentence. If the length of the "rest" slice is 0, the entire byte slice "b" has been processed. The "sentence" byte slice is the sub-slice of the input slice containing the identified sentence.

Given an empty byte slice "b", the function returns nil values.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("This is sentence 1.0. And this is sentence two.")
	state := -1
	var c []byte
	for len(b) > 0 {
		c, b, state = uniseg.FirstSentence(b, state)
		fmt.Printf("(%s)\n", string(c))
	}
}

Output:

(This is sentence 1.0. )
(And this is sentence two.)

func FirstSentenceInString ¶ added in v0.3.0

func FirstSentenceInString(str string, state int) (sentence, rest string, newState int)

FirstSentenceInString is like FirstSentence but its input and outputs are strings.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "This is sentence 1.0. And this is sentence two."
	state := -1
	var c string
	for len(str) > 0 {
		c, str, state = uniseg.FirstSentenceInString(str, state)
		fmt.Printf("(%s)\n", c)
	}
}

Output:

(This is sentence 1.0. )
(And this is sentence two.)

func FirstWord ¶ added in v0.3.0

func FirstWord(b []byte, state int) (word, rest []byte, newState int)

FirstWord returns the first word found in the given byte slice according to the rules of Unicode Standard Annex #29, Word Boundaries. This function can be called continuously to extract all words from a byte slice, as illustrated in the example below.

If you don't know the current state, for example when calling the function for the first time, you must pass -1. For consecutive calls, pass the state and rest slice returned by the previous call.

The "rest" slice is the sub-slice of the original byte slice "b" starting after the last byte of the identified word. If the length of the "rest" slice is 0, the entire byte slice "b" has been processed. The "word" byte slice is the sub-slice of the input slice containing the identified word.

Given an empty byte slice "b", the function returns nil values.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("Hello, world!")
	state := -1
	var c []byte
	for len(b) > 0 {
		c, b, state = uniseg.FirstWord(b, state)
		fmt.Printf("(%s)\n", string(c))
	}
}

Output:

(Hello)
(,)
( )
(world)
(!)

func FirstWordInString ¶ added in v0.3.0

func FirstWordInString(str string, state int) (word, rest string, newState int)

FirstWordInString is like FirstWord but its input and outputs are strings.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "Hello, world!"
	state := -1
	var c string
	for len(str) > 0 {
		c, str, state = uniseg.FirstWordInString(str, state)
		fmt.Printf("(%s)\n", c)
	}
}

Output:

(Hello)
(,)
( )
(world)
(!)

func GraphemeClusterCount ¶

func GraphemeClusterCount(s string) (n int)

GraphemeClusterCount returns the number of user-perceived characters (grapheme clusters) for the given string.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	n := uniseg.GraphemeClusterCount("🇩🇪🏳️‍🌈")
	fmt.Println(n)
}

Output:

2

func HasTrailingLineBreak ¶ added in v0.3.4

func HasTrailingLineBreak(b []byte) bool

HasTrailingLineBreak returns true if the last rune in the given byte slice is one of the hard line break code points defined in LB4 and LB5 of UAX #14.

func HasTrailingLineBreakInString ¶ added in v0.3.4

func HasTrailingLineBreakInString(str string) bool

HasTrailingLineBreakInString is like HasTrailingLineBreak but for a string.

func ReverseString ¶ added in v0.4.3

func ReverseString(s string) string

ReverseString reverses the given string while observing grapheme cluster boundaries.

func Step ¶ added in v0.3.0

func Step(b []byte, state int) (cluster, rest []byte, boundaries int, newState int)

Step returns the first grapheme cluster (user-perceived character) found in the given byte slice. It also returns information about the boundary between that grapheme cluster and the one following it as well as the monospace width of the grapheme cluster. There are three types of boundary information: word boundaries, sentence boundaries, and line breaks. This function is therefore a combination of FirstGraphemeCluster, FirstWord, FirstSentence, and FirstLineSegment.

The "boundaries" return value can be evaluated as follows:

• boundaries&MaskWord != 0: The boundary is a word boundary.
• boundaries&MaskWord == 0: The boundary is not a word boundary.
• boundaries&MaskSentence != 0: The boundary is a sentence boundary.
• boundaries&MaskSentence == 0: The boundary is not a sentence boundary.
• boundaries&MaskLine == LineDontBreak: You must not break the line at the boundary.
• boundaries&MaskLine == LineMustBreak: You must break the line at the boundary.
• boundaries&MaskLine == LineCanBreak: You may or may not break the line at the boundary.
• boundaries >> ShiftWidth: The width of the grapheme cluster for most monospace fonts where a value of 1 represents one character cell.

This function can be called continuously to extract all grapheme clusters from a byte slice, as illustrated in the examples below.

If you don't know which state to pass, for example when calling the function for the first time, you must pass -1. For consecutive calls, pass the state and rest slice returned by the previous call.

The "rest" slice is the sub-slice of the original byte slice "b" starting after the last byte of the identified grapheme cluster. If the length of the "rest" slice is 0, the entire byte slice "b" has been processed. The "cluster" byte slice is the sub-slice of the input slice containing the first identified grapheme cluster.

Given an empty byte slice "b", the function returns nil values.

While slightly less convenient than using the Graphemes class, this function has much better performance and makes no allocations. It lends itself well to large byte slices.

Note that in accordance with UAX #14 LB3, the final segment will end with a mandatory line break (boundaries&MaskLine == LineMustBreak). You can choose to ignore this by checking if the length of the "rest" slice is 0 and calling HasTrailingLineBreak or HasTrailingLineBreakInString on the last rune.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("🇩🇪🏳️‍🌈!")
	state := -1
	var c []byte
	for len(b) > 0 {
		var boundaries int
		c, b, boundaries, state = uniseg.Step(b, state)
		fmt.Println(string(c), boundaries>>uniseg.ShiftWidth)
	}
}

Output:

🇩🇪 2
🏳️‍🌈 2
! 1

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("First line.\nSecond line.")
	state := -1
	var (
		c          []byte
		boundaries int
	)
	for len(b) > 0 {
		c, b, boundaries, state = uniseg.Step(b, state)
		fmt.Print(string(c))
		if boundaries&uniseg.MaskLine == uniseg.LineCanBreak {
			fmt.Print("|")
		} else if boundaries&uniseg.MaskLine == uniseg.LineMustBreak {
			fmt.Print("‖")
		}
	}
}

Output:

First |line.
‖Second |line.‖

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("This is sentence 1.0. And this is sentence two.")
	state := -1
	var (
		c          []byte
		boundaries int
	)
	for len(b) > 0 {
		c, b, boundaries, state = uniseg.Step(b, state)
		fmt.Print(string(c))
		if boundaries&uniseg.MaskSentence != 0 {
			fmt.Print("|")
		}
	}
}

Output:

This is sentence 1.0. |And this is sentence two.|

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	b := []byte("Hello, world!")
	state := -1
	var (
		c          []byte
		boundaries int
	)
	for len(b) > 0 {
		c, b, boundaries, state = uniseg.Step(b, state)
		fmt.Print(string(c))
		if boundaries&uniseg.MaskWord != 0 {
			fmt.Print("|")
		}
	}
}

Output:

Hello|,| |world|!|

func StepString ¶ added in v0.3.0

func StepString(str string, state int) (cluster, rest string, boundaries int, newState int)

StepString is like Step but its input and outputs are strings.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "🇩🇪🏳️‍🌈!"
	state := -1
	var c string
	for len(str) > 0 {
		var boundaries int
		c, str, boundaries, state = uniseg.StepString(str, state)
		fmt.Println(c, boundaries>>uniseg.ShiftWidth)
	}
}

Output:

🇩🇪 2
🏳️‍🌈 2
! 1

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "First line.\nSecond line."
	state := -1
	var (
		c          string
		boundaries int
	)
	for len(str) > 0 {
		c, str, boundaries, state = uniseg.StepString(str, state)
		fmt.Print(c)
		if boundaries&uniseg.MaskLine == uniseg.LineCanBreak {
			fmt.Print("|")
		} else if boundaries&uniseg.MaskLine == uniseg.LineMustBreak {
			fmt.Print("‖")
		}
	}
}

Output:

First |line.
‖Second |line.‖

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "This is sentence 1.0. And this is sentence two."
	state := -1
	var (
		c          string
		boundaries int
	)
	for len(str) > 0 {
		c, str, boundaries, state = uniseg.StepString(str, state)
		fmt.Print(c)
		if boundaries&uniseg.MaskSentence != 0 {
			fmt.Print("|")
		}
	}
}

Output:

This is sentence 1.0. |And this is sentence two.|

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	str := "Hello, world!"
	state := -1
	var (
		c          string
		boundaries int
	)
	for len(str) > 0 {
		c, str, boundaries, state = uniseg.StepString(str, state)
		fmt.Print(c)
		if boundaries&uniseg.MaskWord != 0 {
			fmt.Print("|")
		}
	}
}

Output:

Hello|,| |world|!|

func StringWidth ¶ added in v0.4.0

func StringWidth(s string) (width int)

StringWidth returns the monospace width for the given string, that is, the number of same-size cells to be occupied by the string.

package main

import (
	"fmt"

	"github.com/rivo/uniseg"
)

func main() {
	fmt.Println(uniseg.StringWidth("Hello, 世界"))
}

Output:

11