tokenizer

package

v0.0.110 Latest Latest Go to latest Published: Mar 29, 2024 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/hupe1980/golc

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokenizer provides utilities for tokenizing text.

Index ¶

type Claude
- func NewClaude() (*Claude, error)
type Cohere
- func NewCohere(modelName string) (*Cohere, error)
type GPT2
- func NewGPT2() (*GPT2, error)
type GoogleGenAI
- func NewGoogleGenAI(client GoogleGenAIClient, model string) *GoogleGenAI
- func (t *GoogleGenAI) GetNumTokens(ctx context.Context, text string) (uint, error)
- func (t *GoogleGenAI) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)
type GoogleGenAIClient
type OpenAI
- func NewOpenAI(modelName string) *OpenAI

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Claude ¶ added in v0.0.47

type Claude struct {
	// contains filtered or unexported fields
}

func NewClaude ¶ added in v0.0.47

func NewClaude() (*Claude, error)

func (*Claude) GetNumTokens ¶ added in v0.0.47

func (t *Claude) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*Claude) GetNumTokensFromMessage ¶ added in v0.0.47

func (t *Claude) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*Claude) GetTokenIDs ¶ added in v0.0.47

func (t *Claude) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type Cohere ¶ added in v0.0.30

type Cohere struct {
	// contains filtered or unexported fields
}

func NewCohere ¶ added in v0.0.30

func NewCohere(modelName string) (*Cohere, error)

func (*Cohere) GetNumTokens ¶ added in v0.0.30

func (t *Cohere) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*Cohere) GetNumTokensFromMessage ¶ added in v0.0.30

func (t *Cohere) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*Cohere) GetTokenIDs ¶ added in v0.0.30

func (t *Cohere) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type GPT2 ¶ added in v0.0.24

type GPT2 struct {
	// contains filtered or unexported fields
}

func NewGPT2 ¶ added in v0.0.24

func NewGPT2() (*GPT2, error)

func (*GPT2) GetNumTokens ¶ added in v0.0.24

func (t *GPT2) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*GPT2) GetNumTokensFromMessage ¶ added in v0.0.24

func (t *GPT2) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*GPT2) GetTokenIDs ¶ added in v0.0.24

func (t *GPT2) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type GoogleGenAI ¶ added in v0.0.92

type GoogleGenAI struct {
	// contains filtered or unexported fields
}

func NewGoogleGenAI ¶ added in v0.0.92

func NewGoogleGenAI(client GoogleGenAIClient, model string) *GoogleGenAI

func (*GoogleGenAI) GetNumTokens ¶ added in v0.0.92

func (t *GoogleGenAI) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*GoogleGenAI) GetNumTokensFromMessage ¶ added in v0.0.92

func (t *GoogleGenAI) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

type GoogleGenAIClient ¶ added in v0.0.92

type GoogleGenAIClient interface {
	CountTokens(context.Context, *generativelanguagepb.CountTokensRequest, ...gax.CallOption) (*generativelanguagepb.CountTokensResponse, error)
}

GoogleGenAIClient is an interface for the GoogleGenAI model client.

type OpenAI ¶

type OpenAI struct {
	// contains filtered or unexported fields
}

func NewOpenAI ¶

func NewOpenAI(modelName string) *OpenAI

func (*OpenAI) GetNumTokens ¶

func (t *OpenAI) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*OpenAI) GetNumTokensFromMessage ¶

func (t *OpenAI) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*OpenAI) GetTokenIDs ¶

func (t *OpenAI) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL