README
¶
gotoken generator
This directory contains the generator that converts OpenAI .tiktoken files
into data embedded in Go source files.
Usage
From this directory, run go generate. It will output the generated Go source
at ../{encoding}/data.go, where {encoding} gets replaced with each of the
supported tokenizers.
Documentation
¶
Overview ¶
Package gen generates the data.go files for the gotoken's encoding sub-packages. To run, "go generate". This will overwrite files in ../{encoding}/data.go!
This program generates data structures based on the ".tiktoken" files provided by OpenAI. These files are licensed under the MIT license and include the following notice: Copyright (c) 2022 OpenAI, Shantanu Jain.
Source Files
Click to show internal directories.
Click to hide internal directories.