Documentation
¶
Overview ¶
Package worker implements a NATS-based worker for large language models.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Option ¶
type Option func(*worker)
An Option is a function that alters a worker's behavior.
func Conn ¶
func Conn(conn *nats.Conn) Option
Conn sets the NATS connection to use for getting requests and publishing responses. This is an alternative to letting the worker manage its own connection.
func Hook ¶
func Hook(hook func(ctx context.Context, model llm.Interface, req *msg.PredictRequest) error) Option
A Hook is a function that is called before a prediction is made, allowing it to alter the request. This is useful when a worker can do some preprocessing of input data, such as reducing the input length to fit context.
type Options ¶
type Options struct {
client.Options
// LLM implementation to use.
LLMType string `json:"type"`
// Dir provides the directory where models are stored, defaults to "."
Dir string `json:"dir,omitempty"`
}
Options describes the configuration options for a NATS-based worker. Note that this configuration will be passed on to the LLM driver, so the driver may have additional configuration options.
Source Files
Click to show internal directories.
Click to hide internal directories.