Documentation ¶
Overview ¶
Package nats registers an llm implementation that uses NATS to communicate with a worker process. This is useful for building a multi-tier system where the workers are on isolated machines dedicated to LLM.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
func NewNATS ¶
NewNATS creates a new NATS client that supports prediction using the provided NATS connection and configuration. The configuration should specify `nats_worker_subject` to identify the worker that will be used to perform the request, otherwise llm.worker.default will be used.
func (*Client) Predict ¶
func (ct *Client) Predict(ctx context.Context, options map[string]any, input []string, fn func(llm.Prediction) error) (output string, err error)
Predict implements the llm.Predictor interface by sending a request to the worker and waiting for a response.
type Options ¶ added in v0.2.0
type Options struct { // WorkerSubject identifies the NATS subject where requests should be sent. This defaults to `llm.worker.default`, // which matches the worker's default WorkerSubject. WorkerSubject string `json:"nats_worker_subject"` // ClientName gives the client a name that is used to identify it in NATS server logs. This defaults to // `llm-client`. ClientName string `json:"nats_client_name"` // URL specifies the NATS client URL used to connect to the NATS server. This is used if Conn is nil, which is // the case if `llm.New("nats")` is used to create the client. This defaults to `nats://localhost:4222`. URL string `json:"nats_url"` // NKeyFile provides the path to an nkey seed / secret file that will be used to authenticate with the NATS server. // Ignored if a NATS connection is provided. NKeyFile string `json:"nats_nk"` // CA provides the path to a file containing trusted certificates for verifying the NATS server. Ignored if a // NATS connection is provided. If not provided, the host certificate authorities will be used. CA string `json:"nats_ca"` }
Options describes the options used to create a NATS client. This is unmarshalled from the configuration provided to llm.New or NewNATS.
Directories ¶
Path | Synopsis |
---|---|
Package msg describes the protocol used between the NATS client and worker.
|
Package msg describes the protocol used between the NATS client and worker. |
Package worker implements a NATS-based worker for large language models.
|
Package worker implements a NATS-based worker for large language models. |