Documentation
¶
Overview ¶
Package nats registers an llm implementation that uses NATS to communicate with a worker process. This is useful for building a multi-tier system where the workers are on isolated machines dedicated to LLM.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
func NewNATS ¶
NewNATS creates a new NATS client that supports prediction using the provided NATS connection and configuration. The configuration should specify `nats_worker_subject` to identify the worker that will be used to perform the request, otherwise llm.worker.default will be used.
func (*Client) Predict ¶
func (ct *Client) Predict(ctx context.Context, options map[string]any, input []string, fn func(llm.Prediction) error) (output string, err error)
Predict implements the llm.Predictor interface by sending a request to the worker and waiting for a response.
type Options ¶ added in v0.2.0
type Options struct {
// WorkerSubject identifies the NATS subject where requests should be sent. This defaults to `llm.worker.default`,
// which matches the worker's default WorkerSubject.
WorkerSubject string `json:"nats_worker_subject"`
// ClientName gives the client a name that is used to identify it in NATS server logs. This defaults to
// `llm-client`.
ClientName string `json:"nats_client_name"`
// URL specifies the NATS client URL used to connect to the NATS server. This is used if Conn is nil, which is
// the case if `llm.New("nats")` is used to create the client. This defaults to `nats://localhost:4222`.
URL string `json:"nats_url"`
// NKeyFile provides the path to an nkey seed / secret file that will be used to authenticate with the NATS server.
// Ignored if a NATS connection is provided.
NKeyFile string `json:"nats_nk"`
// CA provides the path to a file containing trusted certificates for verifying the NATS server. Ignored if a
// NATS connection is provided. If not provided, the host certificate authorities will be used.
CA string `json:"nats_ca"`
}
Options describes the options used to create a NATS client. This is unmarshalled from the configuration provided to llm.New or NewNATS.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package msg describes the protocol used between the NATS client and worker.
|
Package msg describes the protocol used between the NATS client and worker. |
|
Package worker implements a NATS-based worker for large language models.
|
Package worker implements a NATS-based worker for large language models. |