libsq

package
v0.48.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 12, 2024 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package libsq implements the core sq functionality. The ExecuteSLQ function is the entrypoint for executing a SLQ query, which may interact with several data sources. The QuerySQL function executes a SQL query against a single source. Both functions ultimately send their result records to a RecordWriter. Implementations of RecordWriter write records to a destination, such as a JSON or CSV file. The NewDBWriter function returns a RecordWriter that writes records to a database.

Index

Constants

View Source
const MsgIngestRecords = "Ingesting records"

MsgIngestRecords is the typical message used with libsq.NewDBWriter to indicate that records are being ingested.

Variables

This section is empty.

Functions

func ExecuteSLQ

func ExecuteSLQ(ctx context.Context, qc *QueryContext, query string, recw RecordWriter) error

ExecuteSLQ executes the slq query, writing the results to recw. The caller is responsible for closing qc.

func QuerySQL

func QuerySQL(ctx context.Context, grip driver.Grip, db sqlz.DB,
	recw RecordWriter, query string, args ...any,
) error

QuerySQL executes the SQL query, writing the results to recw. If db is non-nil, the query is executed against it. Otherwise, the connection is obtained from grip. Note that QuerySQL may return before recw has finished writing, thus the caller may wish to wait for recw to complete. The caller is responsible for closing grip (and db, if non-nil).

func SLQ2SQL added in v0.25.0

func SLQ2SQL(ctx context.Context, qc *QueryContext, query string) (targetSQL string, err error)

SLQ2SQL simulates execution of a SLQ query, but instead of executing the resulting SQL query, that ultimate SQL is returned. Effectively it is equivalent to libsq.ExecuteSLQ, but without the execution.

Types

type DBWriter

type DBWriter struct {
	// contains filtered or unexported fields
}

DBWriter implements RecordWriter, writing records to a database table.

func NewDBWriter

func NewDBWriter(msg string, destGrip driver.Grip, destTbl string, recChSize int,
	preWriteHooks ...DBWriterPreWriteHook,
) *DBWriter

NewDBWriter returns a new writer than implements RecordWriter. The writer writes records from recordCh to destTbl in destGrip. The recChSize param controls the size of recordCh returned by the writer's Open method.

func (*DBWriter) Open

func (w *DBWriter) Open(ctx context.Context, cancelFn context.CancelFunc, recMeta record.Meta) (
	chan<- record.Record, <-chan error, error,
)

Open implements RecordWriter.

func (*DBWriter) Wait

func (w *DBWriter) Wait() (written int64, err error)

Wait implements RecordWriter.

type DBWriterPreWriteHook

type DBWriterPreWriteHook func(ctx context.Context, recMeta record.Meta, destGrip driver.Grip, tx sqlz.DB) error

DBWriterPreWriteHook is a function that is invoked before DBWriter begins writing.

func DBWriterCreateTableIfNotExistsHook

func DBWriterCreateTableIfNotExistsHook(destTblName string) DBWriterPreWriteHook

DBWriterCreateTableIfNotExistsHook returns a hook that creates destTblName if it does not exist.

type QueryContext added in v0.31.0

type QueryContext struct {
	// Collection is the set of sources.
	Collection *source.Collection

	// Grips mediates access to driver.Grip instances.
	Grips *driver.Grips

	// Args defines variables that are substituted into the query.
	// May be nil or empty.
	Args map[string]string

	// PreExecStmts are statements that are executed before the query.
	// These can be used for edge-case behavior, such as setting up
	// variables in the session. These stmts are typically loaded
	// from render.Fragments.PreExecStmts.
	//
	// See also: QueryContext.PostExecStmts.
	PreExecStmts []string

	// PostExecStmts are statements that are executed after the query.
	//
	// See also: QueryContext.PreExecStmts.
	PostExecStmts []string
}

QueryContext encapsulates the context a SLQ query is executed within.

type RecordWriter

type RecordWriter interface {
	// Open opens the RecordWriter for writing records described
	// by recMeta, returning a non-nil err if the initial open
	// operation fails.
	//
	// Records received on recCh are written to the
	// destination, possibly buffered, until recCh is closed.
	// Therefore, the caller must close recCh to indicate that
	// all records have been sent, so that the writer can
	// perform any end-of-stream actions. The caller can use the Wait
	// method to wait for the writer to complete. The returned errCh
	// will also be closed when complete.
	//
	// Any underlying write error is sent on errCh,
	// at which point the writer is defunct. Thus it is the
	// responsibility of the sender to check errCh before
	// sending again on recordCh. Note that implementations
	// may send more than one error on errCh, and that errCh
	// will be closed when the writer completes. Note also that the
	// errors sent on errCh are accumulated internally by the writer and
	// returned from the Wait method (if more than one error, they
	// may be combined into a multierr).
	//
	// The caller can stop the RecordWriter by cancelling ctx.
	// When ctx is done, the writer shuts down processing
	// of recCh and returns ctx.Err on errCh  (possibly
	// with additional errors from the shutdown).
	//
	// If cancelFn is non-nil, it is invoked only by the writer's Wait method.
	// If the Open method itself returns an error, it is the caller's
	// responsibility to invoke cancelFn to prevent resource leakage.
	//
	// It is noted that the existence of the cancelFn param is an unusual
	// construction. This mechanism exists to enable a goroutine to wait
	// on the writer outside the function that invoked Open, without
	// having to pass cancelFn around.
	Open(ctx context.Context, cancelFn context.CancelFunc, recMeta record.Meta) (
		recCh chan<- record.Record, errCh <-chan error, err error)

	// Wait waits for the writer to complete and returns the number of
	// written rows and any error (which may be a multierr).
	// The written value may be non-zero even in the presence of an error.
	// If a cancelFn was passed to Open, it will be invoked before Wait returns.
	Wait() (written int64, err error)
}

RecordWriter is the interface for writing records to a destination. The Open method returns a channel to which the records are sent. The Wait method allows the sender to wait for the writer to complete.

Directories

Path Synopsis
ast
Package ast holds types and functionality for the SLQ AST.
Package ast holds types and functionality for the SLQ AST.
antlrz
Package antlrz contains utilities for working with ANTLR4.
Package antlrz contains utilities for working with ANTLR4.
render
Package render provides the mechanism for rendering ast into SQL.
Package render provides the mechanism for rendering ast into SQL.
Package core contains only libsq core sub-packages.
Package core contains only libsq core sub-packages.
cleanup
Package cleanup provides functionality for executing cleanup functions.
Package cleanup provides functionality for executing cleanup functions.
colorz
Package colorz provides supplemental color functionality.
Package colorz provides supplemental color functionality.
debugz
Package debugz contains functionality for debugging.
Package debugz contains functionality for debugging.
diffdoc
Package diffdoc provides core diff functionality, with a focus on streaming and concurrency.
Package diffdoc provides core diff functionality, with a focus on streaming and concurrency.
diffdoc/internal/go-udiff
Package udiff computes differences between text files or strings.
Package udiff computes differences between text files or strings.
diffdoc/internal/go-udiff/difftest
Package difftest supplies a set of tests that will operate on any implementation of a diff algorithm as exposed by diff "github.com/neilotoole/sq/libsq/core/diffdoc/internal/go-udiff"
Package difftest supplies a set of tests that will operate on any implementation of a diff algorithm as exposed by diff "github.com/neilotoole/sq/libsq/core/diffdoc/internal/go-udiff"
diffdoc/internal/go-udiff/lcs
package lcs contains code to find longest-common-subsequences (and diffs)
package lcs contains code to find longest-common-subsequences (and diffs)
diffdoc/internal/go-udiff/myers
Package myers implements the Myers diff algorithm.
Package myers implements the Myers diff algorithm.
errz
Package errz is sq's error package.
Package errz is sq's error package.
execz
Package execz builds on stdlib os/exec.
Package execz builds on stdlib os/exec.
ioz
Package ioz contains supplemental io functionality.
Package ioz contains supplemental io functionality.
ioz/checksum
Package checksum provides functions for working with checksums.
Package checksum provides functions for working with checksums.
ioz/contextio
Package contextio provides io decorators that are context-aware.
Package contextio provides io decorators that are context-aware.
ioz/httpz
Package httpz provides functionality supplemental to stdlib http.
Package httpz provides functionality supplemental to stdlib http.
ioz/lockfile
Package lockfile implements a pid lock file mechanism.
Package lockfile implements a pid lock file mechanism.
jointype
Package jointype enumerates the various SQL JOIN types.
Package jointype enumerates the various SQL JOIN types.
kind
Package kind encapsulates the notion of data "kind": that is, it is an abstraction over data types across implementations.
Package kind encapsulates the notion of data "kind": that is, it is an abstraction over data types across implementations.
langz
Package langz contains miscellaneous functionality supplemental to core golang stuff like slices and channels.
Package langz contains miscellaneous functionality supplemental to core golang stuff like slices and channels.
lg
Package lg contains utility functions for working with slog.
Package lg contains utility functions for working with slog.
lg/devlog
Package devlog contains a custom slog.Handler for developer-friendly log output.
Package devlog contains a custom slog.Handler for developer-friendly log output.
lg/lga
Package lga ("log attribute") holds constants for log attribute names.
Package lga ("log attribute") holds constants for log attribute names.
lg/lgm
Package lgm ("log message") contains constants for log messages.
Package lgm ("log message") contains constants for log messages.
lg/lgt
Package lgt provides a mechanism for getting a *slog.Logger that outputs to testing.T. See lgt.New.
Package lgt provides a mechanism for getting a *slog.Logger that outputs to testing.T. See lgt.New.
lg/slogbuf
Package slogbuf implements a Buffer that stores log records that can later be replayed on a slog.Handler.
Package slogbuf implements a Buffer that stores log records that can later be replayed on a slog.Handler.
lg/userlogdir
Package userlogdir has a single function, UserLogDir, that returns an OS-specific path for storing user logs.
Package userlogdir has a single function, UserLogDir, that returns an OS-specific path for storing user logs.
oncecache
Package oncecache contains a strongly-typed, concurrency-safe, context-aware, dependency-free, in-memory, on-demand object [Cache], focused on write-once, read-often ergonomics.
Package oncecache contains a strongly-typed, concurrency-safe, context-aware, dependency-free, in-memory, on-demand object [Cache], focused on write-once, read-often ergonomics.
options
Package options implements config options.
Package options implements config options.
progress
Package progress contains progress bar widget functionality.
Package progress contains progress bar widget functionality.
record
Package record holds the record.Record type, which is the core type for holding query results.
Package record holds the record.Record type, which is the core type for holding query results.
retry
Package retry implements retry functionality.
Package retry implements retry functionality.
runtimez
Package runtimez provides functionality supplemental to stdlib's runtime pkg.
Package runtimez provides functionality supplemental to stdlib's runtime pkg.
schema
Package schema provides functionality for modeling SQL constructs.
Package schema provides functionality for modeling SQL constructs.
sqlz
Package sqlz contains core types such as Kind and Record.
Package sqlz contains core types such as Kind and Record.
stringz
Package stringz contains string functions similar in spirit to the stdlib strings package.
Package stringz contains string functions similar in spirit to the stdlib strings package.
tablefq
Package tablefq is a tiny package that holds the tablefq.T type, which is a fully-qualified SQL table name.
Package tablefq is a tiny package that holds the tablefq.T type, which is a fully-qualified SQL table name.
tailbuf
Package tailbuf contains a tail buffer [Buf] of fixed size that provides a window on the tail of the items written via [Buf.Write].
Package tailbuf contains a tail buffer [Buf] of fixed size that provides a window on the tail of the items written via [Buf.Write].
termz
Package termz contains a handful of terminal utilities.
Package termz contains a handful of terminal utilities.
timez
Package timez contains time functionality.
Package timez contains time functionality.
tuning
Package tuning contains tuning options.
Package tuning contains tuning options.
urlz
Package urlz contains URL utility functionality.
Package urlz contains URL utility functionality.
dialect
Package dialect contains functionality for SQL dialects.
Package dialect contains functionality for SQL dialects.
Package files contains functionality for dealing with files, including remote files (e.g.
Package files contains functionality for dealing with files, including remote files (e.g.
internal/downloader
Package downloader provides a mechanism for getting files from HTTP/S URLs, making use of a mostly RFC-compliant cache.
Package downloader provides a mechanism for getting files from HTTP/S URLs, making use of a mostly RFC-compliant cache.
Package source provides functionality for dealing with data sources.
Package source provides functionality for dealing with data sources.
drivertype
Package drivertype defines drivertype.Type, which is the type of a driver, e.g.
Package drivertype defines drivertype.Type, which is the type of a driver, e.g.
location
Package location contains functionality related to source location.
Package location contains functionality related to source location.
mdcache
Package mdcache contains a [Cache] that caches source metadata.
Package mdcache contains a [Cache] that caches source metadata.
metadata
Package metadata contains types that model source metadata.
Package metadata contains types that model source metadata.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL