README
¶
pgx-collect
pgx-collect is a nearly drop-in replacement for the pgx collection functions that offer lower overhead.
pgx offers a very convenient set of functions for parsing values from rows, while handling all of the boilerplate. However, these functions sometimes repeat significant amounts of work when processing each row, and cause unnecessary allocations that scale with O(rows) or even O(rows * columns), which puts more load on the garbage collector. By rewriting these functions to minimize repeated work and allocations, this library provides the same convenient functionality while improving performance.
Usage
For any Collect and RowTo function defined in pgx, pgx-collect has an equivalent with the same name that you can swap in for the same behavior.
For example:
import (
"github.com/jackc/pgx/v5"
)
...
values, err := pgx.CollectRows(rows, pgx.RowToStructByName[Record])
can be replaced with:
import (
"github.com/jackc/pgx/v5"
pgxc "github.com/zolstein/pgx-collect"
)
...
values, err := pgxc.CollectRows(rows, pgxc.RowToStructByName[Record])
If, when migrating from code using the pgx Collect functions, you need to reuse a custom pgx.RowToFunc, you can use the Adapt function as follows. However, be aware that this may have sub-par performance and allocate more than using native pgx-collect constructs.
import (
"github.com/jackc/pgx/v5"
pgxc "github.com/zolstein/pgx-collect"
)
...
var customFunc pgx.RowToFunc[Record]
values, err := pgxc.CollectRows(rows, pgxc.Adapt(customFunc))
Who would benefit from this library?
On some level, everyone using the functions CollectRows or AppendRows could benefit. The biggest wins will be seen by people parsing queries with many rows / columns using the reflective struct-mapping RowTo functions. I.e. RowToStructBy....
By comparison, people querying small numbers of rows and parsing rows as primatives or maps, are unlikely to see much benefit (for now). pgx-collect support for these operations is provided mainly for convenience, rather than because it can provide a large speedup.
How much faster is it?
When using AppendRows to query 1000 rows and 4 columns in a benchmark, I've demonstrated a 3-4x reduction in total runtime, a 10x reduction in memory allocated, and a 100x reduction in objects allocated. These numbers will vary greatly by workload. Real-world benefits are unlikely to be this large, but they may still improve query throughput, especially in pathological cases.
I encourage anyone using these pgx's collect functions to try out pgx-collect, measure the difference, and share the results, particularly if there are further opportunities to optimize.
Why not merge into pgx? Why create a separate library?
I would love for this code to be merged into pgx. However, even though the pgx-collect API is designed as a drop-in replacement for all idiomatic use-cases, the performance improvements required changing the types of some values in the public API - thus, if these changes were integrated into pgx, some working code could stop compiling.
If at some time in the future pgx can make backward-incompatible changes, I hope it integrates these changes. I also hope they can integrate as many of the optimizations as possible, though as far as I can tell the API exposed by pgx makes this impossible. In lieu of this, I want this code to be available to anyone who wants to use it or learn from it.
Documentation
¶
Index ¶
- func AppendRows[T any, S ~[]T](slice S, rows pgx.Rows, into RowSpec[T]) (S, error)
- func AppendRowsUsing[T any, S ~[]T](slice S, rows pgx.Rows, scanner Scanner[T]) (s S, err error)
- func CollectExactlyOneRow[T any](rows pgx.Rows, into RowSpec[T]) (T, error)
- func CollectExactlyOneRowUsing[T any](rows pgx.Rows, scanner Scanner[T]) (T, error)
- func CollectOneRow[T any](rows pgx.Rows, into RowSpec[T]) (T, error)
- func CollectOneRowUsing[T any](rows pgx.Rows, scanner Scanner[T]) (T, error)
- func CollectRows[T any](rows pgx.Rows, into RowSpec[T]) ([]T, error)
- func CollectRowsUsing[T any](rows pgx.Rows, scanner Scanner[T]) ([]T, error)
- func RowTo[T any]() rowSpecRes[T]
- func RowToAddrOf[T any]() rowSpecRes[*T]
- func RowToAddrOfStructByName[T any]() rowSpecRes[*T]
- func RowToAddrOfStructByNameLax[T any]() rowSpecRes[*T]
- func RowToAddrOfStructByPos[T any]() rowSpecRes[*T]
- func RowToMap() rowSpecRes[map[string]any]
- func RowToStructByName[T any]() rowSpecRes[T]
- func RowToStructByNameLax[T any]() rowSpecRes[T]
- func RowToStructByPos[T any]() rowSpecRes[T]
- type RowSpec
- type Scanner
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AppendRows ¶
AppendRows iterates through rows, scanning each row according to into, and appending the results into a slice of T.
func AppendRowsUsing ¶
AppendRowsUsing iterates through rows, scanning each row with the scanner, and appending the results into a slice of T.
func CollectExactlyOneRow ¶
CollectExactlyOneRow scans the first row in rows and returns the result.
- If no rows are found returns an error where errors.Is(pgx.ErrNoRows) is true.
- If more than 1 row is found returns an error where errors.Is(ErrTooManyRows) is true.
func CollectExactlyOneRowUsing ¶
CollectExactlyOneRowUsing scans the first row in rows and returns the result.
- If no rows are found returns an error where errors.Is(pgx.ErrNoRows) is true.
- If more than 1 row is found returns an error where errors.Is(ErrTooManyRows) is true.
func CollectOneRow ¶
CollectOneRow scans the first row in rows and returns the result. If no rows are found returns an error where errors.Is(pgx.ErrNoRows) is true. CollectOneRow is to CollectRows as QueryRow is to Query.
func CollectOneRowUsing ¶
CollectOneRowUsing scans the first row in rows and returns the result. If no rows are found returns an error where errors.Is(pgx.ErrNoRows) is true. CollectOneRowUsing is to CollectRowsUsing as QueryRow is to Query.
func CollectRows ¶
CollectRows iterates through rows, scanning each row according to into, and collecting the results into a slice of T.
func CollectRowsUsing ¶
CollectRowsUsing iterates through rows, scanning each row with the scanner, and collecting the results into a slice of T.
func RowToAddrOfStructByName ¶
func RowToAddrOfStructByName[T any]() rowSpecRes[*T]
RowToAddrOfStructByName scans a row into a *T. T must be a struct. T must have the same number of named public fields as row has fields. The row and T fields will be matched by name. The match is case-insensitive. The database column name can be overridden with a "db" struct tag. If the "db" struct tag is "-" then the field will be ignored.
func RowToAddrOfStructByNameLax ¶
func RowToAddrOfStructByNameLax[T any]() rowSpecRes[*T]
RowToAddrOfStructByNameLax scans a row into a *T. T must be a struct. T must have greater than or equal number of named public fields as row has fields. The row and T fields will be matched by name. The match is case-insensitive. The database column name can be overridden with a "db" struct tag. If the "db" struct tag is "-" then the field will be ignored.
func RowToAddrOfStructByPos ¶
func RowToAddrOfStructByPos[T any]() rowSpecRes[*T]
RowToStructByPos scans a row into a *T. T must be a struct. T must have the same number of public fields as row has fields. The row and T fields will be matched by position. If the "db" struct tag is "-" then the field will be ignored.
func RowToStructByName ¶
func RowToStructByName[T any]() rowSpecRes[T]
RowToStructByName scans a row into a T. T must be a struct. T must have the same number of named public fields as row has fields. The row and T fields will be matched by name. The match is case-insensitive. The database column name can be overridden with a "db" struct tag. If the "db" struct tag is "-" then the field will be ignored.
func RowToStructByNameLax ¶
func RowToStructByNameLax[T any]() rowSpecRes[T]
RowToStructByNameLax scans a row into a T. T must be a struct. T must have greater than or equal number of named public fields as row has fields. The row and T fields will be matched by name. The match is case-insensitive. The database column name can be overridden with a "db" struct tag. If the "db" struct tag is "-" then the field will be ignored.
func RowToStructByPos ¶
func RowToStructByPos[T any]() rowSpecRes[T]
RowToStructByPos scans a row into a T. T must be a struct. T must have the same number of public fields as row has fields. The row and T fields will be matched by position. If the "db" struct tag is "-" then the field will be ignored.
Types ¶
type RowSpec ¶
type RowSpec[T any] func() rowSpecRes[T]
RowSpec defines a specification for scanning rows into a given type.
Note on the weird type definitions: RowSpec returns a struct containing a private function pointer because:
- We want to be able to manage the lifecycle of the returned value inside the collection functions. (E.g. we may decide to pool scanners for reuse.) In order to do this safely, we need to ensure the Scanner returned by the inner function isn't referenced outside of the collecting function. This requires that we have a function returning a scanner.
- Returning a struct allows us to extend this value in the future if necessary. By comparison, returning a function would not, and would require a (technically) breaking change if the type needed to change in the future.
- Returning a non-exported type lets us hide as many details as possible from the public API and restrict the only valid usage to: pgx.CollectRows(rows, RowTo[Type])
- RowSpec is itself a function to provide a place to put the generic type parameter. rowSpecRes cannot be a constant, since then there would be no place to put the type parameter. Since rowSpecRes cannot be constructed in client code (by design) it can't be applied when creating a struct value.
func Adapt ¶
Adapt adapts a RowToFunc (the input to pgx.CollectRows, etc.) into a RowSpec.
This simplifies migration from pgx.CollectRows to pgx_collect.CollectRows by allowing code-bases with custom RowToFunc implementations to replace
pgx.CollectRows(rows, myRowToFunc)
with:
pgxc.CollectRows(rows, pgxc.Adapt(myRowToFunc))
This is only recommendation for custom implementations of RowToFunc during a migration process. Implementations of RowToFunc in pgx have native pgx_collect implementations that are more concise and efficient. Custom implementations can likely be refactored to work with RowToCustom which should be somewhat more efficient for queries over multiple rows.
type Scanner ¶
type Scanner[T any] interface { // Initialize sets up the Scanner and validates it against the rows. // Initialize must be called once before ScanRowInto. Initialize(rows pgx.Rows) error // ScanRowInto scans the row into the receiver. ScanRowInto(receiver *T, rows pgx.Rows) error }
Scanner provides an interface for scanning a row into a receiver value, while doing type-checking and initialization only once per query.
Source Files
Directories