Discover Packages
github.com/pg2es/search-replica
command
module
Version:
v0.2.1
Opens a new window with list of versions in this module.
Published: Dec 14, 2022
License: MIT
Opens a new window with license information.
Imports: 18
Opens a new window with list of imports.
Imported by: 0
Opens a new window with list of known importers.
README
README
¶
Search Replica
Uses postgres logical replication protocol to subscribe and receive any updates almost in realtime.
Stateless app with no external dependencies
Table rows, regardless of types, are encoded into JSON with additional ES metadata, which makes a Message (ES Document).
Those messages are buffered and flushed into ES regularly, on a timeout or when buffer is already large enough.
Usage E.G:
go run *.go --help
PG_PUBLICATION="pub1" go run *.go -slot qwerty -reindex
Roadmap
Testing HELP!
Composite arrays (Too complex; PRs are welcome)
Composite PK / Template fields
Live-reload
Recovery (Reindex when slot is lost)
HealthCheck & Readiness
REST API with basic actions
Improve auto discovery, for inlines/join relations
Helm / K8S setup.
"Strict mode"
Env Config
Variable
Default
Description
PG_SLOT
pg2es
replication slot name
PG_PUBLICATION
search
publication name
PGHOST
localhost
PGPORT
5432
PGDATABASE
-
PGUSER
-
PGPASSWORD
-
SEARCH_HOST
-
URL or host of ElasticSearch/OpenSearch
SEARCH_USERNAME
-
optional
SEARCH_PASSWORD
-
optional
SEARCH_BULK_SIZE
4
(MB) Bulk request size limit.
SEARCH_PUSH_INTERVAL
30s
idle push interval, when there is no enough rows for full bulk request.
LOG_FORMAT
json
json or cli
LOG_LEVEL
warn
from debug to fatal
Notes
The script is single threaded* (not a bottleneck) ... Separate goroutine is used to make ES requests.
Links between Database <-> Schema <-> Table <-> Column, shoudld be considered read only, and safe for multithread use... (not yet)
It's fast. All the the efforts shuld be towards readability, reliability and functionality.
Known Limitations:
Inlines can not update PK/ID/Routing fields (yet)
No 1:1 inlines (yet)
Delete document deletes all inlines (AKA DELETE CASCADE), and they can not be restored.
Expand ▾
Collapse ▴
Documentation
¶
package main ...
_______ _______ _ _______ _______
( ____ )( ____ \ ( \ ( ____ \( ____ \
| ( )|| ( \/ \ \ | ( \/| ( \/
| (____)|| | \ \ | (__ | (_____
| _____)| | ____ ) ) | __) (_____ )
| ( | | \_ ) / / | ( ) |
| ) | (___) | / / | (____/\/\____) |
|/ (_______) (_/ (_______/\_______)
Source Files
¶
Directories
¶
package conftags implements Go struct tag like syntax parser.
package conftags implements Go struct tag like syntax parser.
pgcopy
package pgcopy - parser for PostgreSQL `COPY TO ...
package pgcopy - parser for PostgreSQL `COPY TO ...
Click to show internal directories.
Click to hide internal directories.