pg_webhook

Postgres Webhook

Run webhooks directly from your Postgres database using replication.
Explore the docs »

View Demo

About The Project

This project came about from MovingLake technological development. Webhooks directly from a database using a replication slot was one of our core desires since before we founded MovingLake. This is now a reality and we wanted to give back to the awesome Open-Source Community by creating this Repo. Hope you enjoy it!

(back to top)

Built With

(back to top)

Postgres Setup

Pg_webhook uses a logical replication slot to get real-time data from the main database. As such a few configuration steps must be followed to allow this to happen

Standalone Postgres

Use the following commands to get started. We'll create a database, a user and setup postgres to create a logical replication slot.

create database pg_webhook;
create user pg_webhook with replication password 'pg_webhook';

Next you need to add some lines to Postgres' configuration files. If you do not know where this files are, you can run ps -ef | grep postgres. You shoud see a line such as:

/opt/homebrew/opt/postgresql/bin/postgres -D /opt/homebrew/var/postgres

The last piece of this line (/opt/homebrew/var/postgres) is where your postgres configuration files will be stored. Now open pg_hba.conf and add the following line:

host replication pg_webhook 127.0.0.1/32 md5

Next open postgres.conf and add these following lines:

wal_level=logical
max_wal_senders=5
max_replication_slots=5

Finally when specifying the postgres DNS string when running pg_webhook, make sure it has the replication query parameter ?replication=database. Eg postgres://pg_webhook:pg_webhook@localhost:5432/pg_webhook?replication=database

RDS Postgres

RDS does not let you run with a real superuser, and also doesn't let you change the configuration files. Most likely because of multitenant systems. To circumvent this, the easiest way to go is to use Postgres extension pglogical.

Please follow this link to set-up your RDS instance with the extension. Also checkout this other link if don't know how to turn on the preloaded libraries for RDS.

Once you follow these steps, try and follow the rest from #standalone-postgres

Getting Started

Docker

The easiest way to get started is to build the Dockerfile

docker build -t latest .

and then run the image with the appropriate environment variables

docker run --rm -it --net=host -e PG_DNS='postgres://postgres@localhost:5432/postgres?replication=database' -e WEBHOOK_SERVICE_URL='http://localhost:9000/' latest

Compile and run

You can also run directly the extractor, just use

go run main.go

Do ensure that the environment variables are set beforehand.

System

This extractor runs on Golang. It uses the pglogrepl library to create the replication slots and subscription. It then uses a producer consumer pattern to send received messages to the downstream service. If the downstream service cannot receive the requests, this code retries automatically for upto a day. After that it logs the failed requests into a Sqlite database on disk.

Consumer concurrency can be set higher if a high volume of transactions are being sent to the database.

License

Distributed under the GNU AGPL 3 License. See LICENSE for more information.

(back to top)

Contact

MovingLake - @MovingLake - info@movinglake.com

Project Link: https://github.com/movinglake/pg_webhook

(back to top)

Managed Version

If you ever want to have a managed version of this system built for scale and completely maintenance free then we'd be happy to accommodate your requests. Please use this link to create a Postgres connector with a webhook destination.

(back to top)

Acknowledgments

Want to acknowledge the pglogrepl repo and its creators for such a great tool!

• pglogrepl

(back to top)