fluent-bit-pubsub-plugin

command module

v0.0.1 Latest Latest Go to latest Published: Aug 29, 2022 License: Apache-2.0 Imports: 14 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This plugin allows publishing log records from fluent-bit to Google Cloud PubSub.

fluent-bit -e /path/to/flb_pubsub.so -c fluent-bit.conf

 [OUTPUT]
 match *
 name pubsub
 gcp_project_id noponies4u_logs
 topic_id fluent_logs
 credentials_file /etc/fluent-bit/gcs.json

Option Name	Description	Type	Default	Example
gcp_project_id	Google Cloud project id	string	None	my_gcp_project
topic_id	PubSub topic ID	string	None	fluentbit_logs
credentials_file	Path to service account credentials file.	string	None	/etc/fluent-bit/gcloud.json
timestamp_field	Log record field to populate/update with the fluent-bit timestamp	string	None	fb_ts
attribute_fields	Comma seperated list of fields to use as PubSub message attributes. These are useful since subscribers can filter messages by attributes, but not body content.	comma seperated strings	None	loghost,tag,app
keep_attribute_fields	If set to true, record fields used as attributes are also left in the log record. Otherwise, they are removed.	boolean	false	true
publish_timeout	Timeout to use on the PubSub publisher client.	Duration	60s	2m

Indicates required field

If a credentials file isn't provided, the library attempts to use Application Default Credentials

These correspond to PublishSettings.

Option Name	Description	Type	Default
publish_delay_threshold	Publish a non-empty batch after this many seconds has passed	Duration	1s
publish_byte_threshold	Publish a batch once it reaches this size in bytes.	int	1,000,000
publish_count_threshold	Publish a batch once it has this many messages.	int	100

go build -buildmode=c-shared -o flb_pubsub.so .

Install MSYS2 or another distribution such as Cygwin with mingw-gcc
Install appropriate gcc package (32bit or 64bit depending on what fluent-bit package you have installed)
Add MSYS2 bin directory with the appropriate compiler to your path (e.g. C:\MSYS2\mingw64\bin)
go build -buildmode=c-shared -o flb_pubsub.dll .

There is no documentation for this package.

Path	Synopsis
plugin