rxtx
rxtx is a queue based data collector > data transmitter. Useful for online/offline data collection, back pressure buffering or general queuing. rxtx uses bbolt maintained by CoreOs, a single file database for storing messages before they can be sent.
rtbeat was developed to consume rxtx POST data and publish as events into elasticsearch, logstash, kafka, redis or directly to log files.
Test on MacOs
Install with brew
brew tap txn2/homebrew-tap
brew install rxtx
Help
rxtx -h
Test Docker Container
Help
docker run --rm -it txn2/rxtx -h
on arm 6/7 based device:
docker run --rm -it txn2/rxtx:arm32v6-1.2.0 -h
Test Source
Help
go run ./rxtx.go -h
Usage of rxtx:
-batch int
Batch size. (default 5000)
-ingest string
Ingest server. (default "http://localhost:8081/in")
-interval int
Seconds between intervals. (default 30)
-maxq int
Max number of message in queue. (default 2000000)
-name string
Service name. (default "rxtx")
-path string
Directory to store database. (default "./")
-port string
Server port. (default "8080")
Start server on 8080
go run ./rxtx.go
Add message to queue
The rxtx services accepts HTTP POST data to an API endpoint in the following form /rx/PRODUCER/KEY/LABEL/.../. One label is required, however as many labels as necessary may be added, separated by a forward slash.
curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data
Add message to queue every second
while true; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; sleep 1; done
Add 1000 messages to the queue.
time for i in {1..1000}; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; done
Building and Releasing
rxtx uses GORELEASER to build binaries and Docker containers.
Test Release Steps
Install GORELEASER with brew (MacOS):
brew install goreleaser/tap/goreleaser
Build without releasing:
goreleaser --skip-publish --rm-dist --skip-validate
Release Steps
- Commit latest changes
- Tag a version
git tag -a v2.0 -m "Version 2.0"
- Push tag
git push origin v2.0
- Run:
GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist
Resources
License