konsume

konsume is a powerful and flexible tool designed to consume messages from various message queues like RabbitMQ and Kafka and perform data-driven actions like HTTP requests and insertions into databases based on predefined configurations.

Table of Contents

• Overview
• Features
• Installation
• Usage
• FAQ
• Contributing

Overview

TLDR; If you want to consume messages from message queues such as RabbitMQ or Kafka and perform HTTP requests or insert data to a database based on configurations, konsume is for you.

konsume is a tool that easily connects message queues like RabbitMQ and Kafka with web services, automating data-driven actions. It bridges complex messaging systems and web APIs or databases. Its flexible setup, including various retry options and customizable request formats, suits a range of uses, from basic data transfers to intricate processing tasks.

Features

• Message Consumption: Efficiently consumes messages from specified queues.
• Dynamic HTTP Requests: Sends HTTP requests based on message content and predefined configurations.
• Database Insertions: Inserts data into databases based on message content and predefined configurations.
• Retry Strategies: Supports fixed, exponential, and random retry strategies for handling request failures.
• Request Body Templating: Dynamically constructs request bodies using templates with values extracted from incoming messages.
• Custom HTTP Headers: Allows setting custom HTTP headers for outgoing requests.
• Configurable via YAML and JSON: Easy configuration using a YAML or JSON file for defining queues, routes, and behaviors.
• Monitoring: Provides a Prometheus endpoint for monitoring metrics.

Installation

Easiest way to install konsume is to run via Docker.

docker run -d --name konsume -v /path/to/config.yaml:/config/config.yaml bugrakocabay/konsume:latest

Alternatively, you can download the latest binary from the releases page and run it on your machine.

Usage

konsume depends on a YAML or JSON configuration file for defining queues, routes, and behaviors. By default, konsume looks for a file named config.yaml in the /config directory. Alternatively, you can specify the environment variable KONSUME_CONFIG_PATH to point to your configuration file.

📜 Checkout the CONFIGURATION.md for detailed information about the configuration parameters.
💡 You can also find detailed usage examples in the examples directory.

A simple usage for konsume to send HTTP requests with RabbitMQ:

providers:
  - name: 'rabbit-queue'
    type: 'rabbitmq'
    amqp-config:
      host: 'localhost'
      port: 5672
      username: 'user'
      password: 'password'
queues:
  - name: 'queue-for-rabbit'
    provider: 'rabbit-queue'
    routes:
      - name: 'ServiceA_Queue2'
        type: 'REST'
        method: 'POST'
        url: 'https://someurl.com'

A simple usage for konsume to insert into a database with Kafka:

providers:
  - name: 'kafka-queue'
    type: 'kafka'
    kafka-config:
      brokers:
        - 'localhost:9092'
      topic: 'your_topic_name'
      group: 'group1'
databases:
  - name: "sql-database"
    type: "postgresql"
    connection-string: "postgres://postgres:password@host:5432/dbname"
queues:
  - name: "queue-for-kafka"
    provider: "kafka-queue"
    database-routes:
      - name: "sql-database-route"
        provider: "sql-database"
        table: "some_table"
        mapping:
          userName: "name"
          some-key: "value"

FAQ

{
	"name": "John",
	"email": "john@doe.com"
}

routes:
  - name: 'test-route'
    method: 'POST'
    type: 'REST'
    headers:
      Content-Type: 'application/json'
    body:
      userName: '{{name}}'
      eMail: '{{email}}'
    url: 'http://someurl.com'

{
	"name": "John",
	"email": "john@doe.com"
}

queues:
  - name: "queue-for-kafka"
    provider: "kafka-queue"
    database-routes:
      - name: "sql-database-route"
        provider: "sql-database"
        table: "some_table"
        mapping:
          name: "user_name"
          email: "user_email"

routes:
  - name: 'test-route'
    method: 'POST'
    type: 'graphql'
    headers:
      Content-Type: 'application/json'
    body:
      mutation: |
        mutation {
          addUser(name: {{name1}}, email: {{email1}}) {
            id
            name
            email
          }
        }
    url: 'http://someurl:4000/graphql'

queues:
  - name: 'queue-for-rabbit'
    provider: 'rabbit-queue'
    retry:
      enabled: true
      strategy: 'fixed'
      max_retries: 5
      interval: 5s
      threshold_status: 500
    routes:
      - name: 'ServiceA_Queue2'
        type: 'REST'
        method: 'POST'
        url: 'https://someurl.com'

Contributing

🌟 Your contributions are welcome!

Whether you're looking to fix bugs, add new features, or improve documentation, your help is greatly appreciated. Contributing to konsume is not only a great way to enhance this tool, but it's also an excellent opportunity to get involved with a community of like-minded individuals.

Here's how you can contribute:

Report Issues: Found a bug or have a suggestion? Open an issue and let us know! Submit Pull Requests: Have a fix or a new feature? Submit a pull request! Feedback and Ideas: Share your thoughts and ideas on how we can improve konsume. Documentation: Help us improve or translate the documentation. Before contributing, please read our Contributing Guidelines for more information on how to get started.

💡 No contribution is too small – whether it's fixing typos, improving code readability, or updating documentation, all contributions are valuable and appreciated!

Join us in making konsume better for everyone! 🚀