ppMQ

ppMQ is persistent, prioritized message queue with dedupe & expire functionality

ppMQ is very opinionated and was made to solve a message routing problem in a single server environment with high memory pressure. The environment required:

• Getting messages from multiple sources
• Sending messages to multiple processes (subscriber) using unix sockets (one per subscriber). We expect that process might not be running, and will be started by systemd when ppMQ tries to send a message through the socket.
• ppMQ will receive Acknowledgements though its own unix socket (so, all communications are uni-directional).

ppMQ also has following features beyond simple FIFO

• Multiple topics, each can have different properties
• Message expiration (if a message is not delivered by some deadline, it is thrown away)
• Multiple priorities supported (priority 0 is top priority)
• Dedupe (if two messages have the same key, the oldest message is replaced with newest, and only newest is delivered)
• ACK required, subscriber can send "RECONNECT" command, causing all messages to that weren't acknowledged yet to be redelivered
• ACK expiration, if message delivery is not acknowledge until the deadline, message is resent
• Multiple subscribers per topic

The ppMQ is configured using yaml file. As we deliver new subscribers, that yaml file will change. Restart is required to reload the changes.

Messages sent to ppMQ are encoded in JSON (we can add other encodings later on), so app in any language can connect.

{
"Key": "message key", // used for dedupe purposes only, can be empty
"Payload": "message payload", // the actual payload of the message
"Expires": "expr time", // defined as time.Time, can be empty
"Priority": 0 // can be empty, default & highest priority - 0
}

Defined as struct Message in internal/ppmq/structs.go

They are sent to ppMQ an array in following JSON structure:

{
"Topic": "message topic", 
"Messages": [ {"Key": "A", "Payload": "XYZ"}, /* etc */ ] // array of messages
}

ppMQ sends them to subscribers using JSON structure:

{
"Topic": "message topic", // needed as part as acknowledgement response
"Subscriber": "subscriber name", // needed as part as acknowledgement response
"Messages": [ {"Key": "A", "Payload": "XYZ"}, /* etc */ ] // array of messages - note, even if original message 
//didn't have key, this one will, autogenerated.
}

Subscriber will send to ppMQ ACK using following structure:

{
"Topic": "message topic", 
"Subscriber": "subscriber name", 
"Acks": ["key1", "key2", "key3", "..."] // array of acknowledgements
}

Subscriber might also send to ppMQ command to resend messages that hasn't been ACKed.

{
"Topic": "message topic", 
"Subscriber": "subscriber name", 
"Command": "RECONNECT"
}

Defined as struct MQWrapper in internal/ppmq/structs.go