RabbitMQ is an event bus. In other words, it’s a message broker. Simply put, RabbitMQ accepts and forwards messages between separate applications. Messaging is useful for decoupling applications. This means that different applications are connected through a message broker, but they don’t directly rely on one another. Other open source message brokers include ZeroMQ and qpid. This post will provide an overview of messaging with a focus on RabbitMQ, AMQP, and the pub/sub pattern.
Publish/Subscribe architecture pattern
The publish-subscribe (pub/sub) pattern is a messaging pattern where senders of messages (publishers), do not program the messages to be sent directly to specific receivers (subscribers). Instead, the programmer “publishes” messages (events), without any knowledge of the subscribers. Source: (Wikipedia)
This decoupling allows things to move away from a synchronous programming model to an asynchronous one. An app that publishes data can fire off a request for processing that data and go back to what is what doing without being concerned with how the data is processed. Another cool thing about messaging is that your app can talk to any other app irrespective of language. This means that a Ruby app can publish data while a Python app consumes that data, all through messaging.
In 1985, Goldman Sachs was the first customer to use the original message queuing software: Teknekron’s The Information Bus (TIB) created by Vivek Randaive, an MIT graduate. Before messaging, traders had to use a different terminal for each type of information they were monitoring. With messaging, traders only needed one terminal that could subscribe to multiple streams of information Source: “Driving the Information Bus”, 2001, Harvard Business School
Later, IBM and Microsoft came out with messaging software. It was expensive. It was hard for startups to use because they couldn’t afford it. At the same time, big companies had trouble trying to wire up different types of messaging software. Java Message Service (JMS) tried to solve this, but there were still limitations in trying to integrate multiple interfaces with different configuration rules. There was a need for a standard messaging protocol.
In 2004, JPMorgan Chase created Advanced Message Queuing Protocol (AMQP), which was designed to be an open standard for messaging software. AMQP allowed companies to dynamically connect info from any publisher in real time to any subscribed consumer over a software bus. Today, messaging is popular in the financial services industry because they often build applications that consume high volumes of data that must be quickly routed to different applications. It’s also used at startups.
Around 2004, AMQP was created, but there weren’t any open source message brokers available. RabbitMQ was developed around the same time the first public draft of the AMQP specification was released. Source: RabbitMQ in Action.
How AMQP works at a high level
You know the client/server relationship? Well, forget about that! Let’s learn about the consumer/producer relationship. Keep in mind, RabbitMQ is like a delivery service. A middle man, if you will.
Let’s talk about the components of AMQP:
Producers create messages, labels them for routing, and publishes/sends them to a broker server (Rabbit MQ). This is like a person writing a letter, labeling it with an address, and putting it in the mailbox.
Consumers attach to a broker server (Rabbit MQ) and subscribe to a queue (more on that later). Apps can be both consumers and producers.
Messages are binary blobs of data that have 2 parts: a payload and a label. The payload is the data you want to transmit. Rabbit doesn’t care about the format of the data. For example, the data could be a JSON array. The label describes the payload and also determines who should get a copy of the message. AMQP only describes the message with a label and leaves it to RabbitMQ to send it to interested consumers based on that label. Once the consumer receives the message, it only has the payload. The labels don’t get passed a long with the payload when the messages are routed. If the producer needs the receiver to know who sent the message, they need to include that as part of the message payload.
Queues A queue is like a named mailbox. When a message arrives in a particular queue/mailbox, RabbitMQ sends it to one of the subscribed/listening consumers.
Channels Before you can consume or publish to RabbitMQ, you have to connect to it. By connecting, your creating a TCP connection between your app and the Rabbit broker. Once the TCP connection is open, your app creates an AMQP channel. Every channel has a unique ID. Whether publishing a message, subscribing to a queue, or receiving a message, it’s all done over a channel.
Exchanges are where producers publish their messages. When you want to send a message to a queue, you do it by sending it to an exchange. RabbitMQ will decide to which queue it will deliver the message to based on certain rules called routing keys. If a queue is a named mailbox, an exchange is the act of taking an addressed letter and walking it toward the mailbox that corresponds to the correct address on the letter. There are different types of exchanges: Direct, Topic, Fanout, and Headers.
Bindings are how the messages get routed from the exchange to particular queues. A queue is bound to an exchange by a routing key.
Why not issue AMQP commands over the TCP connection? Setting up and tearing down TCP sessions is expensive for an operating system.
What if a message arrives at a queue with no subscribed consumers? In that case, the message waits in the queue until a consumer subscribes to the queue.
How are messages in a queue distributed when multiple consumers are subscribed to the same queue? In a round robin fashion. I can’t help but imagining Oprah saying “YOU get a message! YOU get a message! YOU get a message!”
What happens to messages once they are distributed? Every message received by a consumer is required to be acknowledged. Message acknowledgments tell RabbitMQ that the consumer received the message and that is can safely be removed from the queue.
What happens if a consumer receives a message and then disconnects from Rabbit/unsubscribes from the queue before the message is acknowledged? RabbitMQ will consider the message undelivered and redeliver it to the next subscribed consumer. If your app crashes, the message will be sent to another consumer for processing
What happens if the consuming app has a bug and forgets to acknowledge a message? RabbitMQ won’t send the consumer any more messages. RabbitMQ thinks that if the last message wasn’t acknowledged, the consumer must not be ready to receive more messages.
Does RabbitMQ support any protocols beyond AMQP? Yes.