Message brokers, like RabbitMQ, accept and forward messages between separate applications. Messaging is useful for decoupling applications. This means that different applications are connected through a message broker, but they don’t directly rely on one another. This post will provide an overview of messaging with a focus on AMQP and the publish-subscribe pattern. Another term for message broker is event bus.

Publish/Subscribe architecture pattern

The publish-subscribe (pub/sub) pattern is a messaging pattern where senders of messages (publishers) do not program the messages to be sent directly to specific receivers (subscribers). Instead, the programmer “publishes” messages (events) without any knowledge of the subscribers. Source: (Wikipedia)

Pub/sub is an asynchronous programming model. An app that publishes data can send a request for processing that data and go back to what it was doing without being concerned with how the data is processed. Another cool thing about messaging is that your app can talk to any other app irrespective of language. This means that a Ruby app can publish data while a Python app consumes that data, all through messaging.

Why messaging?

It started in the finance industry. In 1985, Goldman Sachs was the first customer to use the original message queuing software: Teknekron’s The Information Bus (TIB) created by Vivek Randaive, an MIT graduate. Before messaging, traders had to use a different terminal for each type of information they were monitoring. With messaging, traders only needed one terminal that could subscribe to multiple streams of information. Source: “Driving the Information Bus”, 2001, Harvard Business School

Later, IBM and Microsoft came out with messaging software. It was expensive. Startups typically could not afford it. At the same time, big companies had trouble trying to wire up different types of messaging software. Java Message Service (JMS) tried to solve this, but there were still limitations in trying to integrate multiple interfaces with different configuration rules. There was a need for a standard messaging protocol.

Finally, a messaging protocol!

In 2004, JPMorgan Chase created Advanced Message Queuing Protocol (AMQP), which was designed to be an open standard for messaging software. AMQP allowed companies to dynamically connect info from any publisher in real time to any subscribed consumer over a software bus. Today, messaging is popular in the financial services industry because they often build applications that consume high volumes of data that must be quickly routed to different applications.

With a new protocol on the scene, there was a need for a message broker. RabbitMQ, a popular open source message broker, was developed around the same time the first public draft of the AMQP specification was released. Source: RabbitMQ in Action.

How AMQP works at a high level

You know the client/server relationship? Well, forget about that! Let’s learn about the consumer/producer relationship.

Let’s talk about the components of AMQP:

• Producers
• Consumers
• Queues
• Messages
• Channels
• Exchanges
• Bindings

Producers create messages, labels them for routing, and publishes/sends them to a broker server (Rabbit MQ). This is like a person writing a letter, labeling it with an address, and putting it in the mailbox.

Consumers attach to a broker server (Rabbit MQ) and subscribe to a queue (more on that later). Apps can be both consumers and producers.

Messages are binary blobs of data that have 2 parts: a payload and a label. The payload is the data you want to transmit. Rabbit doesn’t care about the format of the data. For example, the data could be a JSON array. The label describes the payload and also determines who should get a copy of the message. AMQP only describes the message with a label and leaves it to RabbitMQ to send it to interested consumers based on that label. Once the consumer receives the message, it only has the payload. If the producer needs the receiver to know who sent the message, they need to include that as part of the message payload.

Queues A queue is like a named mailbox. When a message arrives in a particular queue, RabbitMQ sends it to one of the subscribed consumers.

Channels Before you can consume or publish to RabbitMQ, you have to connect to it. By connecting, your creating a TCP connection between your app and the Rabbit broker. Once the TCP connection is open, your app creates an AMQP channel. Every channel has a unique ID. Whether publishing a message, subscribing to a queue, or receiving a message, it’s all done over a channel.

Exchanges are where producers publish their messages. When you want to send a message to a queue, you do it by sending it to an exchange. RabbitMQ will decide which queue it will deliver the message to based on certain rules called routing keys. If a queue is a named mailbox, the exchange is like a neighborhood of homes with all of the named mailboxes. There are different types of exchanges: Direct, Topic, Fanout, and Headers. You can read about them here

Bindings are how the messages get routed from the exchange to particular queues. A queue is bound to an exchange by a routing key.

FAQs

• Why not issue AMQP commands over the TCP connection? Setting up and tearing down TCP sessions is expensive for an operating system.

• What if a message arrives at a queue with no subscribed consumers? In that case, the message waits in the queue until a consumer subscribes to the queue.

• How are messages in a queue distributed when multiple consumers are subscribed to the same queue? In a round robin fashion.

• What happens to messages once they are distributed? Every message received by a consumer is required to be acknowledged. Message acknowledgments tell RabbitMQ that the consumer received the message and that it can safely be removed from the queue.

• What happens if a consumer receives a message and then disconnects from Rabbit/unsubscribes from the queue before the message is acknowledged? RabbitMQ will consider the message undelivered and redeliver it to the next subscribed consumer. If your app crashes, the message will be sent to another consumer for processing

• What happens if the consuming app has a bug and forgets to acknowledge a message? RabbitMQ won’t send the consumer any more messages. RabbitMQ thinks that if the last message wasn’t acknowledged, the consumer must not be ready to receive more messages.

• Does RabbitMQ support any protocols beyond AMQP? Yes.