Modern computing relies on a distributed architecture in which separate software components—often in different environments—coordinate with each other. Messaging and event streaming are essential to support microservices, serverless, and other cloud native application architectures.
- One of the programming challenges created by microservices and other distributed architectures is communication between application components. Message brokers are designed to provide a common communication mechanism that scales with application needs. In addition to messaging, message brokers typically provide services such as translation, routing, persistence, and delivery, streamlining communications while reducing the burden on software developers to handle these capabilities in application code.
- An enterprise-grade message broker helps to ensure fast reliable communication between servers, applications and devices in multi-cloud environments, with reliable asynchronous messaging for passing events, commands, data and responses among producing and consuming applications or services. A distributed message broker simplifies communication between microservices, while providing security, automated management, and high performance.
- Distributed microservices applications communicate with each other by exchanging data in the form of messages. Message brokers can help integrate disparate applications and services across a wide variety of languages and platforms. Asynchronous messaging allows producers and consumers to send and receive messages independently and at different times without blocking.
A message broker provides the necessary plumbing to facilitate the exchange of messages (discrete, structured data). A producer or publisher sends the message, and a consumer or subscriber receives the message. A message broker accepts and if necessary, translates messages sent by a producer, then organizes and holds messages in a message queue until received by the consumer(s).
Once a message is received, a message broker typically maintains the message in a message queue until a consumer/subscriber can retrieve it. Message queues are data structures that provide reliable temporary storage for messages until they are read. They store messages in the same order in which they were sent, so they can be processed in sequence. Producers and consumers can share the same queue, and messages can be routed to one or many message queues. Message queues help ensure information doesn’t get lost or destroyed due to incompatible systems, network latency, or application failure. Because the message queue is such an important component of a message broker, the designation “MQ,” (message queue) appears in the name of message brokers such as RabbitMQ and ActiveMQ.
- While many use message broker and event streaming somewhat interchangeably, they are not exactly the same. The confusion arises because of overlap in the use cases for the two technologies.
- A message broker stores a message in a queue until it is consumed and then deletes the message. Message brokers offer flexible routing and reliability. Event streaming platforms append events to the end of a log and don’t delete events as they are consumed.
- Streaming facilitates raw throughput. Consumers can start reading the stream at any point and see the entire history. An “event” could be the same as a message, but with streaming applications, it is often the stream of events that is of interest. Multiple events are often processed together.
- For example, a low-fidelity temperature sensor might report the temperature once per second. Rather than use the temperature reported in a single event, it may be more accurate to average the last ten temperature readings and use the average as a measure of the temperature. Similarly, a stock trading application might analyze changes in a stock price over time, processing a series of events (multiple ticks in the price) in multiple ways before taking an action or generating a notification.
- The blog RabbitMQ vs Kafka: How to Choose an Event-Streaming Broker sheds more light on the differences between a message broker and event streaming.
- A message bus—or enterprise service bus (ESB)—is a type of middleware built on a central system that enables application-to-application communication. Primarily used in old-style Service Oriented Architecture (SOA), ESBs formalize data formats and protocols for use by multiple applications.
- SOA and ESB technologies remain in wide use but are being replaced by messaging and streaming in modern microservices applications. More and more organizations are migrating to open source message brokers and embracing microservices architectures and cloud native frameworks.
The following terms may be employed depending on the messaging model being employed:
- Producer or publisher is the message sender
- Consumer or subscriber is the message recipient
- Message queue or topic is where a message is routed to
A point-to-point messaging model is used when a producer sends a single message directly (point-to-point) to a consumer. Messages could include data to be stored, a command to be carried out, or any other information that needs to be shared between the producer and the consumer.
When a message producer needs to send the same message to many recipients simultaneously, then a publish-subscribe model is used. With this messaging pattern, the message broker publishes a message to a topic, where it is organized, cached, processed and delivered to subscribers. One or many publishers can publish on the same topic, and messages are available to anyone who subscribes to that topic. Publish-subscribe provides a flexible way of communicating between multiple microservices and applications.
Applications often have trouble communicating with other applications or services deployed in different environments. In an increasingly distributed environment, a message broker helps communicate and coordinate with services running on multiple platforms. The message broker you choose depends on your use case and operations.
Many microservices applications use APIs to communicate directly with each other, so you might wonder why a message broker is needed. One of the main reasons to choose a message broker is scale. Once application components begin exchanging thousands of messages, the performance and resiliency of a message broker begin to justify the complexity that comes with the extra software.
There are a number of common use cases that lend themselves to message broker use:
- Microservices: As the number of microservices that make up an application increases, a message broker can rationalize communications between services. For example, suppose a service
needs to send the identical message to ten other services. Rather than make 10 API calls, with a message broker you simply send one message to the broker.
- Long-running tasks: Sometimes a message triggers a long-running task. For instance, when a video is uploaded, a message may be sent to trigger a series of tasks such as transcoding, compression, distribution, etc.These tasks may take many hours to run in the background.
- Mobile apps: Mobile devices and apps may not always be running or reachable. A message broker can ensure that messages (notifications) are available and can be received in order when the mobile device or application regains service.
- Transactions: With transactional systems, you often need to ensure that a message is only processed once and not duplicated. Because message brokers delete messages after they are consumed, they can be used to ensure this happens.
There are many open source and cloud-based message brokers available.
- RabbitMQ is an open source distributed message broker that facilitates the efficient delivery of messages in complex routing scenarios. It delivers flexible routing and command and response between applications. RabbitMQ has recently added event-streaming capabilities that model a Kafka-like append-only log on top of a traditional RabbitMQ message queue.
- VMware RabbitMQ is an enterprise grade message broker with event streaming capabilities based on of open source RabbitMQ.
- Apache ActiveMQ is an open source message broker that facilitates communication between microservices, and supports protocols across a range of languages and platforms.
- Apache Kafka describes itself as a distributed event streaming platform rather than a message broker. However, many use Kafka to satisfy message broker use cases. Kafka’s scalability enables applications to publish and subscribe to multiple message streams simultaneously. Kafka uses a distributed append-only log; messages are written to the end of a log that’s persisted to disk.
- Apache Pulsar provides cloud native, multi-tenant, high-performance messaging and queuing for microservices and serverless functions.
A message broker facilitates the exchange of messages between a producer/publisher, who sends the message and a consumer/subscriber, who receives the message.
Message brokers temporarily store messages in a data structure called a message queue (sometimes referred to as a topic) until a consumer/subscriber can retrieve them.
A point-to-point messaging model is used when a producer sends a single message directly, i.e., point-to-point to another service and waits for a response from the consumer, i.e., the recipient of the message.
A publish-subscribe messaging model is used when a message needs to be sent to many recipients simultaneously. A message broker publishes the message to a topic, then distributes it to everyone who subscribes to that topic.
RabbitMQ is an open source distributed message broker with event-streaming capabilities that delivers flexible routing and command and response between applications. RabbitMQ and Apache Kafka are the two most popular message brokers in use.