The Competing Queue Consumer Patterns enables the processing of messages concurrently by multiple consumers. This will improve resiliency, reliability, scalability, high availability, and traffic distribution.
Use case
Let’s consider that we have one publisher and one consumer publishing and subscribing to the message to/from the queue. In this case, consumers have to consume a limited number of messages from the queue.
In the case above, the publisher is sending a limited number of messages to the queue and the consumer is able to consume and process the message at the speed sent by the publisher.
There might be possibilities that the publisher is sending the message at a high rate or increasing the number of publishers. Due to this, there are more messages coming to the queue, and a single consumer cannot consume and process the message at the rate sent to the queue by publishers.
In such cases, we can increase the number of consumers and this will ensure that messages can be processed faster and reliably. There will be multiple consumer instances listening to the same queue, and messages will be delivered to consumers round robin. This will also ensure that one message is delivered to one consumer even if there are multiple consumers listening to the same queue.
High Availability
In Competing Queue Consumer Patterns, there will be multiple consumers competing for receiving the messages from the queue. In case, one of the consumers crashes or shuts down or goes into offline mode, there are other consumers that can consume and process the messages.
Scalability
In these patterns, the consumer pool can be scaled up or scaled down depending on traffic load. We can easily achieve horizontal scaling by increasing or decreasing the number of consumers.
Reliability and Resilience
In case message processing fails for one of the messages, the messages will be returned to the queue, so they can be reprocessed by other consumers.
When to use Competing Queue Consumer Pattern?
- When you don’t have a requirement of processing the messages in the sequence in which they have been sent to the queue and you have a requirement to process the high volume of messages quickly.
- When your application workload is divided into multiple processes and can run in concurrent mode and asynchronously.
- When there is a need for high availability and reliability of the messages.
How to implement Competing Queue Consumer Patterns in Anypoint Clustering?
Anypoint Clustering is a group of servers or nodes or mule runtime that can act as a single unit.
Application Deployed on a Cluster With a Queue Listener and make sure attributes primaryNodeOnly=false.
In JMS connector configuration, users can configure attribute primaryNodeOnly to define whether a message will be only picked by the primary node or any node in the cluster. By default, the primaryNodeOnly attribute is false. In other words:
- If the primaryNodeOnly attribute is false, the clustering works in active-active mode.
- If the primaryNodeOnly attribute is true, then clustering works in active-passive mode.
| <jms:listener doc:name=”On New Message” doc:id=”39714fd4-8ac7-4a09-a3f8-111b982758a8″ config-ref=”JMS_Config” destination=”TestQueue” numberOfConsumers=”1″ onlyPrimaryNode=”false”> <jms:consumer-type > <jms:queue-consumer /> </jms:consumer-type> </jms:listener> |
In the diagram above, it is showing the message is sent to multiple consumers in a round robin way because in JMS listener configuration onlyPrimaryNode is false. It means all nodes can actively listen to the Queue and process the messages concurrently.
How to implement Competing Queue Consumer Patterns in CloudHub?
In CloudHub, for implementing Competing Queue Consumer patterns, deploy the application with Queue Listener on more than one worker. Applications can easily scale up or scale down by increasing or decreasing the number of workers for the applications.
Conclusion
MuleSoft provides options to implement the competing queue consumer patterns in the CloudHub as well as in customer-hosted environments. This is a very important pattern for enabling high availability, scalability, and resilience in Event-Driven Architecture.
Need help?
As a premium MuleSoft partner, we can help you with implementation. Do you have any questions or would you like to receive more information about our Integration & API services? Don’t hesitate and reach out to us.
Devoteam’s Integration consulting services
Building on 20+ years of extensive Integration experience at both multinationals and SMEs, we support our customers with Integration technology services. Our experts cover the full spectrum of skills, experiences, and proven approaches needed for the Integration of processes regarding customers, services, products, and operating models. In short, we enable seamless ecosystem connectivity for your entire value chain. We enable organizations to easily connect with their prospects, customers, suppliers & partners by:
- Setting up (hybrid) Integration platforms, from architecture to configuration
- Taking care of your Integration environment through a managed services model with different support levels
- APIs & API Management
- Electronic Data Interchange (EDI)
