Menu

This Month in RabbitMQ: November 2019

November 13th, 2019 by Michael Klishin

This Month (and the month before) in RabbitMQ — October and September recap!

We’re a little behind this month! At the beginning of October, we shipped RabbitMQ 3.8. That’s right, folks, RabbitMQ 3.8 is finally out!

Headline features include:

You’ll find some early reviews from folks in the community who have been kicking the tires in the community updates section below. Make sure you are all over the upgrades best practices to avoid potential hazards of upgrading to RabbitMQ 3.8.

Oh, and there were some other rather meaningful ecosystem announcements out there:

SpringOne Platform 2019 talks that highlighted RabbitMQ:

Project updates

Several updates to 3.7.x with bug fixes:

Community Writings and Resources

Webinars and Training

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ

RabbitMQ 3.8 Release Overview

November 11th, 2019 by admin

RabbitMQ 3.8 has just been released and has some major new features which focus on reliability, operations, and observability.

You can find the new 3.8 release on the GitHub releases page which includes information about what is included in the release as well as various installation assets. See our upgrade guide for more information about upgrading to 3.8.0.

Our team dedicates this release to Joe Armstrong, the creator of Erlang. Joe’s work in the fields of concurrent and distributed systems benefits RabbitMQ to this day. Equally importantly, Joe was a rare example of a brilliant engineer who was also very humble and kind.

Let’s take a quick look at the new features in this release.

Quorum Queues

For years, RabbitMQ has offered mirrored queues, also known as HA queues, as a solution for both high availability and data safety. Messages are replicated from a queue master to one or more mirrors so that in the event of the loss of a broker, a mirror can be promoted to master and the queue continues to be available without loss of confirmed messages.

Quorum queues are the next generation of replicated queue and offer both better performance and solve some of the pain points of mirrored queues. Quorum queues use the well established Raft protocol which has now been implemented in countless data systems as a means of achieving reliable and fault tolerant state replication.

Shows a quorum queue consisting of one leader and two followers

One of the main pain points around mirrored queues was blocking synchronization coupled with throwing away data on leaving and rejoining a cluster. This made applying OS patches difficult if queues were large in size as the administrator was forced to choose between lower redundancy or a period of unavailability. Quorum queues completely avoid this issue by not throwing away data and making replication to a single node non blocking. Quorum queues also avoid split-brain scenarios that could provoke message loss and always favour consistency over availability.

From now on we will be referring to classic and quorum queues.

Read more about quorum queues in documentation guides.

Feature Flags

Prior to the new feature flag sub-system, upgrades to RabbitMQ required cluster-wide shutdown. Feature flags allow for rolling upgrades of a cluster enabling continued availability.

As Jean-Sébastien Pédron described in  this blog: “New in RabbitMQ 3.8.0 is the feature flags subsystem: when a single node in a 3.7.x cluster is upgraded to 3.8.0 and restarted, it will not immediately enable the new features or migrate its database schema because the feature flags subsystem told it not to. It could determine this because RabbitMQ 3.7.x supports no feature flags at all, therefore new features or behaviours in RabbitMQ 3.8.0 cannot be used before all nodes in the cluster are upgraded.”

Multi-step process of upgrading and enabling feature flags

Read more about feature flags in documentation guides.

Prometheus and Grafana Monitoring Support

Many systems come with their own custom monitoring visualization solution, the Management Plugin has has been this solution in RabbitMQ for years. The new paradigm is for applications and infrastructure to expose metrics to external observability platforms and delegate the storing, indexing and alerting to those specialized tools. Both Prometheus and Grafana have become an industry standard in the systems observability space and provide powerful visualization and exploratory data analysis capabilities.

RabbitMQ 3.8 comes with new support for exposing its metrics via a Prometheus endpoint. Additionally, many more metrics are now available, vastly improving the overall observability of RabbitMQ. Visualizing these metrics is now as simple as importing pre-built dashboards into Grafana.

The RabitMQ overview Grafana dashboard

Prometheus and Grafana support has a dedicated documentation guide.

OAuth 2.0 Support

RabbitMQ 3.8 allows clients to use JWT access tokens for authentication and authorization. Clients obtain an access token from an OAuth2.0 provider, through any grant type they wish, and use that token to gain access to RabbitMQ. OAuth 2.0 tokens use scopes to communicate what set of permissions a particular client has been granted and RabbitMQ permissions are mapped onto these scopes.

Read more about OAuth2.0 support in the docs.

Additional CLI Tools

You can perform various levels of health checks with the rabbitmq-diagostics CLI tool. The checks range from basic pings to checking queues and vhosts are running to in-depth runtime information.

We have a new CLI tool, rabbitmq-queues, which gives us the ability to modify quorum queue memberships but also gives us new master/leader rebalancing functionality for both quorum and mirrored queues.

One of the pain points of performing a rolling upgrade to the servers of a RabbitMQ cluster was that queue masters would end up concentrated on one or two servers. The new rebalance command will automatically rebalance masters across the cluster. 

rabbitmq-queues has a man page.

Single Active Consumer (SAC)

SAC is also the next generation of an existing feature - exclusive consumers. The objective of exclusive consumers is to ensure that only a single consumer can consume a given queue at a time. The consumer uses the “exclusive” flag when registering itself, and the registration only succeeds if no other consumer is already registered.

SAC improves on this by making exclusivity a feature of the queue itself and making the process transparent to clients. If a second consumer registers itself, the registration succeeds and the consumer sits idle ready to become active if the currently active consumer shuts down or crashes. This gives us an automatic active-backup consumer strategy for when we want only a single consumer, but a secondary to take over quickly in the event the active goes away.

Shows two consumers on a queue where only one is active

Read more about Single Active Consumer in the Consumers guide.

And…

  • Messages can now be deadlettered from the tail of a classic queue with the new queue overflow configuration reject-publish-dlx.
  • High queue creation/deletion rates (queue churn) are now less costly.
  • Maximum message size is now configurable.
  • Quorum queues come with a new poison message feature that allows you to configure messages to be dropped after a given number of redeliveries by setting the delivery-limit policy.
  • RabbitMQ for Kubernetes is coming. Sign up to the beta.
 

If you’re more of a classroom learner, I recommend watching the webinar, “What's new in RabbitMQ 3.8?

Please give 3.8 a try and let us know what you think on the RabbitMQ mailing list!

This Month in RabbitMQ — September 2019

September 9th, 2019 by Michael Klishin

Welcome back for another edition of This Month in RabbitMQ! Exciting news is that the first release candidate for RabbitMQ 3.8 is now available!

Be sure to catch up on what is new in 3.8 by reading the release notes and watching this webinar replay.

We are starting to countdown until RabbitMQ Summit in London on November 4. The RabbitMQ team is looking forward to sharing updates on the project, but we’re also looking forward to hearing from end-users like Bloomberg, WeWork, Softonic, and Zalando. Be sure to register and snag a spot in one of the training add-on courses.

Project updates

Community writings and resources

Events and Training

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ

This Month in RabbitMQ — August 2019

August 12th, 2019 by Michael Klishin

Welcome back for another edition of This Month in RabbitMQ! Some big news last month was Pivotal announced a forthcoming alpha of Pivotal RabbitMQ for Kubernetes.

You can inquire about the alpha here. As part of that announcement, RabbitMQ was mentioned in coverage in Business Insider, Container Journal, SiliconANGLE, Storage Review, The New Stack, and ZDNet. Pretty cool!

Before we move on to the update from the core team and our wonderful community, a reminder that prices for RabbitMQ Summit go up on August 22, so get your tickets now!

You can add on RabbitMQ training to your ticket—basic and advanced courses are available. Great talks planned from Bloomberg, WeWork, Softonic, the Erlang Solutions and CloudAMQP teams, as well as the core RabbitMQ engineers, of course.

Project updates

Community writings and resources

We also came across this article in the International Journal of Research Studies in Computer Science and Engineering (IJRSCSE) on Distributing Messages Using Rabbitmq with Advanced Message Exchanges

Events and Training Courses

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ

This Month in RabbitMQ — July 2019

July 9th, 2019 by Michael Klishin

Welcome back for another edition of This Month in RabbitMQ! In June, we saw the RabbitMQ Summit agenda start to go live, featuring some great returning speakers as well as new faces. There are also a couple of training sessions offered to add onto your ticket. It’s a great way to immerse yourself in all things RabbitMQ for a couple of days. Registration is open, so book your tickets now before the prices go up in August!

Project updates

  • RabbitMQ 3.7.16 has been released with bug fixes, usability improvements and new rabbitmq-diagnostics commands
  • PerfTest 2.8.1 was released with a couple of bug fixes
  • Reactor RabbitMQ 1.3.0.M1 was released as part of the Reactor Dysprosium-M2 release train. More goodies to come in the next few weeks!
  • March Hare 4.0 was released, now based on the 5.7.x series of the RabbitMQ Java client

Community writings and resources

June 5: Vermaden (@vermaden) wrote about setting up RabbitMQ cluster on FreeBSD Jails

June 5: Josh Smeaton (@jarshwah) of Kogan published about monitoring Celery queue length with RabbitMQ

June 7: Emre Tiryaki (@emrtryki) from Hepsiburada published about event ordering with RabbitMQ using the consistent hash exchange (in Turkish)

June 8: Cleison Ferreira Melo (Cleison Ferreira Melo) wrote another installment of his series on building a microservices application, focused on the RabbitMQ container and connection

June 8: Jose Alonso Romero Matias published a four-part video series (in Spanish) showing how to create a messaging project that emulates the sending of invoices through a service, using RabbitMQ: part 1, part 2 on dependency injection with RabbitMQ, part 3 on creating and testing the invoice handler, part 4

June 9: Gilles Robert (@ask4gilles) released v2.0.3 of @opentracing Spring RabbitMQ with a bunch of new instrumented methods on AmqpTemplate and documentation improvements

June 11: Marco Behler (@MarcoBehler) published a video on How to Consume RabbitMQ Messages From Queues With Java

June 13: Maksim Martianov wrote about Kubernetes worker autoscaling based on RabbitMQ queue depth

June 14: Bartha Bela Tibor published about RabbitMQ in Docker with Alpine Linux

June 17: Rafael Capuano published (in Portuguese) a three-part series on the external configuration store pattern, using RabbitMQ for  configuration change propagation: part 1 on contextualizing, part 2 on creating the API, and part 3 on creating the client

June 18: Ram N. published a video and resource links on how to send and receive product objects to or from a queue

June 19: IBM published a tutorial on invoking serverless functions through a message broker

June 23: Karol Galanciak (@Azdaroth) published the third part in a series on Messages on Rails, this one focused on RabbitMQ

June 25: Dhananjay Singh wrote about Spring Cloud Stream with RabbitMQ: Message-Driven Microservices

June 27: Ranga Karanam () published on DZone about Asynchronous Communication With Queues and Microservices: A Perfect Combination?

June 28: Davide Guida (@DavideGuida82) published the first in a series on using message queues in .NET Core (in Italian)

June 29: Teerapong Singthong (@iamgoangle) wrote about Go Messaging System with RabbitMQ and RabbitMQ client for Go (in Thai)

June 30: Md. Al-Amin published about Solving RabbitMQ High CPU/Memory Usages Problem With Celery

Ready to learn more?

Check out these upcoming opportunities to learn more about RabbitMQ:

This Month in RabbitMQ — June 2019

June 6th, 2019 by Michael Klishin

Welcome back for another edition of This Month in RabbitMQ! Keep sharing your war stories and lessons learned out there, and tweet them with #rabbitMQ to get them on our radar for inclusion in these write-ups.

As we march towards RabbitMQ 3.8 going GA, be sure to catch the replay of the webinar we did last month on what’s new in RabbitMQ 3.8. Jack Vanlightly also did some coverage on the Single Active Consumer feature in 3.8, adding to his earlier coverage on Quorum Queues.

Project updates

  • RabbitMQ 3.7.15 was released, includes initial support for Erlang 22
  • Erlang 22 is now GA with a new inter-node communication implementation and initial TLSv1.3 server support
  • Erlang 22.0.2 and 21.3.8.3 were released, addressing ERL-934 and ERL-938, an issue that affected RabbitMQ environments that used TLS and had a high data ingestion volume.
  • RabbitMQ Docker image has transitioned to use Erlang 22
  • PerfTest 2.8.0 has been released with lots of goodies: new options to vary message size and publishing rate, optional polling consumers (instead of asynchronous consumers by default), optionally nack messages instead of acking them, and dependency upgrades
  • Java client 5.7.1 (for Java 8+) and 4.11.1 (for Java 6 & 7) have been released with a bug fix
 

Community writings and resources

May 1: Denis Orehovsky (@apirobotme) published about Distributed systems with RabbitMQ.

May 1: Sam Bently (@sambentley00) published about Building An External Rabbitmq Service For VCloud Director, with particular attention to SSL requirements.

May 1: Nerengen Babu published Integrating RabbitMQ with SpringBoot Application (Receiver Part).

May 5: Chetan Khatri (@khatri_chetan)wrote about How to Setup Airflow Multi-Node Cluster with Celery & RabbitMQ

May 6: mastanggt wrote about migrating RabbitMQ to/within Kubernetes (in Russian).

May 8: Nikita wrote about RabbitMQ Fetching Remote Data, particularly in Rails.

May 10: Jacques Roussel published about Using the Ansible Operator-sdk To Build A RabbitMQ Operator for Kubernetes.

May 10: Aamer Mohammed compares several different messaging technologies, including RabbitMQ, in the context of Asynchronous communication in Microservices.

May 13: Jind?ich Hrabal (@Backglite) published on Dead Letter Queue Reprocessing with Spring Integration and RabbitMQ.

May 14: Prashant Vats published on Kubernetes pod autoscaling in response to the change in the RabbitMQ queue.

May 14: Francisco Cardoso published about what’s different in AMQP 1.0 compared to AMQP 0-9-1 (in Portugese).

May 17: Lakmini Wathsala published How to integrate WSO2 EI with RabbitMQ without passing credentials from address URI.

May 20: Lukasz Lenart (@lukaszlenart?) published on How to configure RabbitMQ via definitions.

May 24: Jeroen Jacobs (@jeroen1205) wrote about troubleshooting sync issues with classic mirrored queues.

May 24: Lovisa Johansson (@lillajja) answered the question “What is the message size limit in RabbitMQ?”

May 24: Ryan Gunn (@Icidis) published about Blazor, RabbitMQ and MQTT using Paho with JSInterop.

May 27: Marcela Sisiliani (@ma_sisiliani) wrote an introduction to the world of queues (in Portugese).

May 28: Hervé Beraud (@4383hberaud) published about How To Play With RabbitMQ And Python Quickly.

May 28: Jack Vanlightly (@vanlightly) wrote about Maintaining Long-Lived Connections with AMQProxy and about Publishing Throughput - Asynchronous vs Synchronous.

May 28: Tomas Henriquez (@Hassek85) published about how he sets up RabbitMQ clusters to scale, noting using the consistent hash plugin and mirrored queues .

May 29: Mohamed Elhachmi published about Efficient design for daemons tasks.

May 29: Denis Setianto published How to Install RabbitMQ Server on Ubuntu 18.04 & 16.04 LTS.

 

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ

6 June 2019, online: Boosting Microservice Performance with Messaging and Spring.

4 November 2019: London: RabbitMQ Summit 2019.

On-demand, online: LearnFly: Learn RabbitMQ Asynchronous Messaging with Java and Spring.

On-demand, online: Udemy: RabbitMQ: Messaging with Java, Spring Boot And Spring MVC.

This Month in RabbitMQ — May 2019

May 13th, 2019 by Michael Klishin

Couple of key public service announcements this month. First, the deadline for submitting a talk for RabbitMQ Summit 2019 (5 November in London UK) was May 10. We had a great line-up last year at the inaugural event and we’re looking forward to an even better event this fall.

Then, on May 23, we’ll be doing an overview of what’s new in RabbitMQ 3.8 (beta 4 of which has dropped recently). Whether you’re a couple versions behind, or on the latest 3.7.14 release, you’re going to want to learn about the latest features and changes.

 

Project updates

  • RabbitMQ 3.7.15-beta.1 is available for community testing
  • And so is 3.8.0-beta.4. See next.rabbitmq.com for documentation of 3.8-specific features.
  • Team RabbitMQ has published an overview of a new feature flag subsystem shipping in RabbitMQ 3.8. The purpose of this subsystem is to simplify rolling upgrades between releases that have incompatible or potentially incompatible changes.
  • RabbitMQ Docker image now ships RabbitMQ 3.7.14 and 3.7.15-beta.1 on latest Erlang and OpenSSL 1.1.1b
  • Java client 5.7.0 (for Java 8+) and 4.11.0 (for Java 6 & 7) have been released with usability improvements and dependency upgrades.
  • Reactor RabbitMQ 1.2.0 GA has been released, with a bug fix, dependency upgrades, and improvements in the publisher confirms support. Reactor RabbitMQ is a reactive API for RabbitMQ based on Reactor and RabbitMQ Java client. Reactor RabbitMQ goal is to enable messages to be published to and consumed from RabbitMQ using functional APIs with non-blocking back-pressure and very low overhead.
  • Debian and RPM packages of several latest Erlang and Elixir releases are now available in Team RabbitMQ's Erlang Bintray repository
 

Community writings and resources

 

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ

Simplifying rolling upgrades between minor versions with feature flags

April 23rd, 2019 by Jean-Sébastien Pédron

In this post we will cover feature flags, a new subsystem in RabbitMQ 3.8. Feature flags will allow a rolling cluster upgrade to the next minor version, without requiring all nodes to be stopped before upgrading.

Minor Version Upgrades Today: RabbitMQ 3.6.x to 3.7.x

It you had to upgrade a cluster from RabbitMQ 3.6.x to 3.7.x, you probably had to use one of the following solutions:

  • Deploy a new cluster alongside the existing one (this strategy is known as the blue-green deployment), then migrate data & clients to the new cluster
  • Stop all nodes in the existing cluster, upgrade the last node that was stopped first, then continue upgrading all other nodes, one-by-one

Blue-green deployment strategy is low risk but also fairly complex to automate. On the other hand, a cluster-wide shutdown affects availability. Feature flags are meant to provide a 3rd option by making rolling cluster upgrades possible and reasonably easy to automate.

The Feature Flags Subsystem

Feature flags indicate a RabbitMQ node's capabilities to its cluster peers. Previously nodes used versions to assess compatibility with cluster versions. There are many ways in which nodes in a distributed system can become incompatible, including Erlang and dependency versions. Many of those aspects are not reflected in a set of version numbers. Feature flags is a better approach as it can reflect more capabilities of a node, whether it is a particular feature or internal communication protocol revision. In fact, with some message protocols RabbitMQ supports clients have a mechanism for clients to indicate their capabilities. This allows client libraries to evolve and be upgraded independently of RabbitMQ nodes.

For example, RabbitMQ 3.8.0 introduces a new queue type, quorum queues. To implement them, an internal data structure and a database schema were modified. This impacts the communication with other nodes because the data structure is exchanged between nodes, and internal data store schema is replicated to all nodes.

Without the feature flags subsystem, it would be impossible to have a RabbitMQ 3.8.0 node inside a cluster where other nodes are running RabbitMQ 3.7.x. Indeed, the 3.7.x nodes would be unable to understand the data structure or the database schema from 3.8.0 node. The opposite is also true. That's why RabbitMQ today prevents this from happening by comparing versions and by denying clustering when versions are considered incompatible (the policy considers different minor/major versions to be incompatible).

New in RabbitMQ 3.8.0 is the feature flags subsystem: when a single node in a 3.7.x cluster is upgraded to 3.8.0 and restarted, it will not immediately enable the new features or migrate its database schema because the feature flags subsystem told it not to. It could determine this because RabbitMQ 3.7.x supports no feature flags at all, therefore new features or behaviours in RabbitMQ 3.8.0 cannot be used before all nodes in the cluster are upgraded.

After a partial upgrade of a cluster to RabbitMQ 3.8.0, all nodes are acting as 3.7.x nodes with regards to incompatible features, even the 3.8.0 one. In this situation, quorum queues are unavailable. Operator must finish the upgrade by upgrading all nodes. When that is done, the operator can decide to enable the new feature flags provided by RabbitMQ 3.8.0: one of them enables quorum queues. This is done using RabbitMQ CLI tools or management UI and supposed to be performed by deployment automation tools. The idea is that the operator needs to confirm that her cluster doesn't have any remaining RabbitMQ 3.7.x nodes that might rejoin the cluster at a later point.

Once a new feature flag is enabled, it is impossible to add a node that runs an older version to that cluster.

Demo with RabbitMQ 3.8.0

Let's go through a complete upgrade of a RabbitMQ 3.7.x cluster. We will take a look at the feature flags in the process.

We have the following 2-node cluster running RabbitMQ 3.7.12:

We now upgrade node A to RabbitMQ 3.8.0 and restart it. Here is what the management overview page looks like after the node is restarted:

We can see the difference of versions in the list of nodes: their version is displayed just below their node name.

The list of feature flags provided by RabbitMQ 3.8.0 is now available in the management UI on node A:

This page will not exist on node B because it is still running RabbitMQ 3.7.12.

On node A, we see that the quorum_queue feature flag is marked as Unavailable. The reason is that node B (still running RabbitMQ 3.7.12) does not known about quorum_queue feature flag, therefore node A is not allowed to use that new feature flag. This feature flag cannot be enabled until all nodes in the cluster support it.

For instance, we could try to declare a quorum queue on node A, but it is denied:

After node B is upgraded, feature flags are available and they can be enabled. We proceed and enable quorum_queue by clicking the Enable button:

Now, we can declare a quorum queue:

To Learn More

The Feature Flags subsystem documentation describes in greater details how it works and what operators and plugin developers should pay attention to.

Note that feature flags are not a guarantee that a cluster shutdown will never be required for upgrades in the future: the ability to implement a change using a feature flag depends on the nature of the change, and is decided on a case-by-case basis. Correct behaviour of a distributed system may require that all of its components behave in a certain way, and sometimes that means that they have to be upgraded in lockstep.

Please give feature flags a try and let us know what you think on the RabbitMQ mailing list!

This Month in RabbitMQ — April 3, 2019

April 3rd, 2019 by Michael Klishin

RabbitMQ 3.8 is coming! If you haven’t already played with the beta (version 3 is now available), it’s time to start familiarizing yourself with what’s coming. Karl Nilsson and I will present on a webinar in May to walk through what’s new, so please register and attend.

We are also starting to look forward to the next RabbitMQ Summit, once again in London this coming November. The call for talks is open until May 10, so please consider sharing how you are using RabbitMQ or something you have tried and learned and want to share with the community.

Project updates

Community writings and resources

Upcoming Courses and Webinars

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ  

This Month in RabbitMQ — March 7, 2019

March 8th, 2019 by Michael Klishin

Welcome back for another issue of This Month in RabbitMQ. Did you know that RabbitMQ was the seventh highest paying tech skill in 2018? AND, that average pay grew a healthy 5.3% since 2017. It’s no wonder that we keep seeing more folks in the community sharing how they are getting started—or getting better—with RabbitMQ. In that spirit, read on for the latest project updates, community writings, and upcoming trainings!

Project updates

Community writings and resources

 

Training

Ready to learn more? Check out these upcoming opportunities to learn more about RabbitMQ: