Archive for the ‘New Features’ Category

Simplifying rolling upgrades between minor versions with feature flags

Tuesday, April 23rd, 2019

In this post we will cover feature flags, a new subsystem in RabbitMQ 3.8. Feature flags will allow a rolling cluster upgrade to the next minor version, without requiring all nodes to be stopped before upgrading.

Minor Version Upgrades Today: RabbitMQ 3.6.x to 3.7.x

It you had to upgrade a cluster from RabbitMQ 3.6.x to 3.7.x, you probably had to use one of the following solutions:

  • Deploy a new cluster alongside the existing one (this strategy is known as the blue-green deployment), then migrate data & clients to the new cluster
  • Stop all nodes in the existing cluster, upgrade the last node that was stopped first, then continue upgrading all other nodes, one-by-one

Blue-green deployment strategy is low risk but also fairly complex to automate. On the other hand, a cluster-wide shutdown affects availability. Feature flags are meant to provide a 3rd option by making rolling cluster upgrades possible and reasonably easy to automate.

The Feature Flags Subsystem

Feature flags indicate a RabbitMQ node's capabilities to its cluster peers. Previously nodes used versions to assess compatibility with cluster versions. There are many ways in which nodes in a distributed system can become incompatible, including Erlang and dependency versions. Many of those aspects are not reflected in a set of version numbers. Feature flags is a better approach as it can reflect more capabilities of a node, whether it is a particular feature or internal communication protocol revision. In fact, with some message protocols RabbitMQ supports clients have a mechanism for clients to indicate their capabilities. This allows client libraries to evolve and be upgraded independently of RabbitMQ nodes.

For example, RabbitMQ 3.8.0 introduces a new queue type, quorum queues. To implement them, an internal data structure and a database schema were modified. This impacts the communication with other nodes because the data structure is exchanged between nodes, and internal data store schema is replicated to all nodes.

Without the feature flags subsystem, it would be impossible to have a RabbitMQ 3.8.0 node inside a cluster where other nodes are running RabbitMQ 3.7.x. Indeed, the 3.7.x nodes would be unable to understand the data structure or the database schema from 3.8.0 node. The opposite is also true. That's why RabbitMQ today prevents this from happening by comparing versions and by denying clustering when versions are considered incompatible (the policy considers different minor/major versions to be incompatible).

New in RabbitMQ 3.8.0 is the feature flags subsystem: when a single node in a 3.7.x cluster is upgraded to 3.8.0 and restarted, it will not immediately enable the new features or migrate its database schema because the feature flags subsystem told it not to. It could determine this because RabbitMQ 3.7.x supports no feature flags at all, therefore new features or behaviours in RabbitMQ 3.8.0 cannot be used before all nodes in the cluster are upgraded.

After a partial upgrade of a cluster to RabbitMQ 3.8.0, all nodes are acting as 3.7.x nodes with regards to incompatible features, even the 3.8.0 one. In this situation, quorum queues are unavailable. Operator must finish the upgrade by upgrading all nodes. When that is done, the operator can decide to enable the new feature flags provided by RabbitMQ 3.8.0: one of them enables quorum queues. This is done using RabbitMQ CLI tools or management UI and supposed to be performed by deployment automation tools. The idea is that the operator needs to confirm that her cluster doesn't have any remaining RabbitMQ 3.7.x nodes that might rejoin the cluster at a later point.

Once a new feature flag is enabled, it is impossible to add a node that runs an older version to that cluster.

Demo with RabbitMQ 3.8.0

Let's go through a complete upgrade of a RabbitMQ 3.7.x cluster. We will take a look at the feature flags in the process.

We have the following 2-node cluster running RabbitMQ 3.7.12:

We now upgrade node A to RabbitMQ 3.8.0 and restart it. Here is what the management overview page looks like after the node is restarted:

We can see the difference of versions in the list of nodes: their version is displayed just below their node name.

The list of feature flags provided by RabbitMQ 3.8.0 is now available in the management UI on node A:

This page will not exist on node B because it is still running RabbitMQ 3.7.12.

On node A, we see that the quorum_queue feature flag is marked as Unavailable. The reason is that node B (still running RabbitMQ 3.7.12) does not known about quorum_queue feature flag, therefore node A is not allowed to use that new feature flag. This feature flag cannot be enabled until all nodes in the cluster support it.

For instance, we could try to declare a quorum queue on node A, but it is denied:

After node B is upgraded, feature flags are available and they can be enabled. We proceed and enable quorum_queue by clicking the Enable button:

Now, we can declare a quorum queue:

To Learn More

The Feature Flags subsystem documentation describes in greater details how it works and what operators and plugin developers should pay attention to.

Note that feature flags are not a guarantee that a cluster shutdown will never be required for upgrades in the future: the ability to implement a change using a feature flag depends on the nature of the change, and is decided on a case-by-case basis. Correct behaviour of a distributed system may require that all of its components behave in a certain way, and sometimes that means that they have to be upgraded in lockstep.

Please give feature flags a try and let us know what you think on the RabbitMQ mailing list!

RabbitMQ Java Client Metrics with Micrometer and Datadog

Tuesday, April 10th, 2018

In this post we'll cover how the RabbitMQ Java client library gathers runtime metrics and sends them to monitoring systems like JMX and Datadog.


New Configuration Format in RabbitMQ 3.7

Thursday, February 22nd, 2018

In this post we'll cover a new configuration format available in RabbitMQ 3.7.0.


Peer Discovery Subsystem in RabbitMQ 3.7

Monday, February 12th, 2018

In this blog post we're going to take a closer look at a new subsystem introduced in RabbitMQ 3.7.0.


What’s New in RabbitMQ 3.7

Monday, February 5th, 2018

After over 1 year in the works, RabbitMQ 3.7.0 has quietly shipped right before the start of the holiday season. The release was heavily inspired by the community feedback on 3.6.x. In this post we’d like to cover some of the highlights in this release.


New Reactive Client for RabbitMQ HTTP API

Wednesday, October 18th, 2017

The RabbitMQ team is happy to announce the release of version 2.0 of HOP, RabbitMQ HTTP API client for Java and other JVM languages. This new release introduce a new reactive client based on Spring Framework 5.0 WebFlux.


RabbitMQ Java Client 5.0 is Released

Friday, September 29th, 2017

The RabbitMQ team is happy to announce the release of version 5.0 of the RabbitMQ Java Client. This new release is now based on Java 8 and comes with a bunch of interesting new features.


Brand new rabbitmqctl in 3.7.0

Thursday, December 15th, 2016

As of v3.7.0 Milestone 8, RabbitMQ ships with a brand new set of CLI tools (rabbitmqctl, rabbitmq-plugins, and more), reworked from the ground up. We had a few goals with this project:

  • We wanted to use a more user-friendly command line parser and produce more useful help and error messages.
  • CLI tools should be extensible from plugins: plugins such as management, federation, shovel, trust store all have functions that are meant to be invoked by CLI tools but the only way of doing it was rabbitmqctl eval, which is error prone and can be dangerous.
  • We wanted to give Elixir a try on a real project and make it easier for developers new to Erlang to extend the CLI functionality.
  • Our CLI tools historically didn't have good test coverage; the new ones should (and do).
  • CLI tools should be able to produce machine-friendly formats, be it JSON, CSV or something else; there was no internal infrastructure for doing that in the original implementation.
  • CLI tools should be a separate repository just like all plugins, client libraries, and so on.
Nine months later the experiment was declared a success and integrated into RabbitMQ distribution.

Please give v3.7.0 Milestone 8 a try and take a look at how easy it is to extend the CLI.

There's also a longer document that covers new features and implementation decisions.

Metrics support in RabbitMQ Java Client 4.0

Wednesday, November 30th, 2016

Version 4.0 of the RabbitMQ Java Client brings support for runtime metrics. This can be especially useful to know how a client application is behaving. Let's see how to enable metrics collection and how to monitor those metrics on JMX or even inside a Spring Boot application.


RabbitMQ Java Client 4.0 is released

Thursday, November 24th, 2016

The RabbitMQ team is happy to announce the release of version 4.0 of the RabbitMQ Java Client. This new release does not introduce any breaking changes and comes with a bunch of interesting new features.