Posts Tagged ‘New Features’

Simplifying rolling upgrades between minor versions with feature flags

Tuesday, April 23rd, 2019

In this post we will cover feature flags, a new subsystem in RabbitMQ 3.8. Feature flags will allow a rolling cluster upgrade to the next minor version, without requiring all nodes to be stopped before upgrading.

Minor Version Upgrades Today: RabbitMQ 3.6.x to 3.7.x

It you had to upgrade a cluster from RabbitMQ 3.6.x to 3.7.x, you probably had to use one of the following solutions:

  • Deploy a new cluster alongside the existing one (this strategy is known as the blue-green deployment), then migrate data & clients to the new cluster
  • Stop all nodes in the existing cluster, upgrade the last node that was stopped first, then continue upgrading all other nodes, one-by-one

Blue-green deployment strategy is low risk but also fairly complex to automate. On the other hand, a cluster-wide shutdown affects availability. Feature flags are meant to provide a 3rd option by making rolling cluster upgrades possible and reasonably easy to automate.

The Feature Flags Subsystem

Feature flags indicate a RabbitMQ node's capabilities to its cluster peers. Previously nodes used versions to assess compatibility with cluster versions. There are many ways in which nodes in a distributed system can become incompatible, including Erlang and dependency versions. Many of those aspects are not reflected in a set of version numbers. Feature flags is a better approach as it can reflect more capabilities of a node, whether it is a particular feature or internal communication protocol revision. In fact, with some message protocols RabbitMQ supports clients have a mechanism for clients to indicate their capabilities. This allows client libraries to evolve and be upgraded independently of RabbitMQ nodes.

For example, RabbitMQ 3.8.0 introduces a new queue type, quorum queues. To implement them, an internal data structure and a database schema were modified. This impacts the communication with other nodes because the data structure is exchanged between nodes, and internal data store schema is replicated to all nodes.

Without the feature flags subsystem, it would be impossible to have a RabbitMQ 3.8.0 node inside a cluster where other nodes are running RabbitMQ 3.7.x. Indeed, the 3.7.x nodes would be unable to understand the data structure or the database schema from 3.8.0 node. The opposite is also true. That's why RabbitMQ today prevents this from happening by comparing versions and by denying clustering when versions are considered incompatible (the policy considers different minor/major versions to be incompatible).

New in RabbitMQ 3.8.0 is the feature flags subsystem: when a single node in a 3.7.x cluster is upgraded to 3.8.0 and restarted, it will not immediately enable the new features or migrate its database schema because the feature flags subsystem told it not to. It could determine this because RabbitMQ 3.7.x supports no feature flags at all, therefore new features or behaviours in RabbitMQ 3.8.0 cannot be used before all nodes in the cluster are upgraded.

After a partial upgrade of a cluster to RabbitMQ 3.8.0, all nodes are acting as 3.7.x nodes with regards to incompatible features, even the 3.8.0 one. In this situation, quorum queues are unavailable. Operator must finish the upgrade by upgrading all nodes. When that is done, the operator can decide to enable the new feature flags provided by RabbitMQ 3.8.0: one of them enables quorum queues. This is done using RabbitMQ CLI tools or management UI and supposed to be performed by deployment automation tools. The idea is that the operator needs to confirm that her cluster doesn't have any remaining RabbitMQ 3.7.x nodes that might rejoin the cluster at a later point.

Once a new feature flag is enabled, it is impossible to add a node that runs an older version to that cluster.

Demo with RabbitMQ 3.8.0

Let's go through a complete upgrade of a RabbitMQ 3.7.x cluster. We will take a look at the feature flags in the process.

We have the following 2-node cluster running RabbitMQ 3.7.12:

We now upgrade node A to RabbitMQ 3.8.0 and restart it. Here is what the management overview page looks like after the node is restarted:

We can see the difference of versions in the list of nodes: their version is displayed just below their node name.

The list of feature flags provided by RabbitMQ 3.8.0 is now available in the management UI on node A:

This page will not exist on node B because it is still running RabbitMQ 3.7.12.

On node A, we see that the quorum_queue feature flag is marked as Unavailable. The reason is that node B (still running RabbitMQ 3.7.12) does not known about quorum_queue feature flag, therefore node A is not allowed to use that new feature flag. This feature flag cannot be enabled until all nodes in the cluster support it.

For instance, we could try to declare a quorum queue on node A, but it is denied:

After node B is upgraded, feature flags are available and they can be enabled. We proceed and enable quorum_queue by clicking the Enable button:

Now, we can declare a quorum queue:

To Learn More

The Feature Flags subsystem documentation describes in greater details how it works and what operators and plugin developers should pay attention to.

Note that feature flags are not a guarantee that a cluster shutdown will never be required for upgrades in the future: the ability to implement a change using a feature flag depends on the nature of the change, and is decided on a case-by-case basis. Correct behaviour of a distributed system may require that all of its components behave in a certain way, and sometimes that means that they have to be upgraded in lockstep.

Please give feature flags a try and let us know what you think on the RabbitMQ mailing list!

New Configuration Format in RabbitMQ 3.7

Thursday, February 22nd, 2018

In this post we'll cover a new configuration format available in RabbitMQ 3.7.0.


Peer Discovery Subsystem in RabbitMQ 3.7

Monday, February 12th, 2018

In this blog post we're going to take a closer look at a new subsystem introduced in RabbitMQ 3.7.0.


Growing Up

Friday, August 27th, 2010

Some three and a half years after we launched RabbitMQ, we have this week released RabbitMQ 2.0.

This means some big changes.  The most important of these is our new Scalable Storage Engine.  RabbitMQ has always provided persistence for failure recovery.  But now, you can happily push data into Rabbit regardless of how much data is already stored, and you don’t need to worry about slow consumers disrupting processing.  As the demands on your application grow, Rabbit can scale with you, in a stable, reliable way.

Before introducing RabbitMQ 2.0, let me reiterate that as Rabbit evolves you can count on the same high level of commitment to you as a customer or end user, regardless of whether you are a large enterprise, or a next-gen start-up, or an open source community.  As always, get in touch if you need help or commercial support.

New capabilities

So what can you expect? In short, a more capable bunny. In particular:

1. RabbitMQ 2.0 has an all new Scalable Storage Engine.  There is also a persistence API.

2. Native support for multi-protocol messaging delivering better interoperability and more choice

3. The release coincides with first class Spring integration for both Java and .NET from SpringSource - with more to come e.g. a Grails plugin

4. Foundations for future additional management and monitoring features as part of rabbit-management and other tools

5. Plugins are now distributed as drop-in binary packages, making them much easier to use.

As always, read the full release notes before upgrading.

Scalable Storage means greater stability

The vision of Rabbit is that messaging should “just work”. Part of this is the need for “stable behaviour at scale”. You ideally need a broker that you can just start up and forget about. Our new Scalable Storage Engine takes us closer to this.

RabbitMQ has always had its own storage engine whose job is to persist messages. We use this to provide guaranteed eventual delivery, support for transactions, and high availability.

But, in RabbitMQ 1.x, the broker used a naïve albeit effective storage model: every message stored on disk would also be cached in RAM. While this helps performance it makes it much harder to manage scale unless you are able to predict growth and overprovision memory accordingly.

With RabbitMQ 2.0, the broker is capable of completely flushing messages to disk. It has a much smarter caching and paging model. This improves memory usage by paging messages out to disk, and in from disk, as needed. In other words, you don’t have to worry about one slow consumer disrupting your entire set up, and can happily push data into Rabbit regardless of how much data is already stored. This improves stability at scale.

We'll blog about this soon.  In the meantime, please try it out.  Storage enthusiasts might like to check out the notes in the codebase.

Storage API

RabbitMQ now provides a storage API. This allows “pluggable” persistence.

RabbitMQ includes its own persistent message database, which is optimised for messaging use cases. But what if you just have to use Oracle or MySQL? Or, what if you want to experiment with combinations of RabbitMQ with the latest and greatest NoSQL key-value store? Now you can do that.  Please contact us if you want to write a driver for your favourite store.

Natively Multiprotocol

Our aim is lower the cost of integration by making messaging simpler, providing a stable manageable server, and supporting the main technologies that you need.

Rabbit has supported AMQP since its very inception. RabbitMQ began with support for AMQP 0-8, and as AMQP evolved, added support for 0-9-1 on a branch.  AMQP 0-9-1 is fewer than 50 pages long, making it highly readable, as well as tree-friendly. A shorter, simpler protocol is easier to implement and validate which is important for customers who want AMQP from more than one vendor.

With the 2.0 release, RabbitMQ can now directly support multiple protocols at its core - with AMQP 0-8 and 0-9-1 pre-integrated.

In addition, RabbitMQ extensions provide support for a wide range of protocols including XMPP, STOMP, HTTP JSON/RPC, pubsub for HTTP (e.g. Google's PubSubHubBub), and SMTP.

Contact us if you want support for more protocols, or have questions about our future plans; e.g.: providing a safe evolutionary path to future versions of AMQP.

Better Plugin Support

We aim to provide support for a wide range of messaging applications without making the broker bloated and complex.  Plugins are key to this.  They let us and you extend and customise the capability of RabbitMQ.  Previously you could only load our plugins by building from source.  With the 2.0 release we are distributing pre-compiled plugins.  Drop them into a directory from where RabbitMQ can load them.

To get a feel for what you can do, I recommend taking a look at Tony Garnock-Jones excellent introduction to plugins from the last Erlang Factory.  Congratulations are due to Jon Brisbin for being the first person to create a plugin for RabbitMQ 2.0, in less than a day or two, adding webhooks for RabbitMQ.  Matthew Sackman and Tony Garnock-Jones have also created some custom exchanges.  Please note that many of these are demos and examples so the usual caveats apply.

First class Spring support in Java and .NET from SpringSource

We think that messaging should be intuitive regardless of the application platform you develop for. In Java, the clear leader is Spring and we are part of the SpringSource division of Vmware, thus adding fully fledged Spring support was a must. We are extremely grateful to Mark Fisher and Mark Pollack from the SpringSource team for bringing this to fruition. With the release of RabbitMQ 2.0 we are highlighting to the whole community that Spring support is officially available.

If you are a .NET user, you have been able to run RabbitMQ as a Windows service, and use it from .NET languages and WCF for some years now. This is great if you want to do messaging from Windows based applications, such as Excel, to back end services written in Java or any other of the hundreds of AMQP integration points. Now, with Spring.NET support we offer you a common application development model as well, that works equally well in both Java and .NET.

Putting it all together: more freedom to choose

We believe that protocol interoperability and easier integration give you choice. What if you are deploying for the enterprise, and need to connect RabbitMQ to legacy enterprise messaging systems that do not support AMQP? Spring gives you a way forward. With Spring Integration, you have first class access to most messaging systems. Our commitment is to support customers' choice of technology.  Freedom from lock-in means that you can expect to evolve your systems in line with business needs, instead of being constrained by vendors’ product plans and pricing schemes.

Management and monitoring

We have made important improvements to Rabbit’s management, and overall serviceability.

Rock solid management, across ‘any’ use case, is the heart of what makes messaging non-trivial. It’s easy to find messaging tools that focus on just one or two use cases, and do a reasonable job. But providing that all important “it just works” experience in the majority of cases at once, and then running stably over time, and not crashing once a month at 2am, well that’s hard. It takes care and it really needs good management tooling.

Rabbit has always provided some tools for management and the community has done an outstanding job in extending them and making them more useful and relevant. But we want more features, and this means making changes to the broker, and especially at the API level.  These APIs are critical for showing end users what we think is important about messaging, and what is not. How people interact with a tool, and how they use it, defines their relationship with that tool.

In 2.0 you will find support for instrumentation for gathering metrics and statistics about the health of your Rabbit, without significantly impacting performance. Typically people expect to see: current message rates, emerging bottlenecks, support for remedial action such as telling specific clients to throttle back when the broker is busy. Features like these will become visible as part of the rabbit-management plugin and other tools. The foundations are now in place for their addition incrementally.

Tell us what else you want from management

Please contact us with requests and ideas for more management and monitoring features. We’ll be adding more to RabbitMQ’s core, making it even easier to operate a large Rabbit installation 24/7, indefinitely. We’ll be looking for feedback from the community to make sure it’s really easy for people to integrate management and monitoring feeds into their favourite tools.

Plus, for SpringSource customers: expect each new feature to also show up in the Hyperic plugin, enabling you to monitor and manage RabbitMQ brokers, queues and exchanges inside your complete Spring stack and thus derive immediate “in context” intelligence about the overall behaviour of your application.

What else is next and how can I help?

As much as possible, future features will appear in stages. Our release philosophy will be evolutionary, while keeping the same focus on quality as with the previous releases. We’ll be working very hard on all of the following items, and more besides, so please get in touch if you want to use them or help:

1. Making Rabbit even easier to use.

2. Even better support for community plugins!

3. “Elastic” messaging for cloud services on EC2 and elsewhere

4. New styles of federation, complementing our existing rabbitmq-shovel relay

5. Even better HA, to cover a wider class of failover cases

We want your feedback! The best way you can help is to talk about what you want to do on the rabbitmq-discuss mailing list. That’s also the best place for you to get help if you need it. Or, if you want to go public to the max: talk to us on Twitter where we are @rabbitmq


We want to thank two groups of people for getting us to here.

First – the community. We especially wish to thank those people who tested the new persistence technology throughout this year.

Second – we want to thank our customers and everyone else who has commercially sponsored RabbitMQ over the years. Whether you “bought a new feature”, or took a real risk with us, your contribution is just as important to us.

RabbitMQ and AMQP 0-9-1

Wednesday, August 4th, 2010

Since the beginning, RabbitMQ has implemented the 0-8 version of the AMQP specification. This was the first publicly-available version, but there's been plenty of development since then. In particular, we've wanted to support the 0-9-1 version of AMQP for a while now.

For around the last year we've maintained a 0-9-1-supporting branch of the broker and clients in Mercurial, and interested users have been running that version as it matures. Well, it's finally mature and we're now merging the 0-9-1 support into the default branch. This means it'll be in the next release.

At this point, you're probably wondering what the differences are between 0-8 and 0-9-1. Well, the good news is that the differences are not huge, and are entirely sensible. 0-9-1 mostly cleans up the 0-8 spec, explaining more clearly how the broker and clients are expected to behave in certain edge cases, and removing a whole load of ambiguity and half (or less) thought out features from 0-8.

In fact, if you're running 1.8.0 or later, you're already running a broker with most of the semantic changes from 0-9-1. But the wire protocol has changed a bit too, so it matters whether you're speaking 0-9-1 or 0-8.

For the Java and Erlang clients, we're intending to simply switch to supporting AMQP 0-9-1 exclusively in the next release. The .NET library already supports multiple protocols, so we're adding 0-9-1 as another option (and making it the default). For the broker we're going to support both AMQP 0-9-1 and AMQP 0-8. For any other client libraries we encourage you to speak to the library developer 😉

This means that this time it's safe to upgrade just your broker, or your broker and all your clients, but it's not safe to upgrade your clients without upgrading your broker!

Thank you for your attention.