Archive for August, 2010

Growing Up

Friday, August 27th, 2010

Some three and a half years after we launched RabbitMQ, we have this week released RabbitMQ 2.0.

This means some big changes.  The most important of these is our new Scalable Storage Engine.  RabbitMQ has always provided persistence for failure recovery.  But now, you can happily push data into Rabbit regardless of how much data is already stored, and you don’t need to worry about slow consumers disrupting processing.  As the demands on your application grow, Rabbit can scale with you, in a stable, reliable way.

Before introducing RabbitMQ 2.0, let me reiterate that as Rabbit evolves you can count on the same high level of commitment to you as a customer or end user, regardless of whether you are a large enterprise, or a next-gen start-up, or an open source community.  As always, get in touch if you need help or commercial support.

New capabilities

So what can you expect? In short, a more capable bunny. In particular:

1. RabbitMQ 2.0 has an all new Scalable Storage Engine.  There is also a persistence API.

2. Native support for multi-protocol messaging delivering better interoperability and more choice

3. The release coincides with first class Spring integration for both Java and .NET from SpringSource - with more to come e.g. a Grails plugin

4. Foundations for future additional management and monitoring features as part of rabbit-management and other tools

5. Plugins are now distributed as drop-in binary packages, making them much easier to use.

As always, read the full release notes before upgrading.

Scalable Storage means greater stability

The vision of Rabbit is that messaging should “just work”. Part of this is the need for “stable behaviour at scale”. You ideally need a broker that you can just start up and forget about. Our new Scalable Storage Engine takes us closer to this.

RabbitMQ has always had its own storage engine whose job is to persist messages. We use this to provide guaranteed eventual delivery, support for transactions, and high availability.

But, in RabbitMQ 1.x, the broker used a naïve albeit effective storage model: every message stored on disk would also be cached in RAM. While this helps performance it makes it much harder to manage scale unless you are able to predict growth and overprovision memory accordingly.

With RabbitMQ 2.0, the broker is capable of completely flushing messages to disk. It has a much smarter caching and paging model. This improves memory usage by paging messages out to disk, and in from disk, as needed. In other words, you don’t have to worry about one slow consumer disrupting your entire set up, and can happily push data into Rabbit regardless of how much data is already stored. This improves stability at scale.

We'll blog about this soon.  In the meantime, please try it out.  Storage enthusiasts might like to check out the notes in the codebase.

Storage API

RabbitMQ now provides a storage API. This allows “pluggable” persistence.

RabbitMQ includes its own persistent message database, which is optimised for messaging use cases. But what if you just have to use Oracle or MySQL? Or, what if you want to experiment with combinations of RabbitMQ with the latest and greatest NoSQL key-value store? Now you can do that.  Please contact us if you want to write a driver for your favourite store.

Natively Multiprotocol

Our aim is lower the cost of integration by making messaging simpler, providing a stable manageable server, and supporting the main technologies that you need.

Rabbit has supported AMQP since its very inception. RabbitMQ began with support for AMQP 0-8, and as AMQP evolved, added support for 0-9-1 on a branch.  AMQP 0-9-1 is fewer than 50 pages long, making it highly readable, as well as tree-friendly. A shorter, simpler protocol is easier to implement and validate which is important for customers who want AMQP from more than one vendor.

With the 2.0 release, RabbitMQ can now directly support multiple protocols at its core - with AMQP 0-8 and 0-9-1 pre-integrated.

In addition, RabbitMQ extensions provide support for a wide range of protocols including XMPP, STOMP, HTTP JSON/RPC, pubsub for HTTP (e.g. Google's PubSubHubBub), and SMTP.

Contact us if you want support for more protocols, or have questions about our future plans; e.g.: providing a safe evolutionary path to future versions of AMQP.

Better Plugin Support

We aim to provide support for a wide range of messaging applications without making the broker bloated and complex.  Plugins are key to this.  They let us and you extend and customise the capability of RabbitMQ.  Previously you could only load our plugins by building from source.  With the 2.0 release we are distributing pre-compiled plugins.  Drop them into a directory from where RabbitMQ can load them.

To get a feel for what you can do, I recommend taking a look at Tony Garnock-Jones excellent introduction to plugins from the last Erlang Factory.  Congratulations are due to Jon Brisbin for being the first person to create a plugin for RabbitMQ 2.0, in less than a day or two, adding webhooks for RabbitMQ.  Matthew Sackman and Tony Garnock-Jones have also created some custom exchanges.  Please note that many of these are demos and examples so the usual caveats apply.

First class Spring support in Java and .NET from SpringSource

We think that messaging should be intuitive regardless of the application platform you develop for. In Java, the clear leader is Spring and we are part of the SpringSource division of Vmware, thus adding fully fledged Spring support was a must. We are extremely grateful to Mark Fisher and Mark Pollack from the SpringSource team for bringing this to fruition. With the release of RabbitMQ 2.0 we are highlighting to the whole community that Spring support is officially available.

If you are a .NET user, you have been able to run RabbitMQ as a Windows service, and use it from .NET languages and WCF for some years now. This is great if you want to do messaging from Windows based applications, such as Excel, to back end services written in Java or any other of the hundreds of AMQP integration points. Now, with Spring.NET support we offer you a common application development model as well, that works equally well in both Java and .NET.

Putting it all together: more freedom to choose

We believe that protocol interoperability and easier integration give you choice. What if you are deploying for the enterprise, and need to connect RabbitMQ to legacy enterprise messaging systems that do not support AMQP? Spring gives you a way forward. With Spring Integration, you have first class access to most messaging systems. Our commitment is to support customers' choice of technology.  Freedom from lock-in means that you can expect to evolve your systems in line with business needs, instead of being constrained by vendors’ product plans and pricing schemes.

Management and monitoring

We have made important improvements to Rabbit’s management, and overall serviceability.

Rock solid management, across ‘any’ use case, is the heart of what makes messaging non-trivial. It’s easy to find messaging tools that focus on just one or two use cases, and do a reasonable job. But providing that all important “it just works” experience in the majority of cases at once, and then running stably over time, and not crashing once a month at 2am, well that’s hard. It takes care and it really needs good management tooling.

Rabbit has always provided some tools for management and the community has done an outstanding job in extending them and making them more useful and relevant. But we want more features, and this means making changes to the broker, and especially at the API level.  These APIs are critical for showing end users what we think is important about messaging, and what is not. How people interact with a tool, and how they use it, defines their relationship with that tool.

In 2.0 you will find support for instrumentation for gathering metrics and statistics about the health of your Rabbit, without significantly impacting performance. Typically people expect to see: current message rates, emerging bottlenecks, support for remedial action such as telling specific clients to throttle back when the broker is busy. Features like these will become visible as part of the rabbit-management plugin and other tools. The foundations are now in place for their addition incrementally.

Tell us what else you want from management

Please contact us with requests and ideas for more management and monitoring features. We’ll be adding more to RabbitMQ’s core, making it even easier to operate a large Rabbit installation 24/7, indefinitely. We’ll be looking for feedback from the community to make sure it’s really easy for people to integrate management and monitoring feeds into their favourite tools.

Plus, for SpringSource customers: expect each new feature to also show up in the Hyperic plugin, enabling you to monitor and manage RabbitMQ brokers, queues and exchanges inside your complete Spring stack and thus derive immediate “in context” intelligence about the overall behaviour of your application.

What else is next and how can I help?

As much as possible, future features will appear in stages. Our release philosophy will be evolutionary, while keeping the same focus on quality as with the previous releases. We’ll be working very hard on all of the following items, and more besides, so please get in touch if you want to use them or help:

1. Making Rabbit even easier to use.

2. Even better support for community plugins!

3. “Elastic” messaging for cloud services on EC2 and elsewhere

4. New styles of federation, complementing our existing rabbitmq-shovel relay

5. Even better HA, to cover a wider class of failover cases

We want your feedback! The best way you can help is to talk about what you want to do on the rabbitmq-discuss mailing list. That’s also the best place for you to get help if you need it. Or, if you want to go public to the max: talk to us on Twitter where we are @rabbitmq


We want to thank two groups of people for getting us to here.

First – the community. We especially wish to thank those people who tested the new persistence technology throughout this year.

Second – we want to thank our customers and everyone else who has commercially sponsored RabbitMQ over the years. Whether you “bought a new feature”, or took a real risk with us, your contribution is just as important to us.

Management, monitoring and statistics

Friday, August 6th, 2010

For a long time the management and monitoring capabilities built into RabbitMQ have consisted of rabbitmqctl. While it's a reasonable tool for management (assuming you like the command line), rabbitmqctl has never been very powerful as a monitoring tool. So we're going to build something better.

Of course, plenty of people don't like command line tools and so several people have built alternate means of managing RabbitMQ. Alice / Wonderland and Spring AMQP are two that leap to mind. However there isn't much standardisation, and people don't always find it very convenient to talk to Rabbit via epmd. So we're going to build something easier.

With that in mind, I'd like to announce the existence of RabbitMQ Management. This is a plugin to provide management and monitoring via a RESTful interface, with a web GUI. Think of it as being like a super-Alice, and also an integration point for any other tools that people want to build.

So RabbitMQ Management will allow easier management and better monitoring. How better? Well, as part of the upcoming rabbit release we've added a statistics gathering feature to the broker. This can count messages as they're published, routed, delivered and acked, on a per channel / queue / exchange basis, so we can determine things like which channels are publishing fast or consuming slowly, which queues are being published to from which exchanges, which connections and hosts are busiest, and so on.

Of course, doing all this extra bookkeeping comes at a cost; when statistics gathering is turned on the server can run around 10% slower (assuming it's CPU-bound; which usually means handling transient messages. If it's IO bound the performance impact will likely be less). At the moment statistics gathering is turned on automatically when the management plugin is installed but we'll make it configurable.

With that, I'd like to add a warning: it's currently at a very early stage of development, and really should not be trusted for anything other than experimentation and playing with. The REST API will change, the UI will change, the plugin might crash your server, and the TODO is currently almost hilariously long. But hopefully this gives you an idea of what we're going to do.

RabbitMQ and AMQP 0-9-1

Wednesday, August 4th, 2010

Since the beginning, RabbitMQ has implemented the 0-8 version of the AMQP specification. This was the first publicly-available version, but there's been plenty of development since then. In particular, we've wanted to support the 0-9-1 version of AMQP for a while now.

For around the last year we've maintained a 0-9-1-supporting branch of the broker and clients in Mercurial, and interested users have been running that version as it matures. Well, it's finally mature and we're now merging the 0-9-1 support into the default branch. This means it'll be in the next release.

At this point, you're probably wondering what the differences are between 0-8 and 0-9-1. Well, the good news is that the differences are not huge, and are entirely sensible. 0-9-1 mostly cleans up the 0-8 spec, explaining more clearly how the broker and clients are expected to behave in certain edge cases, and removing a whole load of ambiguity and half (or less) thought out features from 0-8.

In fact, if you're running 1.8.0 or later, you're already running a broker with most of the semantic changes from 0-9-1. But the wire protocol has changed a bit too, so it matters whether you're speaking 0-9-1 or 0-8.

For the Java and Erlang clients, we're intending to simply switch to supporting AMQP 0-9-1 exclusively in the next release. The .NET library already supports multiple protocols, so we're adding 0-9-1 as another option (and making it the default). For the broker we're going to support both AMQP 0-9-1 and AMQP 0-8. For any other client libraries we encourage you to speak to the library developer 😉

This means that this time it's safe to upgrade just your broker, or your broker and all your clients, but it's not safe to upgrade your clients without upgrading your broker!

Thank you for your attention.

Well, I’ll let you go … basic.reject in RabbitMQ

Tuesday, August 3rd, 2010

Support for AMQP's basic.reject just landed on default. It's taken this long because we couldn't agree on a single set of semantics that followed the specification, was actually useful, and wasn't too complicated to implement.

First up, this is what RabbitMQ's basic.reject will do: if you supply requeue=false it will discard the message (this is in lieu of dead-lettering it, until we have a dead letter feature); if you supply requeue=true, it will release it back on to the queue, to be delivered again.

The first is useful from a error handling point of view; if your application cannot process a particular message, you can get rid of it. At the minute, it's semantically the same as just acking the message; but, given a dead-letter mechanism, it will mean unprocessable messages can be picked up elsewhere for diagnosis.

The second, with requeue=true, is useful for example if your application needs a "message lock" semantics. In this scenario, a consumer can be delivered a message, then decide not to deal with it after all, and place it back on the queue. Note that RabbitMQ doesn't take care to stop the same consumer getting the message again -- see below.

The AMQP 0-9-1 specification says a number of seemingly incompatible things about basic.reject. For a start, it says in the method description

The client MUST NOT use this method as a means of selecting messages to process.
and in the specification XML (but not in the generated PDF),
The server MUST NOT deliver the message to the same client within the context of the current channel. The recommended strategy is to attempt to deliver the message to an alternative consumer, and if that is not possible, to move the message to a dead-letter queue. The server MAY use more sophisticated tracking to hold the message on the queue and redeliver it to the same client at a later stage.

This seems to suggest that the server has to take care not to deliver the message to the same consumer twice, but consumers are not allowed to rely on this prohibition. This means basic.reject could either redeliver the message or dead-letter it, which makes it useless for the "message lock" scenario given above.

So, we have chosen to implement the simplest thing that is useful, which is to re-enqueue the message and treat it as though it were completely new. This means the consumer can receive again a message it has rejected.

Hello, World!

Sunday, August 1st, 2010

And welcome to the all-new RabbitMQ blog!

As you may know, RabbitMQ was recently acquired by SpringSource, a division of VMware.  Prior to that, the RabbitMQ developers worked at LShift, and we used to write on the LShift blog.  You can still read the old RabbitMQ posts there.

But now it's time for the RabbitMQ team to have its own blog. We have lots to write about: RabbitMQ itself, and new features as they appear; our more experimental projects; and what's going on in the RabbitMQ community and the world of messaging in general. The first real post will follow shortly.