Menu

Archive for the ‘New Features’ Category

AMQP 1.0 prototyping

Wednesday, December 1st, 2010

We have been prototyping support for a new protocol, as is our wont. This one is called "AMQP 1.0 R0", and it is the new issue from the AMQP working group (of which RabbitMQ, and latterly VMware, are a member). The "R0" indicates that it's the first revision of a recommendation. The specification is incomplete: there are many TODOs, and to a large extent it is unproven. Those two facts are part of what prompted this prototyping.

The prototype code is mirrored at github: http://github.com/rabbitmq/rabbitmq-amqp1.0. It is built just the same as all our plugins.

The AMQP 1.0 R0 specification differs from the specification of previous versions of AMQP, in that it does not define a broker model; i.e., it doesn't define exchanges queues and bindings, or their equivalents. The protocol is really only about transferring messages from one agent to another, and then agreeing on what the outcome was. That means it is amenable to bolting on to a message broker implementation, among other uses -- the idea is that one can adapt an existing model to suit.

In our case, the incumbent model is that of AMQP 0-9-1, with some generalisations and extensions (for example, chained bindings). Our target with the prototype is therefore to be able to get something useful done with both 1.0 clients and 0-9-1 clients connected at the same time.

Well, the good news is, we've achieved that. In fact the plugin can be set up to replace Rabbit's usual network listener, and will happily talk to AMQP 0-8, 0-9-1, and 1.0 clients. We did have to do some invention along the way, and there are some parts of the specification that we are conspicuously not implementing. These will be detailed in the README soon.

One large part of the invention is to fill in semantics where the specification is silent. Some of these are detailed in this client-broker protocol work we did for the AMQP working group. We're hoping the prototyping will help fill this out some more.

Next week I'll be taking our prototype to the AMQP 1.0 "Connectathon", where it'll be tested against other implementations of the core protocol (not all of which are open source). Again, this will help to flush out barriers to interoperability in the specification.

Exchange to Exchange bindings

Tuesday, October 19th, 2010

Arriving in RabbitMQ 2.1.1, is support for bindings between exchanges. This is an extension of the AMQP specification and making use of this feature will (currently) result in your application only functioning with RabbitMQ, and not the myriad of other AMQP 0-9-1 broker implementations out there. However, this extension brings a massive increase to the expressivity and flexibility of routing topologies, and solves some scalability issues at the same time.

Normal bindings allow exchanges to be bound to queues: messages published to an exchange will, provided the various criteria  of the exchange and its bindings are met, pass through the various bindings and be appended to the queue at the end of each binding. That's fine for a lot of use cases, but there's very little flexibility there: it's always just one hop -- the message being published to one exchange, with one set of bindings, and consequently one possible set of destinations. If you need something more flexible then you'd have to resort to publishing the same message multiple times. With exchange-to-exchange bindings, a message published once, can flow through any number of exchanges, with different types, and vastly more sophisticated routing topologies than previously possible.

(more…)

RabbitMQ/0MQ bridge

Monday, October 18th, 2010

Recently, Michael Bridgen and I implemented a bridge to connect the RabbitMQ broker with applications using 0MQ for messaging.

Here it is:

http://github.com/rabbitmq/rmq-0mq

So: What kind of benefit the users can get by using both products in parallel?

In short, there are two different points of view involved here. RabbitMQ users will experience different benefits than users of 0MQ. Those already using both will simply get an easy way to interconnect the two technologies.

This is RabbitMQ blog so let's start with Rabbit-focused point of view.

First and most obviously, using 0MQ as a client for RabbitMQ broker makes sense on the platforms where there's no AMQP client available. Ever fancied speaking to RabbitMQ broker from Lisp? Or -- say -- from Cobol? The bridge may come handy here.

The same applies to OS/HW platforms. Do you need to speak to the broker from a device heavily constrained on memory, with very slow CPU or limited battery life? 0MQ client is pretty concise and very fast, which means that you'll get reasonable performance even on slow chips. Effectiveness also translates to lower power consumption, which in turn makes the battery last longer. The r0mq bridge itself is collocated with the RabbitMQ broker requiring no additional resources on the client.

Another usage of the bridge is much more sophisticated. If you are using RabbitMQ in very simple scenarios you may not even appreciate it. If you are managing a big geographically distributed system, it may become a lifesaver. The bridge can be used to interconnect RabbitMQ brokers into loose federation. Just edit the configuration files for the two brokers and everything will just work. You won't have to care about order of starting up the brokers, managing the reconnections after network disruptions etc. A nice feature is that these federations are truly distributed. There's no need for central governance of the federation. Simply connect your broker to a broker in a different company. That one in turn connects to brokers in yet different companies etc. Ultimately you end up with loose world-wide federation maintained in collaborative fashion by all the participants.

Yet one more advantage you may take advantage of is efficient usage of network resources. Unlike AMQP, 0MQ allows you to split your messaging traffic into logically separate flows. You pass your entertainment video in a separate flow from commands used to keep the aircraft flying. This not only means that you'll never experience head-of-line blocking problems, but also that network-level QoS can be set separately for the entertainment channel and for the steering system. Also, network engineers will appreciate that you can monitor the network traffic flow-by-flow rather than having a big opaque chunk of bandwidth used by "messaging".

Finally, 0MQ is bundled with OpenPGM library which implements a reliable mutlicast protocol called PGM. The r0mq bridge thus allows to multicast messages from RabbitMQ broker to the clients (0MQ clients to be precise -- AMQP has no multicast support). This kind of functionality is extremely useful in scenarios where a lot of identical data is passed to many boxes on the LAN. If a separate copy of each datum is sent to each subscriber, you can easily exceed capacity of your network. With multicast, data is sent once only to all the subscribers thus keeping the bandwidth usage constant even when the number of subscribers grows.

When you looking at r0mq bridge from the other side, you are probably developing at a low level using 0MQ as your network transport.

Using RabbitMQ broker has some obvious uses. The most trivial of them is to use the broker as a bridge to connect 0MQ applications to AMQP, STOMP or XMPP applications.

However, the real use case is to use RabbitMQ as a "device" in 0MQ network. 0MQ comes with few simple precompiled devices. Some hardware can be used as 0MQ device (say IP switch in case of multicast transport). There've been some attempts to create more sophisticated devices but these are in very early stages of development. So, what 0MQ developers are missing is a full-blown, sophisticated and production-ready device.

RabbitMQ broker can serve as such a device. First of all, it has been deployed widely and thus it is stable enough to be used in production environment safely. As for the feature set, it offers much more than anything found in 0MQ world. The two most useful features are persistence and monitoring.

Persistence means that the messages passing through the broker are saved on the disk. When you shut down the broker, if the box crashes because of power outage or a different technical problem, the messages are still available on the disk. When the broker is restarted they will be sent further as if the crash hasn't happened. It is similar to how email works.

Monitoring is asked for maybe even more often than persistence. Once you have a non-trivial system you want to know what each node is doing: How many messages there are stored for a particular feed What's the current throughput? And so on. RabbitMQ can tell you these things through its command-line tools or through the management plugin.

As a conclusion I would like to stress that bridging 0MQ and RabbitMQ is not just that dumb kind of bridge you get when you bridge two incompatible but more or less equivalent products. RabbitMQ and 0MQ are focusing on different aspects of messaging. 0MQ puts much more focus on how the messages are transferred over the wire. RabbitMQ, on the other hand, focuses on how messages are stored, filtered and monitored. By combining the two technologies you can get the best from both worlds.

Enjoy!

Management plugin – preview release

Tuesday, September 7th, 2010

The previously mentioned management plugin is now in a state where it's worth looking at and testing. In order to make this easy, I've made a special once-only binary release just for the management plugin (in future we'll make binary releases of it just like the other plugins). Download all the .ez files from here and install them as described here, then let us know what you think. (Update 2010-09-22: Note that the plugins referenced in this blog post are for version 2.0.0 of RabbitMQ. We've now released 2.1.0 - for this and subsequent versions you can get the management plugin from here).

After installation, point your browser at http://server-name:55672/mgmt/. You will need to authenticate as a RabbitMQ user (on a fresh installation the user "guest" is created with password "guest"). From here you can manage exchanges, queues, bindings, virtual hosts, users and permissions. Hopefully the UI is fairly self-explanatory.

The management UI is implemented as a single static HTML page which makes background queries to the HTTP API. As such it makes heavy use of Javascript. It has been tested with recent versions of Firefox, Chromium and Safari, and with versions of Microsoft Internet Explorer back to 6.0. Lynx users should use the HTTP API directly :)

The management plugin will create an HTTP-based API at http://server-name:55672/api/. Browse to that location for more information on the API. For convenience the documentation can also be obtained from our Mercurial server.

WARNING: The management plugin is still at an early stage of development. You should be aware of the following limitations:

  • Permissions are only enforced sporadically. If a user can authenticate with the HTTP API, they can do anything.
  • Installing the management plugin will turn on fine-grained statistics in the server. This can slow a CPU-bound server by 5-10%.
  • All sorts of other features may be missing or buggy. See the TODO file for more information.

Note: if you want to build the plugin yourself, you should be aware that right now the Erlang client does not work in the default branch, so you need a mix of versions. The following commands should work:

hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
make checkout
hg update -r rabbitmq_v2_0_0 -R rabbitmq-server
hg update -r rabbitmq_v2_0_0 -R rabbitmq-codegen
hg update -r rabbitmq_v2_0_0 -R rabbitmq-erlang-client
hg clone http://hg.rabbitmq.com/rabbitmq-management
make
cd rabbitmq-management
make

Of course this will be fixed soon. (Ignore the above, this is fixed.)

Finally, this post would not be complete without some screenshots...

Management, monitoring and statistics

Friday, August 6th, 2010

For a long time the management and monitoring capabilities built into RabbitMQ have consisted of rabbitmqctl. While it's a reasonable tool for management (assuming you like the command line), rabbitmqctl has never been very powerful as a monitoring tool. So we're going to build something better.

Of course, plenty of people don't like command line tools and so several people have built alternate means of managing RabbitMQ. Alice / Wonderland and Spring AMQP are two that leap to mind. However there isn't much standardisation, and people don't always find it very convenient to talk to Rabbit via epmd. So we're going to build something easier.

With that in mind, I'd like to announce the existence of RabbitMQ Management. This is a plugin to provide management and monitoring via a RESTful interface, with a web GUI. Think of it as being like a super-Alice, and also an integration point for any other tools that people want to build.

So RabbitMQ Management will allow easier management and better monitoring. How better? Well, as part of the upcoming rabbit release we've added a statistics gathering feature to the broker. This can count messages as they're published, routed, delivered and acked, on a per channel / queue / exchange basis, so we can determine things like which channels are publishing fast or consuming slowly, which queues are being published to from which exchanges, which connections and hosts are busiest, and so on.

Of course, doing all this extra bookkeeping comes at a cost; when statistics gathering is turned on the server can run around 10% slower (assuming it's CPU-bound; which usually means handling transient messages. If it's IO bound the performance impact will likely be less). At the moment statistics gathering is turned on automatically when the management plugin is installed but we'll make it configurable.

With that, I'd like to add a warning: it's currently at a very early stage of development, and really should not be trusted for anything other than experimentation and playing with. The REST API will change, the UI will change, the plugin might crash your server, and the TODO is currently almost hilariously long. But hopefully this gives you an idea of what we're going to do.

RabbitMQ and AMQP 0-9-1

Wednesday, August 4th, 2010

Since the beginning, RabbitMQ has implemented the 0-8 version of the AMQP specification. This was the first publicly-available version, but there's been plenty of development since then. In particular, we've wanted to support the 0-9-1 version of AMQP for a while now.

For around the last year we've maintained a 0-9-1-supporting branch of the broker and clients in Mercurial, and interested users have been running that version as it matures. Well, it's finally mature and we're now merging the 0-9-1 support into the default branch. This means it'll be in the next release.

At this point, you're probably wondering what the differences are between 0-8 and 0-9-1. Well, the good news is that the differences are not huge, and are entirely sensible. 0-9-1 mostly cleans up the 0-8 spec, explaining more clearly how the broker and clients are expected to behave in certain edge cases, and removing a whole load of ambiguity and half (or less) thought out features from 0-8.

In fact, if you're running 1.8.0 or later, you're already running a broker with most of the semantic changes from 0-9-1. But the wire protocol has changed a bit too, so it matters whether you're speaking 0-9-1 or 0-8.

For the Java and Erlang clients, we're intending to simply switch to supporting AMQP 0-9-1 exclusively in the next release. The .NET library already supports multiple protocols, so we're adding 0-9-1 as another option (and making it the default). For the broker we're going to support both AMQP 0-9-1 and AMQP 0-8. For any other client libraries we encourage you to speak to the library developer ;)

This means that this time it's safe to upgrade just your broker, or your broker and all your clients, but it's not safe to upgrade your clients without upgrading your broker!

Thank you for your attention.

Well, I’ll let you go … basic.reject in RabbitMQ

Tuesday, August 3rd, 2010

Support for AMQP's basic.reject just landed on default. It's taken this long because we couldn't agree on a single set of semantics that followed the specification, was actually useful, and wasn't too complicated to implement.

First up, this is what RabbitMQ's basic.reject will do: if you supply requeue=false it will discard the message (this is in lieu of dead-lettering it, until we have a dead letter feature); if you supply requeue=true, it will release it back on to the queue, to be delivered again.

The first is useful from a error handling point of view; if your application cannot process a particular message, you can get rid of it. At the minute, it's semantically the same as just acking the message; but, given a dead-letter mechanism, it will mean unprocessable messages can be picked up elsewhere for diagnosis.

The second, with requeue=true, is useful for example if your application needs a "message lock" semantics. In this scenario, a consumer can be delivered a message, then decide not to deal with it after all, and place it back on the queue. Note that RabbitMQ doesn't take care to stop the same consumer getting the message again -- see below.

The AMQP 0-9-1 specification says a number of seemingly incompatible things about basic.reject. For a start, it says in the method description

The client MUST NOT use this method as a means of selecting messages to process.

and in the specification XML (but not in the generated PDF),

The server MUST NOT deliver the message to the same client within the context of the current channel. The recommended strategy is to attempt to deliver the message to an alternative consumer, and if that is not possible, to move the message to a dead-letter queue. The server MAY use more sophisticated tracking to hold the message on the queue and redeliver it to the same client at a later stage.

This seems to suggest that the server has to take care not to deliver the message to the same consumer twice, but consumers are not allowed to rely on this prohibition. This means basic.reject could either redeliver the message or dead-letter it, which makes it useless for the "message lock" scenario given above.

So, we have chosen to implement the simplest thing that is useful, which is to re-enqueue the message and treat it as though it were completely new. This means the consumer can receive again a message it has rejected.