Syncing

Limited-length queues

Filters

Less scrolling

AMQP 1.0 support

autoheal

(OK that last one was a bit of a stretch...)

Policy-based mirroring

You can now define queue mirroring in a much simpler and more flexible way. Your applications no longer need to know about it, and can work the same way in development and production. You can take an existing unmirrored queue and make it become mirrored with no downtime (and vice versa of course). Furthermore you can now mirror queues on to a set number of nodes in a cluster.

Faster mirroring

Mirrored queues have also become much faster - while the improvement in performance depends on a variety of factors, it is not unknown to see mirrored queues running an order of magnitude faster, or even more.

Dynamic federation

Federation has also become more flexible, transparent to applications, and dynamic. You can federate or unfederate exchanges at any time; and add, remove and reconfigure upstreams without changing anything.

New clustering commands

Clustering is now easier to set up, and more checks for cluster consistency are performed at each step in the process. It's also now possible to remove a dead node from a cluster without its cooperation.

Partition detection

Network partitions in clusters are a bad thing - so RabbitMQ will now display a large warning in the management plugin when one has occurred (this is also available in rabbitmqctl cluster_status).

Per-message TTL

As well as being able to specify a TTL for messages on a per-queue basis, individual messages can also have a TTL set when they are published.

Enabling heartbeats by default

Since so many people have experienced problems with long-running idle connections being interrupted by network infrastructure, the server now proposes AMQP heartbeats every 10 minutes by default during connection negotiation. This value can be configured here.

Memory use statistics

It's now possible to get a simple overview of where the memory used by your broker is going.

Background GC

Certain long-running processes have ended up consuming a lot of memory while waiting for garbage collection. RabbitMQ now forcibly GCs idle processes in the background, cutting down on excess memory use.

Reverse DNS lookups

Just a small thing - but you can now have RabbitMQ perform reverse DNS lookups on connecting clients to give clearer names in management and rabbitmqctl. Just set {reverse_dns_lookups, true} in your rabbitmq.config.

Web-STOMP and MQTT

We've added plugins to support STOMP-over-websockets (with SockJS support for non-websocket browsers) and MQTT 3.1. We've also added support for STOMP 1.2.

Mirror queue policies

What changed? In RabbitMQ 3.0, queue mirroring is no longer controlled by the x-ha-policy argument when declaring a queue. Your applications can continue to declare this argument, but it won't cause queues to be mirrored. Instead you can declare one or more policies which control which queues are mirrored, and how.

Why did it change? As anyone who's used mirrored queues will tell you, requiring applications to know which queues are mirrored is a pain. The new approach puts configuration in the broker, where it belongs, and also supports changing mirroring policy at any time.

What should I do? You need to make sure your queues are still mirrored. For the full documentation see here, but if you just want to make sure that all queues (except those with auto-generated names) are mirrored across all nodes, run:

rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}'

New federation

What changed? Federation is configured quite differently in RabbitMQ 3.0. The x-federation exchange type no longer exists; instead normal exchanges are made federated by policy in the same way that HA queues are. Furthermore, upstreams are defined dynamically as well.

Why did it change? Again, your applications should not need to know about federation. Federation configuration in rabbitmq.config was complicated and confused many people. And needing to restart the broker to add a new upstream was not fun.

But I have a working federation setup! You broke it. Migrating to the new way of doing federation will take a bit of work. In the mean time you can use the rabbitmq_old_federation plugin. This is a backport of the 2.8.7 federation plugin for RabbitMQ 3.0. To use it:

rabbitmq-plugins disable rabbitmq_federation

rabbitmq-plugins enable rabbitmq_old_federation

and then edit your rabbitmq.config file so that the rabbitmq_federation section is renamed to rabbitmq_old_federation.

New clustering

What changed? The clustering-setup commands in rabbitmqctl have changed.

Why did it change? The old ones were not very user friendly.

What do I need to do? If you have an existing cluster, nothing. If you write scripts to create clusters, you will need to edit them. In particular, rabbitmqctl cluster should be replaced with rabbitmqctl join_cluster, but:

• You don't need to invoke rabbitmqctl reset first
• You don't need to list all the nodes on the command line; if you give more than one node then they will be taken as a list of nodes to try to cluster with
• Whether the new node is a disc or RAM node is determined by the --disc and --ram flags. The default is to be a disc node.

For more details, see the documentation.

Removal of "immediate" flag

What changed? We removed support for the rarely-used "immediate" flag on AMQP's basic.publish.

Why on earth did you do that? Support for "immediate" made many parts of the codebase more complex, particularly around mirrored queues. It also stood in the way of our being able to deliver substantial performance improvements in mirrored queues.

What do I need to do? If you just want to be able to publish messages that will be dropped if they are not consumed immediately, you can publish to a queue with a TTL of 0.

If you also need your publisher to be able to determine that this has happened, you can also use the DLX feature to route such messages to another queue, from which the publisher can consume them.

frame_max

What changed? The RabbitMQ server now disconnects clients which send frames larger than the negotiated frame_max setting for the connection.

Why did it change? Malicious (or badly written) clients could send arbitrarily large frames and cause the server to run out of memory.

Why do I care? Unfortunately some clients don't implement AMQP framing correctly. RabbitMQ 3.0 will allow clients to exceed frame_max by a fudge factor of a few bytes (to allow for off by one errors and incorrectly excluding the frame header) but if your client has broken framing you will be disconnected after trying to send a message larger than frame_max (which by default comes out to 128kb; see the documentation on how to raise this).

Management and JSON-RPC channel port changes

What changed? The management plugin now listens on port 15672, not 55672. JSON-RPC channel now listens on 15670, not 55670.

Why did it change? The old ports were in the ephemeral port range on many operating systems, meaning that web browsers and other client applications might use these ports arbitrarily. You're not supposed to listen on these ports.

In particular we noticed that the management plugin web UI could, when pointed at a stopped broker on localhost, end up getting the browser to connect to itself on port 55672. This prevented the broker from starting again.

What do I have to do? Hopefully nothing. RabbitMQ will attempt to open the old port and send HTTP redirects to the new one. But if you're using an application other than a web browser to talk to the HTTP API, it might not support HTTP redirects. If it doesn't, you'll need to point it at the new port.

Note that rabbitmqadmin prior to version 3.0 was such an application. Oops.

Also note that the STOMP plugin still listens on port 61313. Although this is in the ephemeral range, it's the closest thing STOMP has to a standard port, so we have to stick with it.

expiration property

What changed? We now expect the expiration field in message properties to be parseable as an integer if it's set at all.

Why did it change? In order to support per-message TTL we need a place to get the TTL of the message from, and this is the obvious place. Unfortunately the AMQP standard defines it as a string, so we try to parse it as an integer and will throw a channel exception if it is not.

What do I have to do? Make sure that if you're using that property then you're using it because you expect RabbitMQ to expire the message, and make sure it's set to a string which can be parsed as an integer.

Supported features

QoS 0 and QoS 1

The MQTT adapter supports QoS 0 (at most once) and QoS 1 (at least once) semantics. In MQTT QoS relates to transport assurance as well as persistence - the plugin honours both. While clients are permitted to request QoS 2 subscriptions, the adapter will only grant subscriptions up to QoS 1.

Last Will and Testament (LWT)

Clients can provide a LWT message during connection that will only be published if the client disconnects unexpectedly, e.g. due to a network failure.

Sticky sessions

Clients can make use of sticky (or non-clean) sessions to ensure they receive messages that were published whilst they were disconnected.

Default logins

Default authentication details can be optionally be configured so that the MQTT adapter authenticates to the RabbitMQ broker as a default user in case a connecting MQTT client provides no login details.

Extended features

While not explicitly enumerated, all core broker features are available to MQTT clients.

MQTT subscription wildcards are limited in that they may only appear as a suffix. AMQP topics are not limited in this way, so wildcards can appear in any position. The MQTT adapter implements the more flexible AMQP patterns, but with MQTT syntax.

The MQTT specification does not mention SSL or any interaction between SSL and authentication. The MQTT adapter includes SSL capability now, with the possibility of integrating  certificates with authentication on the future.

The MQTT concept of "bridging" can be realised with RabbitMQ's federation by federating the exchanges that the MQTT adapter publishes to.

Future features

AMQP 0-9-1 does not define "exactly once" semantics for message delivery. For this reason the MQTT adapter does not support publishing messages at the QoS 2 (exactly once) level, or exchanging PUBREC, PUBREL or PUBCOMP messages with clients.

"Retained  messages" is an MQTT feature where the broker retains flagged messages and delivers them to future subscribing clients. E.g. in a topic for sensor readings, a retained message allows a client to receive the last reading without  needing to wait for the next reading. By default AMQP 0-9-1 exchanges do not retain any message state. Therefore the MQTT adapter makes no attempt to honour the "Retained" flag, which will be silently ignored.

These are areas where we are especially interested in obtaining feedback from the community. There is scope to enhance the core broker with these features not only for MQTT clients, but potentially for (extended) AMQP and other clients as well - provided there is sufficient demand.

Interoperability with other MQTT implementations

The MQTT adapter has been successfully tested with the MQTT clients of the following products, when restricting operation to the supported features:

• Really Small Message Broker
• Mosquitto
• Paho
• WebSphere MQ

Interoperability with other protocols

The MQTT adapter uses one configurable exchange for publishing, and subscriptions are implemented as AMQP bindings. In combination these allow interoperability with any clients that know the name of the exchange or the topics used by MQTT clients.

Installation

First make sure you have  rabbitmq-server 2.8.6 installed. (The plugin should also be compatible with other v2.8.x releases.)

The MQTT adapter is currently available as a preview release. You must download and install the plugin manually until it  is included as a regular plugin in a future release. The plugin can be downloaded from the preview release downloads, e.g.

 wget http://www.rabbitmq.com/releases/plugins/v2.8.6-mqtt-preview/rabbitmq_mqtt-2.8.6.ez

The *.ez file must be copied to the plugins directory. On my Debian-based workstation this is in /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.6/plugins:

 sudo cp rabbitmq_mqtt-2.8.6.ez /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.6/plugins

Enable the plugin using rabbitmq-plugins:

 sudo rabbitmq-plugins enable rabbitmq_mqtt

Restart the rabbitmq server

 sudo /etc/init.d/rabbitmq-server restart 

The broker logfile should now include a new line indicating that it is ready to accept MQTT connections:

  =INFO REPORT==== 12-Sep-2012::14:21:26 ===
  started MQTT TCP Listener on [::]:1883

The default configuration options should work fine  in most cases. A description of all configuration options is included in the readme. You will need to provide further configuration if you wish to set up SSL, or define a different exchange in order to facilitate federation.

You can now try to execute the included tests (based on the Java Paho client library), or your own MQTT application.

See the contact page for details on how to provide feedback.

Today we'd like to say thank-you to Jason and Alvaro.  Thank-you Jason and Alvaro!  You did an amazing job and infinite beers are on us.

But there's more...  Manning have kindly offered a promotional discount of 37% to readers of this blog.  All is revealed below, in a guest post by Jason Williams himself...

RabbitMQ in Action is here

Well, it's finally here. After 18 months of writing, re-writing and updating, RabbitMQ in Action is finished and in the flesh. It's hard to believe that when we started, RabbitMQ was at version 1.8.0 and now we're at 2.8.2. So much has changed in Rabbit that required rewrites of whole sections along the way, that it feels like we're really at 5 or 6.0. It's a testament to the Rabbit team members that helped us that the book kept pace with it all. So now that it's out why should you read it (besides the 37% discount code below)?

If you feel like you want a deeper understanding than the online tutorials offer, we wrote this for you. Whether it's figuring out clustering and mirrored queues, or just getting a better understanding of messaging fabrics (queues, bindings and routing exchanges, relays and federations.), our goal was to write the book we wished had existed when we started, and that we hope will help you. From the management console and API to building real world applications and plugins, we've tried to cover everything you need to get a good foundation of Rabbit under your belt…and hopefully that you can use as a desk reference too.

Lots of example code on Github to get you started

One thing we tried to focus on was using RabbitMQ to link together different applications written in completely different languages.  That's one of the main reasons we wrote the examples in Python and PHP. However, we had two other reasons also:

1.) Python reads almost like pseudo-code and produces incredibly readable programs…which makes it an excellent teaching language. You can focus on what the example program's doing, without a lot of class declarations and boiler plate clouding up the works.

2.) There are a ton of books on messaging targeted at Java and the old-line enterprise brokers. We wanted to write something different... something that was easier to read and more accessible to people without any background in messaging. RabbitMQ in Action is very much a book for people of all languages and backgrounds. Writing in Python and PHP helped us do that (there's appendices on using Rabbit with Java and .NET too).

With that last one in mind, we’ve done something a little different than other Manning books…all of our examples are in a public repo on Github.

We did this so that if you feel like converting the examples into the language of your choice to help those like you, you can. As long as the license on your contribution is BSD, we'll merge in your pull requests and hopefully build a huge library of RabbitMQ examples that can help everyone. There are already Ruby versions of the examples merged in!

So if those aren't reasons enough to give RabbitMQ in Action a shot…how about a 37% discount just because you read this blog?

Save 37% on RabbitMQ in Action with Promotional Discount Code 12rmqb when you checkout at the Manning web site.

For quite a while here, at RabbitMQ headquarters, we were struggling to find a good way to expose messaging in a web browser. In the past we tried many things ranging from the old-and-famous JsonRPC plugin (which basically exposes AMQP via AJAX), to Rabbit-Socks (an attempt to create a generic protocol hub), to the management plugin (which can be used for basic things like sending and receiving messages from the browser).

Over time we've learned that the messaging on the web is very different to what we're used to. None of our attempts really addressed that, and it is likely that messaging on the web will not be a fully solved problem for some time yet.

That said, there is a simple thing RabbitMQ users keep on asking about, and although not perfect, it's far from the worst way do messaging in the browser: exposing STOMP through Websockets.

STOMP + Websockets

We're delighted to introduce a new plugin for RabbitMQ:

• RabbitMQ-Web-Stomp

It is basically a bridge between RabbitMQ-STOMP plugin and a Websockets server (SockJS). Hopefully, it is a decent solution for a set of rabbit-on-the-web use cases.

What it actually does

Within RabbitMQ

RabbitMQ-Web-Stomp is a simple beast. It takes the STOMP protocol as provided by the RabbitMQ-STOMP plugin and exposes it using the SockJS server.

Wire protocol

One can connect to the SockJS endpoint from any browser using the reliable SockJS protocol. This will work even in browsers that don't support native websockets or in environments behind broken proxies that block non-http transports.

Alternatively, for users that don't need this level of sophistication, SockJS exposes a raw websockets url that can be accessed directly from a recent, websocket-capable browser.

In the browser

Within the browser, the connection to SockJS endpoint is basically a raw STOMP connection. You can send and receive normal STOMP frames.

Any decent javascript STOMP library should be able to handle that. For our examples we're using the stomp-websocket library by Jeff Mesnil and Jeff Lindsay.

We use this code in the examples:

<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"></script>
<script src="stomp.js"></script>
<script>
   WebSocketStompMock = SockJS;

    var client = Stomp.client('http://127.0.0.1:55674/stomp');
    [...]

Installation

Rabbitmq-Web-Stomp is an experimental plugin. It's not distributed with vanilla RabbitMQ releases; you need to install it manually.

1. You need at least Erlang R14 (more info).
2. You need Rabbitmq-server 2.8.2 installed
3. Grab the needed erlang plugin .ez files:

   wget \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/cowboy-0.5.0-rmq2.8.2-git4b93c2d.ez \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/sockjs-0.2.1-rmq2.8.2-gitfa1db96.ez \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/rabbitmq_web_stomp-2.8.2.ez \
   
   http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/rabbitmq_web_stomp_examples-2.8.2.ez

4. Next, copy them to the plugins directory. For example, on my Ubuntu box this will be:

   sudo cp *.ez /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.2/plugins
   

5. Now, you're ready to enable them using rabbitmq-plugins:

   sudo rabbitmq-plugins enable rabbitmq_web_stomp
   sudo rabbitmq-plugins enable rabbitmq_web_stomp_examples
   

6. Restart the rabbitmq server. On ubuntu:

   sudo /etc/init.d/rabbitmq-server restart
   

As you may have noticed, we enabled two plugins:

• First, RabbitMQ-web-stomp, which is the main plugin. It exposes SockJS endpoint on port 55674, like: http://127.0.0.1:55674/stomp.
• Second, RabbitMQ-web-stomp-examples, which only hosts a few javascript and html files with examples. This is accessible under: http://127.0.0.1:55670/.

Keep in mind, that RabbitMQ-web-stomp depends on RabbitMQ-STOMP which by default will bind to port 61613.

The usage

If you enabled RabbitMQ-web-stomp-examples plugin, you should be able to instantly run two examples prepared by us. Just open a web browser at http://127.0.0.1:55670/.

• "echo" - shows how to use STOMP to do simple message broadcasting (source)
• "bunny" - example of a simple collaboration canvas painting app (source)

Summary

RabbitMQ-web-stomp is quite a simple plugin, but opens wide possibilities, exposing the STOMP protocol to the browser.

Like always, feedback welcome. We're also looking for inspiration for more examples!

So, the default QoS prefetch setting gives clients an unlimited buffer, and that can result in poor behaviour and performance. But what should you set the QoS prefetch buffer size to? The goal is to keep the consumers saturated with work, but to minimise the client's buffer size so that more messages stay in Rabbit's queue and are thus available for new consumers or to just be sent out to consumers as they become free.

Let's say it takes 50ms for Rabbit to take a message from this queue, put it on the network and for it to arrive at the consumer. It takes 4ms for the client to process the message. Once the consumer has processed the message, it sends an ack back to Rabbit, which takes a further 50ms to be sent to and processed by Rabbit. So we have a total round trip time of 104ms. If we have a QoS prefetch setting of 1 message then Rabbit won't sent out the next message until after this round trip completes. Thus the client will be busy for only 4ms of every 104ms, or 3.8% of the time. We want it to be busy 100% of the time.

If we do total round trip time / processing time on the client for each message, we get 104 / 4 = 26. If we have a QoS prefetch of 26 messages this solves our problem: assume that the client has 26 messages buffered, ready and waiting for processing. (This is a sensible assumption: once you set basic.qos and then consume from a queue, Rabbit will send as many messages as it can from the queue you've subscribed to to the client, up to the QoS limit. If you assume messages aren't very big and bandwidth is high, it's likely Rabbit will be able to send messages to your consuming client faster than your client can process them. Thus it's reasonable (and simpler) to do all the maths from the assumption of a full client-side buffer.) If each message takes 4ms of processing to deal with then it'll take a total of 26 * 4 = 104ms to deal with the entire buffer. The first 4ms is the client processing of the first message. The client then issues an ack and goes on to process the next message from the buffer. That ack takes 50ms to get to the broker. The broker than issues a new message to the client, which takes 50ms to get there, so by the time 104ms has passed and the client has finished processing its buffer, the next message from the broker has already arrived and is ready and waiting for the client to process it. Thus the client remains busy all the time: having a bigger QoS prefetch will not make it go faster; but we minimise the buffer size and thus latency of messages in the client: messages are buffered by the client for no longer than they need to be in order to keep the client saturated with work. In fact, the client is able to fully drain the buffer before the next message arrives, thus the buffer actually stays empty.

This solution is absolutely fine, provided processing time and network behaviour remains the same. But consider what happens if suddenly the network halves in speed: your prefetch buffer is no longer big enough and now the client will sit idle, waiting for new messages to arrive as the client is able to process messages faster than Rabbit can supply fresh messages.

To address this problem, we might just decide to double (or nearly double) the QoS prefetch size. If we push it to 51 from 26, then if the client processing remains at 4ms per message, we now have 51 * 4 = 204ms of messages in the buffer, of which 4ms will be spent processing a message, leaving 200ms for the sending an ack back to Rabbit and receiving the next message. Thus we can now cope with the network halving in speed.

But if the network's performing normally, doubling our QoS prefetch now means each message will sit in the client side buffer for a while, instead of being processed immediately upon arrival at the client. Again, starting from a full buffer of now 51 messages we know that new messages will start appearing at the client 100ms after the client finishes processing the first message. But in those 100ms, the client will have processed 100 / 4 = 25 messages out of the 50 available. Which means as a new message arrives at the client, it'll be added to the end of the buffer as the client removes from the head of the buffer. The buffer will thus always stay 50 - 25 = 25 messages long and every message will thus sit in the buffer for 25 * 4 = 100ms, increasing the latency between Rabbit sending it to the client and the client starting to process it from 50ms to 150ms.

Thus we see that increasing the prefetch buffer so that the client can cope with deteriorated network performance whilst keeping the client busy, substantially increases the latency when the network is performing normally.

Equally, rather than the network's performance deteriorating, what happens if the client starts taking 40ms to process each message rather than 4ms? If the queue in Rabbit was previously at a steady length (i.e. ingress and egress rates were the same), it'll now start growing rapidly, as the egress rate has dropped to a tenth of what it was. You might decide to try and work through this growing backlog by adding more consumers, but there are messages now being buffered by the existing clients. Assuming the original buffer size of 26 messages, the client will spend 40ms processing the first message, will then send the ack back to Rabbit and move onto the next message. The ack still takes 50ms to get to Rabbit and a further 50ms for Rabbit to send out a new message, but in that 100ms, the client has only worked through 100 / 40 = 2.5 further messages rather than the remaining 25 messages. Thus the buffer is at this point 25 - 3 = 22 messages long. The new message arriving from Rabbit, rather than being processed immediately, now sits in 23rd place, behind 22 other messages still waiting to be processed, and will not be touched by the client for a further 22 * 40 = 880ms. Given the network delay from Rabbit to the client is only 50ms, this additional 880ms delay is now 95% of the latency (880 / (880 + 50) = 0.946).

Even worse, what happens if we doubled the buffer size to 51 messages in order to cope with network performance degradation? After the first message has been processed, there will be 50 further messages buffered in the client. 100ms later (assuming the network is running normally), a new message will arrive from Rabbit, and the client will be half way through processing the 3rd of those 50 messages (the buffer will now be 47 messages long), thus the new message will be 48th in the buffer, and will not be touched for a further 47 * 40 = 1880ms. Again, given the network delay of getting the message to the client is only 50ms, this further 1880ms delay now means client side buffering is responsible for over 97% of the latency (1880 / (1880 + 50) = 0.974). This may very well be unacceptable: the data may only be valid and useful if it's processed promptly, not some 2 seconds after the client received it! If other consuming clients are idle, there's nothing they can do: once Rabbit has sent a message to a client, the message is the client's responsibility until it acks or rejects the message. Clients can't steal messages from each other once the message has been sent to a client. What you want is for clients to be kept busy, but for clients to buffer as few messages as possible so that messages are not delayed by client-side buffers and thus new consuming clients can be quickly fed with messages from Rabbit's queue.

So, too small a buffer results in clients going idle if the network gets slower, but too big a buffer results in lots of extra latency if the network performs normally, and huge amounts of extra latency if the client suddenly starts taking longer to process each message than normal. It's clear that what you really want is a varying buffer size. These problems are common across network devices and have been the subject of much study. Active Queue Management algorithms seek to try and drop or reject messages so that you avoid messages sitting in buffers for long periods of time. The lowest latency is achieved when the buffer is kept empty (each message suffers network latency only and does not sit around in a buffer at all) and buffers are there to absorb spikes. Jim Gettys has been working on this problem from the point of view of network routers: differences between performance of the LAN and the WAN suffer exactly the same sorts of problems. Indeed whenever you have a buffer between a producer (in our case Rabbit) and a consumer (the client-side application logic) where the performance of both sides can vary dynamically, you will suffer these sorts of problems. Recently a new algorithm called Controlled Delay has been published which appears to work well in solving these problems.

The authors claim that their CoDel ("coddle") algorithm is a "knob free" algorithm. This is a bit of a lie really: there are two knobs and they do need setting appropriately. But they don't need changing every time performance changes, which is a massive benefit. I have implemented this algorithm for our AMQP Java Client as a variant of the QueueingConsumer. Whilst the original algorithm is aimed at the TCP layer, where it's valid to just drop packets (TCP itself will take care of re-transmission of lost packets), in AMQP that's not so polite! As a result, my implementation uses Rabbit's basic.nack extension to explicitly return messages to the queue so they can be processed by others.

Using it is pretty much the same as the normal QueueingConsumer except that you should provide three extra parameters to the constructor to get the best performance.

1. The first is requeue which says whether, when messages are nacked, should they be requeued or discarded. If false, they will be discarded which may trigger the dead letter exchange mechanisms if they're set up.
2. The second is the targetDelay which is the acceptable time in milliseconds for messages to wait in the client-side QoS prefetch buffer.
3. The third is the interval and is the expected worst case processing time of one message in milliseconds. This doesn't have to be spot on, but within an order of magnitude certainly helps.

You should still set a QoS prefetch size appropriately. If you do not, what is likely is that the client will be sent a lot of messages, and the algorithm will then have to return them to Rabbit if they sit in the buffer for too long. It's easy to end up with a lot of extra network traffic as messages are returned to Rabbit. The CoDel algorithm is meant to only start dropping (or rejecting) messages once performance diverges from the norm, thus a worked example might help.

Again, assume network traversal time in each direction of 50ms, and we expect the client to spend 4ms on average processing each message, but this can spike to 20ms. We thus set the interval parameter of CoDel to 20. Sometimes the network halves in speed, so the traversal time can be 100ms in each direction. To cater for that, we set the basic.qos prefetch to 204 / 4 = 51. Yes, this means that the buffer will remain 25 messages long most of the time when the network is running normally (see workings earlier), but we decide that's OK. Each message will thus sit in the buffer for an expected 25 * 4 = 100ms, so we set the targetDelay of CoDel to 100.

When things are running normally, CoDel should not get in the way, and few if any messages should be being nacked. But should the client start processing messages more slowly than normal, CoDel will spot that messages have been buffered by the client for too long, and will return those messages to the queue. If those messages are requeued then they will become available for delivery to other clients.

This is very much experimental at the moment, and it's possible to see reasons why CoDel isn't as appropriate for dealing with AMQP messages as it is for plain IP. It's also worth remembering that requeuing messages via nacks is a fairly expensive operation, so it's a good idea to set the parameters of CoDel to ensure in normal operation very few if any messages are being nacked. The management plugin is an easy way to inspect how many messages are being nacked. As ever, comments, feedback and improvements are most welcome!

Welcome back! Last time we talked about flow control and latency; today let's talk about how different features affect the performance we see. Here are some simple scenarios. As before, they're all variations on the theme of one publisher and one consumer publishing as fast as they can.

Some Simple Scenarios

Of course to some extent this quest for large numbers is a bit silly, we're more interested in relative performance. So let's revert to one producer and one consumer.

Message Sizes

Notably, all the messages we've been sending until now have only been a few bytes long. There are a couple of reasons for this:

• Quite a lot of the work done by RabbitMQ is per-message, not per-byte-of-message.
• It's always nice to look at big numbers.

But in the real world we will often want to send bigger messages. So let's look at the next chart:

1 -> 1 sending rate message sizes

Here (again) we're sending unacked / unconfirmed messages as fast as possible, but this time we vary the message size. We can see that (of course) the message rate drops further as the size increases, but the actual number of bytes sent increases as we have less and less routing overhead.

So how does the message size affect horizontal scaling? Let's vary the number of producers with different message sizes. Just for a change, in this test we're not going to have any consumers ar all.

n -> 0 sending msg rate vs number of producers, for various message sizes

n -> 0 sending bytes rate vs number of producers, for various message sizes

In these tests we can see that for small messages it only takes a couple of producers to reach an upper bound on how many messages we can publish, but that for larger messages we need more producers to use the available bandwidth.

Another frequently confusing issue is performance around consumers with a prefetch count. RabbitMQ (well, AMQP) defaults to sending all the messages it can to any consumer that looks ready to accept them. The maximum number of these unacknowledged messages per channel can be limited by setting the prefetch count. However, small prefetch counts can hurt performance (since we can be waiting for acks to arrive before sending out more messages).

So let's have a look at prefetch count and, while we're there, also consider the number of consumers consuming from a single queue. This chart contains some deliberately absurd extremes.

1 -> n recving rate vs consumer count / prefetch count

The first thing to notice is that tiny prefetch counts really hurt performance. Note the large difference in performance between prefetch = 1 and prefetch = 2! But we also get into diminishing returns - notice that the difference between prefetch = 20 and prefetch = 50 is hard to see, and the difference between prefetch = 50 and prefetch = 10000 is almost invisible. Of course, this is because for our particular network link prefetch = 50 already ensures that we never starve the consumer while waiting for acks. Of course, this test was run over a low latency link - more latent links will benefit from a higher prefetch count.

The second thing to notice is that when we have a small number of consumers, adding one more will increase performance (we get more parallellism). And with a tiny prefetch count, increasing consumers even up to a large number has benefits (since each individual consumer spends much of its time starved). But when we have a larger prefetch count, increasing the number of consumers is not so helpful, since even a small number can kept busy enough to max out our queue, but the more consumers we have the more work RabbitMQ has to do to keep track of all of them.

Large queues

All the examples we've looked at so far have one thing in common: very few messages actually get queued. In general we've looked at scenarios where messages get consumed as quickly as they get produced, and thus each queue has an average length of 0.

So what happens whe queues get big? When queues are small(ish) they will reside entirely within memory. Persistent messages will also get written to disc, but they will only get read again if the broker restarts.

But when queues get larger, they will get paged to disc, persistent or not. In this case performance can take a hit as suddenly we need to access the disc to send messages to consumers. So let's run a test: publish a lot of non-persistent messages to a queue, and then consume them all.

Queue load / drain 500k messages

In this small case we can see fairly consistent performance: the messages go into the queue fairly quickly and then come out even more quickly.

Queue load / drain 10M messages

But when we have a larger queue we see that the performance varies a lot more. We see that when loading the queue we initially get a very high throughput, then a pause while some of the queue is paged out to disc, then a more consistent lower throughput. Similarly when draining the queue we see a much lower rate when pulling the messages from disc.

Performance of disc-bound queues is a complex topic - see Matthew's blog post on the subject for some more talk on the subject.

As a co-sponsor we had one of the iPad prizes to award. We decided to allow hacks that used one or more of RabbitMQ, SockJS, or Cloud Foundry. This meant that about half of the twenty-seven hacks were eligible when it came to judging, making the choice rather difficult.

In the end we chose YouChoose, which uses SockJS and is hosted on CloudFoundry. The idea was nifty, and the hack complete and well-executed; and it let one of the judges, who shall remain unnamed, subject the audience to the Titanic theme (until they worked out how to collectively vote it off). Empowering stuff.

(photo by Olly Plumstead)

All the hacks were imaginative and almost all made it to a working demo stage -- which was pretty amazing (or perhaps explained by) considering the lack of sleep some people were suffering on Sunday.

Among our favourites were: Err, microgames with location-based matchmaking (one of the games is "run away as fast as you can"); The Worm, a live popularity chart with optional rageface meter; and Support Net, a site and mobile app for people quitting smoking.

Full coverage of the event is over on GoSquared's blog, including video footage and interviews.

Pusher.com are diversifying

So today I would like to talk about some aspects of RabbitMQ's performance. There are a huge number of variables that feed into the overall level of performance you can get from a RabbitMQ server, and today we're going to try tweaking some of them and seeing what we can see.

The aim of this piece is not to try to convince you that RabbitMQ is the fastest message broker in the world - it often isn't (although we like to think we're still pretty decent) - but to give you some ideas about what sort of performance you can expect in different situations.

All the charts and statistics shown were measured on a PowerEdge R610 with dual Xeon E5530s and 40GB RAM. Largely because it was the fastest machine we had lying around. One major thing that's not ideal is we ran the clients on the same machine as the server - just due to the limited hardware we had available. We used RabbitMQ 2.8.1 (in most cases) and Erlang R15B with HiPE compilation enabled.

By the way, the code to produce all these statistics is available in branch bug24527 of rabbitmq-java-client (although it's currently rather rough) Eventually it will get merged to default, and also become easier to work with. We hope.

Flow control in RabbitMQ 2.8.0+

But first of all I need to introduce a new feature in RabbitMQ 2.8.0+ - internal flow control. RabbitMQ is internally made up of a number of Erlang processes which pass messages to each other. Each process has a mailbox which contains messages it has received and not yet handled. And these mailboxes can grow to an unbounded size.

What this means is that unless the first process to receive data off a network socket is the slowest in the chain, (it's not) then when you have a heavily-loaded RabbitMQ server messages can build up in process mailboxes forever. Or rather, until we run out of memory. Or rather, until the memory alarm goes off. At which point the server will stop accepting new messages while it sorts itself out.

The trouble is, this can take some time. The following chart (the only one in this post made against RabbitMQ 2.7.1) shows a simple process that publishes small messages into the broker as fast as possible, and also consumes them as fast as possible, with acknowledgement, confirms, persistence and so on all switched off. We plot the sending rate, the receiving rate, and the latency (time taken for a sent message to be received), over time. Note that the latency is a logarithmic scale.

Simple 1 -> 1 autoack (2.7.1)

Ouch! That's rather unpleasant. Several things should be obvious:

• The send rate and receive rate fluctuate quite a lot.
• The send rate drops to zero for two minutes (this is the first time the memory alarm went off). In fact the memory alarm goes off again at the end.
• The latency increases steadily (and look at the scale - we show microseconds, but we could just as easily measure it in minutes).

(The small drop in latency around 440s is due to all the messages published before 200s being consumed, and the long gap afterwards.)

Of course, this is only the sort of behaviour you would expect when stressing a server to the limit. But we're benchmarking - we want to do that. And anyway, servers get stressed in production too.

So now let's look at the same experiment conducted against a RabbitMQ 2.8.1 server:

Simple 1 -> 1 autoack (2.8.1)

That looks like a much calmer experience! The send rate, receive rate and latency are all near-constant. The reason is internal flow control. The latency is around 400ms (which is still quite high compared to a less loaded server for reasons I'll discuss in a minute).

These charts don't show memory consumption, but the story is the same - in this circumstance 2.7.1 will eat lots of memory and bounce off the memory alarm threshold, and 2.8.1 will use a fairly constant, fairly low quantity of memory.

Each process in the chain issues credit to the processes that can send messages to it. Processes consume credit as they send messages, and issue more credit as they receive them. When a process runs out of credit it will stop issuing more to its upstream processes. Eventually we reach the process which is reading bytes off a network socket. When that process runs out of credit, it stops reading until it gets more. This is the same as when the memory alarm goes off for the 2.7.1 broker, except that it happens many times per second rather than taking minutes, and we control memory use a lot more.

So where does that 400ms latency come from? Well, there are still messages queueing up at each stage in the pipeline, so it takes a while for a message to get from the beginning to the end. That accounts for some of the latency. However, most of it comes from an invisible "mailbox" in front of the entire server - the TCP buffers provided by the operating system. On Linux the OS will allow up to 8MB of messages to back up in the TCP stack. 8MB doesn't sound like a lot of course, but we're dealing with tiny messages (and each one needs routing decisions, permissions check and so on to be made).

But it's important to remember that we tend to see the worst latency when running at the limit of what we can do. So here's one final chart for this week:

1 -> 1 sending rate attempted vs latency

Note that the horizontal axis is no longer time. We're now showing the results of many runs like the ones above, with each point representing one run.

In the charts above we were running as fast as we can, but here we limit the rate at varying points up to the maximum rate we can achieve. So the yellow line shows rate attempted vs rate achieved - see that it goes most of the way purely 1:1 linearly (when we have spare capacity and so if we try to publish faster we will succeed) and then stops growing as we reach the limit of what we can do.

But look at the latency! With low publishing rates we have latency of considerably less than a millisecond. But this drifts up as the server gets busier. As we stop being able to publish any faster, we hit a wall of latency - the TCP buffers start to fill up and soon messages are taking hundreds of milliseconds to get through them.

So hopefully we've shown how RabbitMQ 2.8.1 offers much more reliable performance when heavily loaded than previous versions, and shown how latency can reach for the skies when your message broker is overloaded. Tune in next time to see how some different ways of using messaging affect performance!