Here at RabbitMQ HQ we spend quite a lot of time arguing. Occasionally, it's about important things, like what messaging really means, and the range of different APIs that can be used to achieve messaging. RabbitMQ and AMQP present a very explicit interface to messaging: you very much have verbs send and receive and you need to think about what your messaging patterns are. There's a lot (of often quite clever stuff) going on under the bonnet but nevertheless, the interface is quite low-level and explicit, which gives a good degree of flexibility. Sometimes though, that style of API is not the most natural fit for the problem you're trying to solve - do you really reach an impasse and think "What I need here is an AMQP-message broker", or do you, from pre-existing knowledge, realise that you could choose to use an AMQP-message broker to solve your current problem?

AtomizeJS exists at the opposite end of the spectrum. There is lots of messaging involved, but you almost never get to see any of it. Instead, you write transactions in JavaScript that modify objects, and those objects are shared between all clients that are connected to the same AtomizeJS server. The API that you're given lets you do slightly more powerful things than you're used to from database transactions, in particular, retry allows you to abort a transaction but then restart it automatically once someone else has changed one of the variables you read. This means you have the observer-pattern, and from that you can then build any explicit messaging patterns you want. In most cases, I doubt you'll be building APIs that say send or receive, instead you'll be building richer data-structures - work queues, shared dictionaries etc. The question to pose then is: is it easier to build these things based on top of a transaction-like API such as offered by AtomizeJS, or on top of an explicit messaging API such as offered by RabbitMQ and AMQP brokers. There is no one solution and horses-for-courses etc, but please leave your thoughts below.

The gain that AtomizeJS provides is not so much in the use of STM from the browser, but the use of STM against a distributed object. This allows you to trivially share state between browsers, modify it safely in intuitive terms, and thus build your applications with little or no application-specific server-side code. Currently, it's a little clunky to use with browsers that don't support bleeding-edge JavaScript features (though I've provided some tooling to try and mitigate this), and everything does work with the latest versions of Chrome, Firefox, IE, Safari, and Opera. Please have a go and let us know what you think!

SockJS version 0.2 has been released:

• https://groups.google.com/group/sockjs/browse_thread/thread/79893c8545c49f06

You can test it in the usual playground:

• http://sockjs.popcnt.org/example-cursors.html
• http://sockjs.cloudfoundry.com/tests-qunit.html

Apart from ton of general updates and few API tweaks, SockJS 0.2 contains two major features:

• Faster connection times - due to a better fallback-detection algorithm.
• Raw websocket api - should make it easier to write command line clients for SockJS.

That means two out of three updates to SockJS protocol I proposed about a month ago are done. The last major feature remaining is binary data support.

Unfortunately the releases rarely go smoothly, but thanks to alert SockJS users the problem was quickly spotted and fixed.

So, happy playing with SockJS!

plugins

Prior to 2.7.0 if you wanted to use a plugin then a .ez file was placed in the plugins directory, and the broker restarted. Any plugin found in this directory was installed on startup.  This meant two things: plugins weren't supplied with the server (even our supported ones), and installing or uninstalling a plugin involved moving files about and ensuring other plugin dependencies were installed. Administration of plugins was unnecessarily messy.

In 2.7.0, we introduced a command rabbitmq-plugins, to enable or disable any plugin in the plugins directory. Simply issue the command rabbitmq-plugins list to see what plugin files are there, and rabbitmq-plugins enable <plugin-name> to use one the next time the broker starts. No files need to be moved around -- they can stay in the plugins directory all the time -- and all the rabbit plugins are now supplied with the server, disabled by default.  The command also understands which plugins depend upon which others, and enables dependencies automatically. Using and managing plugins is now much easier. See the plugins page for more.

In 2.7.1 there is a fix to the consistent-hash-exchange plugin which mis-routed messages when handling multiple exchanges.

connect by URI

All of the rabbit clients (.NET, Java and Erlang) clients accept an amqp URI scheme for connections. This is a convenient one-stop shop allowing the username and password, hostname and port, and virtual host, to be supplied by a single URI or String parameter. For example:

amqp://guest:ghost@rabbit01.coderus.moc:5672/vhost01

See the individual client APIs for details.

Java threads

In Release 2.7.0 the threading structure of the Java client has been significantly redesigned. Before this release there were restrictions in what could be done in the Java client's Consumer callback methods, and also on which application threads can call Channel methods. These have been due to the underlying threading structure of the Java client which shared the channel thread with the callbacks. Locks on the channel and connection objects meant that calling Channel methods directly from the Consumer resulted in deadlocks (with a few exceptions, like acknowledgements). The QueueingConsumer helper class was made available to isolate applications from some of these issues, at the cost of introducing another queue (in the Java client).

With the new threading structure, there are far fewer restrictions upon which application threads can call channel operations, because all Consumer callbacks are executed on threads which are separate from the channel. In fact, it is now possible to configure the connection to manage a pool of threads used specifically for callbacks — preserving the order of execution of these within each channel. Simple client applications can take the default which provides a small pool of callback threads and sophisticated clients can provide their own ExecutorService object, which allows them to create and manage the size and behaviour of the thread pool themselves. QueueingConsumer is now no longer necessary, for everything one would want to do as a result of a Consumer callback can be done in the Consumer method directly, without deadlocks. See the Java API Guide for more.

2.7.1 fixed a few irritations in the Java Client adjustments in 2.7.0: we inadvertently hid some of the API, and this has now been restored. There were also some potential resource leaks which we have now fixed.

performance

A number of small performance improvements have been made to the server in both 2.7.0 and 2.7.1. These have been quite varied in scope, and I can only touch on some of them here.

• In the first place, basic file I/O has been improved by the simple expedient of using lower-level basic file operations. This has allowed certain operations to occur in parallel that were previously serialised through an Erlang process. This results in removing some bottlenecks, and slightly speeds up several areas, including server shutdown.
• An area of intense I/O, interestingly unaffected by the tweak above, is what is known as the 'message store'. This is, unsurprisingly, where messages are stored (stored for various reasons, not only message persistence). Rather than using a conventional database, RabbitMQ manages its own file storage for messages.  (A conventional database has almost the entirely wrong performance profile for queueing -- the most recently used items are likely to be the ones accessed last.)  The messages store is one of the most sophisticated parts of the server because it needs to respond very quickly without gating the rest of the system by the relatively slow I/O operations it performs. It behaves something like a paging system cache, in that while messages are waiting to be written they can potentially be 'stolen' from the write list if they are subsequently read, rewritten or even deleted. The 'overtaking' rules are quite different from those of a paging system, however, and in this release, the organisation was changed to allow some deletes to 'cancel' store requests which hadn't yet occurred. This results in reducing unnecessary writes and therefore higher overall throughput for every queue. Extensive tests have shown an improvement in performance with preserved reliability, even under load.
• There was sub-optimal treatment of a connection with a large number of consumers, especially if they were of low utilisation. There appeared to be an overhead for consumers that were relatively inactive. In 2.7.0 this has been improved, which means that having lots of low-use consumers has far less of an impact on overall performance.
• Deletion of queues with a large number of bindings and exchanges with a large number of bindings took rather longer than we would have liked. This has been speeded up in 2.7.1.

HiPE option

Erlang offers a High Performance compiler (HiPE) for some platforms whereby Erlang modules can be compiled to native code. This compilation does not always produce a faster system, however, and is not supported by all Erlang environments and versions. In 2.7.0 we introduced a configuration option to use HiPE, and a re-compilation is performed automatically at server startup. Not every rabbit module is re-compiled — only those that we have determined may benefit from this treatment. Although this option delays startup by some tens of seconds, it produces significant performance improvements at runtime which may be crucial for some larger rabbit installations.

This option is disabled by default, since it may actually affect behaviour (not that we have detected this), and the performance improvements are not tested on all the environments our users employ. However, if it works for you, go for it. If your Erlang environment does not support HiPE, there is a brief diagnostic message and the option is ignored.

We would be very interested in your experiences with this feature.

re-queued messages

RabbitMQ deals with FIFO queues. If all goes well, the order the messages are put on a queue is the same as the order they are consumed. When a consumer fails, however, some of the messages it received may not have been acknowledged, and in this case these messages are re-queued so that they may be delivered again. In this way messages may appear to be re-ordered. Before this release, there was no guarantee of the order of re-queued messages.

From 2.7.0 the relative order of re-queued messages from a single consumer is preserved. Therefore, if another consumer receives them later, they will be consumed in the same order they originally appeared. Of course, if two or more consumers on the same queue fail, there is no guarantee that messages re-queued by distinct consumers will retain their relative order. But in the majority of cases where order matters, this guarantee should be enough.

high availability problems

A number of fixes for high availability features (mostly introduced in 2.7.0) were included in 2.7.1. These relate to some memory leaks; recovery of master queues; frequent restarting causing HA queues to fail; and promotion of slaves (to master) failing under some circumstances.  The general quality of this code area is high, but the complex nature of the failure scenarios (which this feature is specifically designed to protect against) makes it a fruitful one for bugs to lurk. Nearly all the bugs fixed in 2.7.1 are due to rare or obscure combinations of recovery or restart events, and we confidently expect there to be few surprises left. Of course, High Availability doesn't mean Guaranteed Availability, so there are going to be situations from which we cannot recover.

other small improvements and bug fixes

• If you ran a broker for a very long time it was possible to wrap one of the internal GUIDs (Globally Unique IDentifiers). Clearly this was not intended, and in any case it can't cause a problem in practice can it?  Well it did!  (Wouldn't you know it -- some people run brokers for a very long time!) We've fixed it in 2.7.1.
• The management plugin interface now displays a bit more information about queue lengths and shovel information is presented a little more nicely, plus there were a number of small problems with statistics and HA slave information that are now fixed.
• rabbitmqctl eval <expr> is new (in 2.7.1) to evaluate an arbitrary Erlang expression in the broker node.
• .net client session autoclose could sometimes return an AlreadyClosed exception (it is supposed not to do this).
• The STOMP adapter didn't support reply-to queues properly (they weren't re-usable), and could supply multiple message-id headers on a MESSAGE frame if the SEND frame supplied one. We now check for this latter condition and reject the SEND.
• Some functions used which are no longer in Erlang R15B have been removed (and the code rewritten), so that RabbitMQ should now build and run under the latest Erlang Release. Let us know if not!

thank you for listening

As usual, the rabbit team welcome feedback of your experiences, good or bad. We encourage you to use the rabbitmq-discuss mailing list.

With questions like that it's always good to open a history book and review previous generation of computer geeks with no artistic skills. What were they doing? On consoles with green letters they were playing geeky computer games, MUDs (Multi User Dungeons) were especially popular.

Hey, we can do that!

So here it is, a rough and dirty, hacked together in an afternoon MUD! But it aint a normal MUD, it's a unique one:

• The world isn't exactly large, with five locations and 6 commands in total.
• But it's an in-browser game, using SockJS underneath.
• It's built using Django, and the state is handled using Django ORM.

So, forget the 21st century and dive into an ancient world of dragons, at least for a few minutes:

• http://mud.sockjs.org

If you're interested in technology feel free to take a look at the sources. Also, as the project is using Django ORM you can add new locations using Django Admin (user: guest, pass: guest). Unleash your creativity! Unfortunately there isn't a simple way of restricting Django Admin users, so you can't see what you added. You may also want to take a look at the initial database fixture.

Here's a diagram illustrating the architecture of this demo:

As you can see it's quite simple, and it follows one of the recommended SockJS deployment models. It should be possible to scale it horizontally, although this game is only a toy and we haven't really tested that.

Some recent discussion with a client made us return to what we'd thought was a fairly solved problem and has prompted us to make some changes. First, we took some time to understand better how CPU use per message varies as queues get longer and go to disk, and what the effect of that is. The conclusions are not necessarily obvious. Then we started thinking about the justifications behind some of the decisions queues take in the process of going from a purely-in-RAM queue to a purely-on-disk queue.

An AMQP queue in Rabbit is not simple functional FIFO queue. There are in fact at least four "queues" used internally by each AMQP queue, each of which are allowed to hold messages in various different states. These states are things like: Is the message held in RAM (regardless of whether it has additionally been written to disk)?; Is the message itself on disk, but its location within the queue still only held in RAM? That sort of thing. Each message in the AMQP queue will only appear at any one time in one of these internal queues, but they can move from queue to queue if and when their state changes (though the movement between these internal queues respects the overall order of messages within the AMQP queue). There is then a fifth "queue" which is not really a queue at all. It is more a couple of numbers that indicate the range of messages that are held solely on disk (if you like, these are pointers to the head and tail of the "queue" which is solely on disk). Messages in this form have, in theory, zero RAM-cost (depending on how you count (numbers cost RAM too you know!), and elsewhere in Rabbit you can be fairly sure the best you can get down to is a few bytes per message). The full gory details can be gleaned from the essay at the top of the variable_queue module. It's not really that terrifying, but it isn't exactly noddy stuff either. The tricky bits are working out how you decide which messages should be in which states, and when, and I'm not going to cover those decisions in this post.

A message that is captured by this fifth "queue" will potentially take two reads to go from that purely-on-disk form right back to a fully-in-RAM message. This is because every message has a message ID (which is random, unordered, and allocated to each message before it arrives at any queue, so is useless for determining relative position within an AMQP queue), and within each AMQP queue each message is known by its per-queue sequence ID, which enforces the relative order of messages within an AMQP queue. This fifth "queue" can be thought of as a mapping from sequence ID to message ID (plus some per-message-per-queue state), and then you can use a different subsystem to transform that message ID to the actual message.

As a result of these two reads (though the way we structure it, one of the reads is shared between 16k messages, so it's probably closer to 1+(1/16384) reads per message, at least by default), we've previously tried to prevent the use of this fifth "queue": in the past, even when memory is running very low, we'll write messages out to disk fully, but then still hold onto a record in RAM (though by this point it's a fairly small record per message), assuming that this will give us an advantage later on: yes, it costs more RAM, but if some other big AMQP queue suddenly gets deleted and frees up lots of RAM then by holding onto this smallish record per message, we avoid having to do the two reads to go from the fifth "queue" back to a full message, and will only have to do a single read instead. Only when RAM runs out completely will we suddenly dump (almost) everything into this fifth "queue" (though by this point, everything will be on disk by now anyway so it's more or less a no-op -- we're just freeing up RAM in this transition).

However, because of the effective amortisation of at least one of these reads, using the fifth "queue" isn't as expensive as we feared. Plus, if you start to use it earlier then the queue grows in RAM at a slower rate: messages have the lowest per-message RAM cost when they're in this fifth "queue", and thus the greater your use of this "queue", the lower the rate at which your queue will consume RAM. This on its own helps Rabbit smooth out the transition to purely on disk operation (given the same growth rate in messages, a lower growth rate in RAM will result in a lower rate of disk ops).

So we've changed the behaviour of Rabbit's AMQP queues to use this fifth "queue" more eagerly. Benchmarks suggest that this seems to result in an AMQP queue's memory use flat-lining earlier on in its growth, and actually seems to make Rabbit able to deliver messages from large AMQP queues out to consumers faster (probably because by limiting the size of the other four internal queues, some operations which were found to be very inefficient (such as joining two functional queues together (Erlang's default queue module does a plain append (++) here, which is expensive)) are avoided, thus more CPU is available for driving consumers). The downside is that queues now spend more time doing reads, but that seems to have been more than offset by the lower user jiffies per message.

Below is a graph. This is very exciting -- not just because of the fact that most of my blog posts are endless words. It shows three runs of the same test program. This test program does the following:

1. It creates 3 queues.
2. It binds those 3 queues to a fanout exchange.
3. It then starts publishing 200-byte messages to that exchange at 600 messages / second.
4. For the first 120 seconds, it has 20 consumers per queue consuming without auto-ack, one ack per message, and a QoS pre-fetch of 1. This is known to be a very expensive way of consuming messages. Further, the ack is deliberately delayed so that, ignoring network latency, the maximum aggregate consuming rate will be 1200 messages / second.
5. After 120 seconds, the consumers are stopped, and not started again until there is a total backlog of 500,000 messages (i.e. each of the 3 queues will have around 166,000 messages).
6. After that, the consumers resume as before, and you hope that the queues can cope with the continuing publishes and drive their backlog out to the consumers. Hopefully, all the queues will eventually become empty again.

Now depending on your CPU, RAM, network latency, and high_watermark settings, this backlog can be purely in RAM only, and so no disk operations ever take place; or it could be purely on disk; or anywhere in between. Our desktops in the office here tend to be too powerful for this test to cause any problems (the backlog always drains), but on some EC2 hosts, with older versions of Erlang and older versions of Rabbit, it was possible to reach a point where this backlog would never drain, and instead grow.

In the following graph, we have three successful runs of this test, both on m1.large EC2 instances, with the test being run on a separate EC2 instance (i.e. we really were going across the network). These are running Ubuntu images, but with a locally compiled Erlang R14B04 installation. The three runs are: 1) what was on our default branch prior to this work being merged; 2) our default branch after this work was merged; 3) the 2.6.1 release.

Since 2.6.1 was released, a fairly large number of performance tweaks have been made, and this is shown by the faster rate at which the backlog disappears. But the "default prior to change" and "2.6.1" memory usages are fairly similar, whereas, on average, the "default post change" has a lower memory usage. The memory measurements are not especially compelling however as, owing to Erlang being an auto-garbage-collected language, it is not always the case that improving memory efficiency internally results in the VM requesting less RAM or returning RAM to the OS faster. What is more compelling is the faster rate at which the backlog is eliminated and the lower cumulative jiffies: even though "default post change" is doing more messages per second than either of the other runs, it still uses fewer jiffies per second than the other runs.

Hopefully this change will make improvements to many users across many scenarios. It's possible there may be some use cases where it performs worse -- we certainly can't rule that out. No problem in software engineering that's worth solving has a single correct solution. This case shows that sometimes, using less memory and doing apparently more disk ops can actually make things go faster overall.

High Availability (HA) is a typically over-used term and means different things to different people. In the context of RabbitMQ, there are a number of aspects of high availability, some of which this work does solve and some of which it does not. The things it does not solve include:

1. Maintaining connections to a RabbitMQ broker or node: using some sort of TCP load-balancer or proxy is the best route here, though other solutions such as dynamically updating DNS entries or just pre-loading your clients with a list of addresses to connect to may work just as well.

2. Recovery from failure: in the event that a client is disconnected from the broker owing to failure of the node to which the client was connected, if the client was a publishing client, it's possible for the broker to have accepted and passed on messages from the client without the client having received confirmation for them; and likewise on the consuming side it's possible for the client to have issued acknowledgements for messages and have no idea whether or not those acknowledgements made it to the broker and were processed before the failure occurred. In short, you still need to make sure your consuming clients can identify and deal with duplicate messages.

3. Auto-healing from network partitions or splits. RabbitMQ makes use of the Erlang distributed database Mnesia. This database itself does not cope with network partitions: it very much chooses Consistency and Availability and not Partitions from the CAP triangle. As RabbitMQ depends on Mnesia, RabbitMQ itself has the same properties. Thus the HA work in RabbitMQ can prevent queues from disappearing in the event of a node failure, but does not have anything to say about automatically rejoining the failed node when it is repaired: this still requires manual intervention.

These are not new problems at all; and RabbitMQ's HA work does not attempt to address these problems. Instead, it focuses solely on preventing queues from being bound to a single node in a cluster.

The previous situation was that a queue exists only on one node. If that node fails, the queue becomes unavailable. The HA work solves this by mirroring a queue on other nodes: all actions that occur on the queue's master are intercepted and applied in the same order to each of the slaves within the mirror.

This requires:

1. The ability to intercept all actions being performed on a queue. Fortunately, the code abstractions we already have makes this fairly easy.

2. The ability for those actions to be communicated reliably, consistently and in order to all the slaves within the mirror. For this we have written a new guaranteed multicast module (also known as atomic broadcast).

3. The ability to reliably detect the loss of a node in such a way that no messages sent from that node reach a subset of the slaves: to ensure the members of the mirrored queue stay in sync with each other, it's crucial that in the event of the failure of the master, any messages that the master was in the process of sending to the slaves either fail completely or succeed completely (this is really the atomic in atomic broadcast).

In addition, all this communication between the members of the mirror occurs in an asynchronous fashion. This has advantages such as it prevents the master from being slowed down if one of the slaves starts struggling; but it also has disadvantages such as the complexity of interleavings of actions in the event of failure of the master and promotion of a slave.

Once the master does fail, a slave is chosen for promotion. The slave chosen is the eldest slave, in the belief that it's the most likely to have contents that match the contents of the failed master queue. This is important because currently there is no eager synchronisation of mirrored queues. Thus if you create a mirrored queue, send messages into it, and then add another node which then mirrors that queue, the slave on the new node will not receive the existing messages. Only new messages published to the queue will be sent to all current members of the mirrored queue. Thus by consuming from the queue and thus processing the messages at the head of the queue, the non-fully-mirrored messages will be eliminated. Consequently, by promoting the eldest slave, you minimise the number of messages at the head of the queue that may have only been known to the failed master.

At the RabbitMQ HQ we're interested in developments in the wide world of messaging, and we're particularly excited about the JavaScript angle on messaging - namely WebSockets and related technologies.

The good news is - there will be a conference dedicated to these topics:

• Keeping It Realtime Conference

The conference will take two days on Nov 7 and 8 in Portland, OR.

I'm delighted to announce that I'll be giving a talk there. I'll speak about our view on messaging in the web environment, our attempt (and failure) to build a generic web-messaging service, and finally I'll describe what we came up after all this - the SockJS project. (Which is a simple WebSocket emulation layer, but it took us a while to understand why this is the best approach to do web-messaging.)

I'll be hanging around there for the two days - feel free to talk to me about SockJS, RabbitMQ, AMQP or just messaging in general in any flavour.

See you in Portland!

In the presentation I present the evolution of simple shared-nothing architecture and I try to explain why, in order to use web messaging, the website architecture should be truly asynchronous on every stage.

All that is an attempt to express a rationale for the SockJS project. Actually, the full story behind SockJS is much deeper, but probably nobody cares about every detail, I am guessing.

The presentation is available here:

• PubSubHuddle - Realtime Web PDF (5MiB) (Scribd)

The slides are rather terse, in order to understand them you may want to listen to the talk:

• Skills Matter recording

For more details about the reasons behind SockJS, read on:

• Web messaging ain't easy
• WebSocket emulation

RabbitMQ's queues are fastest when they're empty. When a queue is empty, and it has consumers ready to receive messages, then as soon as a message is received by the queue, it goes straight out to the consumer. In the case of a persistent message in a durable queue, yes, it will also go to disk, but that's done in an asynchronous manner and is buffered heavily. The main point is that very little book-keeping needs to be done, very few data structures are modified, and very little additional memory needs allocating. Consequently, the CPU load of a message going through an empty queue is very small.

If the queue is not empty then a bit more work has to be done: the messages have to actually be queued up. Initially, this too is fast and cheap as the underlying functional data structures are very fast. Nevertheless, by holding on to messages, the overall memory usage of the queue will be higher, and we are doing more work than before per message (each message is being both enqueued and dequeued now, whereas before each message was just going straight out to a consumer), so the CPU cost per message is higher. Consequently, the top speed you'll be able to achieve with an empty queue will be higher than the top speed of a queue with a fixed N messages in it, even if N is very small.

If the queue receives messages at a faster rate than it can pump out to consumers then things get slower. As the queue grows, it will require more memory. Additionally, if a queue receives a spike of publications, then the queue must spend time dealing with those publications, which takes CPU time away from sending existing messages out to consumers: a queue of a million messages will be able to be drained out to ready consumers at a much higher rate if there are no publications arriving at the queue to distract it. Not exactly rocket science, but worth remembering that publications arriving at a queue can reduce the rate at which the queue drives its consumers.

Eventually, as a queue grows, it'll become so big that we have to start writing messages out to disk and forgetting about them from RAM in order to free up RAM. At this point, the CPU cost per message is much higher than had the message been dealt with by an empty queue.

None of this seems particularly profound, but keeping these points in mind when building your application turns out to be very important.

Let's say you design and build your application, using RabbitMQ. There will be some set of publishers, and some set of consumers. You test this, and let's say that in total, in one part of the system, you find that the maximum rate that'll ensure your queues stay empty or very near empty is 2000 msgs/second. You then choose a machine on which to run RabbitMQ, which might be on some sort of virtual server. When testing at 2000 msgs/second you found the CPU load of the box running RabbitMQ was not very high: the bottleneck was elsewhere in the application -- most likely in the consumers of your queues (you were measuring the maximum stable end-to-end performance) -- so RabbitMQ itself wasn't being overly stressed and so wasn't eating up much CPU. Consequently, you choose a virtual server which isn't enormously powerful. You then launch the application and sure enough, everything looks OK.

Over time, your application becomes more popular, and so your rates increase.

Eventually, you get to a point where your consumers are running flat out, and your queues are staying nearly empty. But then, at the most popular time of the day for your application, your publishers push a few more messages into your queues than before. This is just normal growth -- you have more users now and so it's not surprising messages are being published a bit faster than before. What you hope will happen is that RabbitMQ will just happily buffer up the messages and will eventually feed them to your consumers who will be able to work through the back-log during quieter times of the day.

The problem is that this might not be able to happen. Because your queues are now (albeit briefly) receiving more messages than your consumers can cope with, the queues spend more CPU time dealing with each message than they used to when the queue was empty (the messages now have to be queued up). That takes away CPU time from driving the consumers and unfortunately, as you chose a machine for RabbitMQ which didn't have a great deal of spare CPU capacity, you start maxing out the CPU. Thus your queues can't drive your consumers quite as hard as before, which in turn makes the rate of growth of the queues increase. This in turn starts to push the queues to sizes where they have to start pushing messages to disk in order to free up RAM which again takes up more CPU which you don't have, and by this point, you're likely in deep trouble.

What can you do?

At this immediate point, you need to get your queues drained. Because your queues are spending more time dealing with new messages arriving than with pushing messages out to consumers, it's unlikely that throwing more consumers at the queues is going to significantly help. You really need to get the publishers to stop.

If you have the luxury of just turning off the publishers then do that, and turn them back on when the queues become empty again. If you can't do that, then you need to divert their load somewhere else, but given that your Rabbit is writing out to disk to avoid running out of memory, and is maxing out the CPU, adding new queues to your current Rabbit is not going to help -- you need a new Rabbit on a different machine. If you've set up a cluster, then you could provision new queues on a node of your RabbitMQ cluster that is not so heavily loaded, then attach lots of new consumers to that, and divert the publishers into there. At this point, you'll realise the value of not using the default nameless exchange and addressing queues directly, and instead will be very glad that you had your publishers publish to an exchange you created, thus allowing you to add a new binding to your fresh new queue, and deleting the binding to your old queue, diverting the load, and not having to interrupt your publishers at all. The old queues will then be able to drive their consumers (which you've not removed!) as fast as possible, and the queues will drain. Now in this situation, you have the prospect of messages being processed out of order (new messages arriving in the fresh new queues may be processed by your new consumers before the old messages in the old queues are processed), but if you have multiple consumers off a single queue then you're probably already dealing with that problem anyway.

Prevention, we are often told, is preferable to cure. So how could you design your application to be able to help RabbitMQ cope with these potentially catastrophic situations?

Firstly, don't use a very low basic.qos prefetch value in your consumers. If you use a value of 1 then it'll mean that a queue sends out one message to a consumer, and then can't send anything more to that consumer until it receives the acknowledgement back. Only when it's done that can it send out the next message. If it takes a while for the acknowledgement to make its way back to the queue (for example, high latency on the network, or the load that your Rabbit is under may mean it takes a while for that acknowledgement to get all the way through to the queue) then in the meantime, that consumer is sitting there idle. If you use a basic.qos prefetch value of 20, for example, then the broker will ensure that 20 messages are sent to the consumer, and then even as the acknowledgement for the first message is (maybe slowly) making its way back to the queue, the consumer still has work to be getting on with (i.e. the next 19 messages). Essentially, the higher the prefetch value, the greater insulation the consumer has from spikes in the round trip time back to the queue.

Secondly, consider not acknowledging every message, but instead acknowledging every N messages and setting the multiple flag on the acknowledgement. When the queue is overloaded, it's because it has too much work to do (profound, I know). As a consumer, you can reduce the amount of work it has to do by ensuring you don't flood it with acknowledgements. Thus, for example, you set the basic.qos prefetch to 20, but you only send an acknowledgement after you've processed every 10 messages and you set the multiple flag to true on the acknowledgement. The queue will now receive one-tenth of the acknowledgements it would have previously received. It will still internally acknowledge all ten messages, but it can do it in a more efficient way if it receives one acknowledgement that accounts for several messages, rather than lots of individual acknowledgements. However, if you're only acknowledging every N messages, be sure that your basic.qos prefetch value is higher than N. Probably at least 2*N. If your prefetch value is the same as N, then your consumer will once again be left idle whilst the acknowledgement makes its way back to the queue and the queue sends out a fresh batch of messages.

Thirdly, have a strategy to pivot the load to other queues on others machines if the worst comes to the worst. Yes, it is a good idea to use RabbitMQ as a buffer which insulates publishers from consumers and can absorb spikes. But equally, you need to remember that RabbitMQ's queues go fastest when they're empty, and we always say that you should design your application so that the queues are normally empty. Or to put it another way, the performance of a queue is lowest when you likely need more than ever for it to absorb a large spike. The upshot of that is that unless you test to makes sure you know it will recover, you might be in for a surprise should a non-trivial spike occur which substantially increases the length of your queues. You may not have factored into your thinking that in such a situation, your existing consumers may end up being driven more slowly than before, simply because your queues are busy doing other things (enqueuing messages), and that this can cause a vicious cycle that eventually results in a catastrophic loss of performance of the queue.

The nub of the issue is that a queue is a single-threaded resource. If you've designed your routing topology in such a way that it already does, or at least can, spread messages across multiple queues rather than just hammering all messages into a single queue, then you're far more likely to be able to respond to such problems occurring quickly and easily, by diverting load and being able to take advantage of additional CPU resources which ensure that you can minimise the CPU-hit per message.

It's free as in beer. There's free beer. And, at last count, there's spaces left.

We have some neat talks lined up

• Martin Sústrik -- The Future of Messaging
• Andy Piper -- Introducing MQTT
• Marek Majkowski -- Realtime web: Not there yet!
• Julien Genestoux -- PubSub for the web : PubSubHubbub, XMPP and Superfeedr

(full descriptions)

If you can't come along, do not fret. The talks will all be taped.

If you can come along, here's the important bit: we want you to bring along your own projects (and laptops). Because in the afternoon, we'll be asking for people to stand up and give a quick talk about what they are doing with messaging, then we'll all break out into huddles. This is your chance to win people to your cause -- or to find a great project to get involved in.

We'll have the speakers and various RabbitMQ and ZeroMQ folk floating around, so it's also your chance to put them on the spot.

pubsubwhatnow?

Going back a few years, James Governor suggested there was an emerging community of people who cared about messaging and that we should all meet up. Thus was born an occasional series of events called "PubSub - putting the pub back into pubsub". Many people you'll see at the huddle went to those pubs.

The idea of a conference came during the ZeroMQ meetup in Brussels, when someone said "wouldn't it be great to bring messaging folks together for a day?" -- RabbitMQ or ZeroMQ, we think messaging is fundamental.

To make it happen, we partnered with Skills Matter, who know a thing or two about running this kind of event.

You said free beer?

Yes. We're providing coffee and tea, lunch, and later on beer and pizza, courtesy of VMware. Register for your free beer and conference here.