For quite a while here, at RabbitMQ headquarters, we were struggling to find a good way to expose messaging in a web browser. In the past we tried many things ranging from the old-and-famous JsonRPC plugin (which basically exposes AMQP via AJAX), to Rabbit-Socks (an attempt to create a generic protocol hub), to the management plugin (which can be used for basic things like sending and receiving messages from the browser).

Over time we've learned that the messaging on the web is very different to what we're used to. None of our attempts really addressed that, and it is likely that messaging on the web will not be a fully solved problem for some time yet.

That said, there is a simple thing RabbitMQ users keep on asking about, and although not perfect, it's far from the worst way do messaging in the browser: exposing STOMP through Websockets.

STOMP + Websockets

We're delighted to introduce a new plugin for RabbitMQ:

• RabbitMQ-Web-Stomp

It is basically a bridge between RabbitMQ-STOMP plugin and a Websockets server (SockJS). Hopefully, it is a decent solution for a set of rabbit-on-the-web use cases.

What it actually does

Within RabbitMQ

RabbitMQ-Web-Stomp is a simple beast. It takes the STOMP protocol as provided by the RabbitMQ-STOMP plugin and exposes it using the SockJS server.

Wire protocol

One can connect to the SockJS endpoint from any browser using the reliable SockJS protocol. This will work even in browsers that don't support native websockets or in environments behind broken proxies that block non-http transports.

Alternatively, for users that don't need this level of sophistication, SockJS exposes a raw websockets url that can be accessed directly from a recent, websocket-capable browser.

In the browser

Within the browser, the connection to SockJS endpoint is basically a raw STOMP connection. You can send and receive normal STOMP frames.

Any decent javascript STOMP library should be able to handle that. For our examples we're using the stomp-websocket library by Jeff Mesnil and Jeff Lindsay.

We use this code in the examples:

<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"></script>
<script src="stomp.js"></script>
<script>
   WebSocketStompMock = SockJS;

    var client = Stomp.client('http://127.0.0.1:55674/stomp');
    [...]

Installation

Rabbitmq-Web-Stomp is an experimental plugin. It's not distributed with vanilla RabbitMQ releases; you need to install it manually.

1. You need at least Erlang R14 (more info).
2. You need Rabbitmq-server 2.8.2 installed
3. Grab the needed erlang plugin .ez files:

   wget \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/cowboy-0.5.0-rmq2.8.2-git4b93c2d.ez \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/sockjs-0.2.1-rmq2.8.2-gitfa1db96.ez \
     http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/rabbitmq_web_stomp-2.8.2.ez \
   
   http://www.rabbitmq.com/releases/plugins/v2.8.2-web-stomp-preview/rabbitmq_web_stomp_examples-2.8.2.ez

4. Next, copy them to the plugins directory. For example, on my Ubuntu box this will be:

   sudo cp *.ez /usr/lib/rabbitmq/lib/rabbitmq_server-2.8.2/plugins
   

5. Now, you're ready to enable them using rabbitmq-plugins:

   rabbitmq-plugins enable rabbitmq-web-stomp
   rabbitmq-plugins enable rabbitmq-web-stomp-examples
   

6. Restart the rabbitmq server. On ubuntu:

   sudo /etc/init.d/rabbitmq-server restart
   

As you may have noticed, we enabled two plugins:

• First, RabbitMQ-web-stomp, which is the main plugin. It exposes SockJS endpoint on port 55674, like: http://127.0.0.1:55674/stomp.
• Second, RabbitMQ-web-stomp-examples, which only hosts a few javascript and html files with examples. This is accessible under: http://127.0.0.1:55670/.

Keep in mind, that RabbitMQ-web-stomp depends on RabbitMQ-STOMP which by default will bind to port 61613.

The usage

If you enabled RabbitMQ-web-stomp-examples plugin, you should be able to instantly run two examples prepared by us. Just open a web browser at http://127.0.0.1:55670/.

• "echo" - shows how to use STOMP to do simple message broadcasting (source)
• "bunny" - example of a simple collaboration canvas painting app (source)

Summary

RabbitMQ-web-stomp is quite a simple plugin, but opens wide possibilities, exposing the STOMP protocol to the browser.

Like always, feedback welcome. We're also looking for inspiration for more examples!

So, the default QoS prefetch setting gives clients an unlimited buffer, and that can result in poor behaviour and performance. But what should you set the QoS prefetch buffer size to? The goal is to keep the consumers saturated with work, but to minimise the client's buffer size so that more messages stay in Rabbit's queue and are thus available for new consumers or to just be sent out to consumers as they become free.

Let's say it takes 50ms for Rabbit to take a message from this queue, put it on the network and for it to arrive at the consumer. It takes 4ms for the client to process the message. Once the consumer has processed the message, it sends an ack back to Rabbit, which takes a further 50ms to be sent to and processed by Rabbit. So we have a total round trip time of 104ms. If we have a QoS prefetch setting of 1 message then Rabbit won't sent out the next message until after this round trip completes. Thus the client will be busy for only 4ms of every 104ms, or 3.8% of the time. We want it to be busy 100% of the time.

If we do total round trip time / processing time on the client for each message, we get 104 / 4 = 26. If we have a QoS prefetch of 26 messages this solves our problem: assume that the client has 26 messages buffered, ready and waiting for processing. (This is a sensible assumption: once you set basic.qos and then consume from a queue, Rabbit will send as many messages as it can from the queue you've subscribed to to the client, up to the QoS limit. If you assume messages aren't very big and bandwidth is high, it's likely Rabbit will be able to send messages to your consuming client faster than your client can process them. Thus it's reasonable (and simpler) to do all the maths from the assumption of a full client-side buffer.) If each message takes 4ms of processing to deal with then it'll take a total of 26 * 4 = 104ms to deal with the entire buffer. The first 4ms is the client processing of the first message. The client then issues an ack and goes on to process the next message from the buffer. That ack takes 50ms to get to the broker. The broker than issues a new message to the client, which takes 50ms to get there, so by the time 104ms has passed and the client has finished processing its buffer, the next message from the broker has already arrived and is ready and waiting for the client to process it. Thus the client remains busy all the time: having a bigger QoS prefetch will not make it go faster; but we minimise the buffer size and thus latency of messages in the client: messages are buffered by the client for no longer than they need to be in order to keep the client saturated with work. In fact, the client is able to fully drain the buffer before the next message arrives, thus the buffer actually stays empty.

This solution is absolutely fine, provided processing time and network behaviour remains the same. But consider what happens if suddenly the network halves in speed: your prefetch buffer is no longer big enough and now the client will sit idle, waiting for new messages to arrive as the client is able to process messages faster than Rabbit can supply fresh messages.

To address this problem, we might just decide to double (or nearly double) the QoS prefetch size. If we push it to 51 from 26, then if the client processing remains at 4ms per message, we now have 51 * 4 = 204ms of messages in the buffer, of which 4ms will be spent processing a message, leaving 200ms for the sending an ack back to Rabbit and receiving the next message. Thus we can now cope with the network halving in speed.

But if the network's performing normally, doubling our QoS prefetch now means each message will sit in the client side buffer for a while, instead of being processed immediately upon arrival at the client. Again, starting from a full buffer of now 51 messages we know that new messages will start appearing at the client 100ms after the client finishes processing the first message. But in those 100ms, the client will have processed 100 / 4 = 25 messages out of the 50 available. Which means as a new message arrives at the client, it'll be added to the end of the buffer as the client removes from the head of the buffer. The buffer will thus always stay 50 - 25 = 25 messages long and every message will thus sit in the buffer for 25 * 4 = 100ms, increasing the latency between Rabbit sending it to the client and the client starting to process it from 50ms to 150ms.

Thus we see that increasing the prefetch buffer so that the client can cope with deteriorated network performance whilst keeping the client busy, substantially increases the latency when the network is performing normally.

Equally, rather than the network's performance deteriorating, what happens if the client starts taking 40ms to process each message rather than 4ms? If the queue in Rabbit was previously at a steady length (i.e. ingress and egress rates were the same), it'll now start growing rapidly, as the egress rate has dropped to a tenth of what it was. You might decide to try and work through this growing backlog by adding more consumers, but there are messages now being buffered by the existing clients. Assuming the original buffer size of 26 messages, the client will spend 40ms processing the first message, will then send the ack back to Rabbit and move onto the next message. The ack still takes 50ms to get to Rabbit and a further 50ms for Rabbit to send out a new message, but in that 100ms, the client has only worked through 100 / 40 = 2.5 further messages rather than the remaining 25 messages. Thus the buffer is at this point 25 - 3 = 22 messages long. The new message arriving from Rabbit, rather than being processed immediately, now sits in 23rd place, behind 22 other messages still waiting to be processed, and will not be touched by the client for a further 22 * 40 = 880ms. Given the network delay from Rabbit to the client is only 50ms, this additional 880ms delay is now 95% of the latency (880 / (880 + 50) = 0.946).

Even worse, what happens if we doubled the buffer size to 51 messages in order to cope with network performance degradation? After the first message has been processed, there will be 50 further messages buffered in the client. 100ms later (assuming the network is running normally), a new message will arrive from Rabbit, and the client will be half way through processing the 3rd of those 50 messages (the buffer will now be 47 messages long), thus the new message will be 48th in the buffer, and will not be touched for a further 47 * 40 = 1880ms. Again, given the network delay of getting the message to the client is only 50ms, this further 1880ms delay now means client side buffering is responsible for over 97% of the latency (1880 / (1880 + 50) = 0.974). This may very well be unacceptable: the data may only be valid and useful if it's processed promptly, not some 2 seconds after the client received it! If other consuming clients are idle, there's nothing they can do: once Rabbit has sent a message to a client, the message is the client's responsibility until it acks or rejects the message. Clients can't steal messages from each other once the message has been sent to a client. What you want is for clients to be kept busy, but for clients to buffer as few messages as possible so that messages are not delayed by client-side buffers and thus new consuming clients can be quickly fed with messages from Rabbit's queue.

So, too small a buffer results in clients going idle if the network gets slower, but too big a buffer results in lots of extra latency if the network performs normally, and huge amounts of extra latency if the client suddenly starts taking longer to process each message than normal. It's clear that what you really want is a varying buffer size. These problems are common across network devices and have been the subject of much study. Active Queue Management algorithms seek to try and drop or reject messages so that you avoid messages sitting in buffers for long periods of time. The lowest latency is achieved when the buffer is kept empty (each message suffers network latency only and does not sit around in a buffer at all) and buffers are there to absorb spikes. Jim Gettys has been working on this problem from the point of view of network routers: differences between performance of the LAN and the WAN suffer exactly the same sorts of problems. Indeed whenever you have a buffer between a producer (in our case Rabbit) and a consumer (the client-side application logic) where the performance of both sides can vary dynamically, you will suffer these sorts of problems. Recently a new algorithm called Controlled Delay has been published which appears to work well in solving these problems.

The authors claim that their CoDel ("coddle") algorithm is a "knob free" algorithm. This is a bit of a lie really: there are two knobs and they do need setting appropriately. But they don't need changing every time performance changes, which is a massive benefit. I have implemented this algorithm for our AMQP Java Client as a variant of the QueueingConsumer. Whilst the original algorithm is aimed at the TCP layer, where it's valid to just drop packets (TCP itself will take care of re-transmission of lost packets), in AMQP that's not so polite! As a result, my implementation uses Rabbit's basic.nack extension to explicitly return messages to the queue so they can be processed by others.

Using it is pretty much the same as the normal QueueingConsumer except that you should provide three extra parameters to the constructor to get the best performance.

1. The first is requeue which says whether, when messages are nacked, should they be requeued or discarded. If false, they will be discarded which may trigger the dead letter exchange mechanisms if they're set up.
2. The second is the targetDelay which is the acceptable time in milliseconds for messages to wait in the client-side QoS prefetch buffer.
3. The third is the interval and is the expected worst case processing time of one message in milliseconds. This doesn't have to be spot on, but within an order of magnitude certainly helps.

You should still set a QoS prefetch size appropriately. If you do not, what is likely is that the client will be sent a lot of messages, and the algorithm will then have to return them to Rabbit if they sit in the buffer for too long. It's easy to end up with a lot of extra network traffic as messages are returned to Rabbit. The CoDel algorithm is meant to only start dropping (or rejecting) messages once performance diverges from the norm, thus a worked example might help.

Again, assume network traversal time in each direction of 50ms, and we expect the client to spend 4ms on average processing each message, but this can spike to 20ms. We thus set the interval parameter of CoDel to 20. Sometimes the network halves in speed, so the traversal time can be 100ms in each direction. To cater for that, we set the basic.qos prefetch to 204 / 4 = 51. Yes, this means that the buffer will remain 25 messages long most of the time when the network is running normally (see workings earlier), but we decide that's OK. Each message will thus sit in the buffer for an expected 25 * 4 = 100ms, so we set the targetDelay of CoDel to 100.

When things are running normally, CoDel should not get in the way, and few if any messages should be being nacked. But should the client start processing messages more slowly than normal, CoDel will spot that messages have been buffered by the client for too long, and will return those messages to the queue. If those messages are requeued then they will become available for delivery to other clients.

This is very much experimental at the moment, and it's possible to see reasons why CoDel isn't as appropriate for dealing with AMQP messages as it is for plain IP. It's also worth remembering that requeuing messages via nacks is a fairly expensive operation, so it's a good idea to set the parameters of CoDel to ensure in normal operation very few if any messages are being nacked. The management plugin is an easy way to inspect how many messages are being nacked. As ever, comments, feedback and improvements are most welcome!

Welcome back! Last time we talked about flow control and latency; today let's talk about how different features affect the performance we see. Here are some simple scenarios. As before, they're all variations on the theme of one publisher and one consumer publishing as fast as they can.

Some Simple Scenarios

Of course to some extent this quest for large numbers is a bit silly, we're more interested in relative performance. So let's revert to one producer and one consumer.

Message Sizes

Notably, all the messages we've been sending until now have only been a few bytes long. There are a couple of reasons for this:

• Quite a lot of the work done by RabbitMQ is per-message, not per-byte-of-message.
• It's always nice to look at big numbers.

But in the real world we will often want to send bigger messages. So let's look at the next chart:

1 -> 1 sending rate message sizes

Here (again) we're sending unacked / unconfirmed messages as fast as possible, but this time we vary the message size. We can see that (of course) the message rate drops further as the size increases, but the actual number of bytes sent increases as we have less and less routing overhead.

So how does the message size affect horizontal scaling? Let's vary the number of producers with different message sizes. Just for a change, in this test we're not going to have any consumers ar all.

n -> 0 sending msg rate vs number of producers, for various message sizes

n -> 0 sending bytes rate vs number of producers, for various message sizes

In these tests we can see that for small messages it only takes a couple of producers to reach an upper bound on how many messages we can publish, but that for larger messages we need more producers to use the available bandwidth.

Another frequently confusing issue is performance around consumers with a prefetch count. RabbitMQ (well, AMQP) defaults to sending all the messages it can to any consumer that looks ready to accept them. The maximum number of these unacknowledged messages per channel can be limited by setting the prefetch count. However, small prefetch counts can hurt performance (since we can be waiting for acks to arrive before sending out more messages).

So let's have a look at prefetch count and, while we're there, also consider the number of consumers consuming from a single queue. This chart contains some deliberately absurd extremes.

1 -> n recving rate vs consumer count / prefetch count

The first thing to notice is that tiny prefetch counts really hurt performance. Note the large difference in performance between prefetch = 1 and prefetch = 2! But we also get into diminishing returns - notice that the difference between prefetch = 20 and prefetch = 50 is hard to see, and the difference between prefetch = 50 and prefetch = 10000 is almost invisible. Of course, this is because for our particular network link prefetch = 50 already ensures that we never starve the consumer while waiting for acks. Of course, this test was run over a low latency link - more latent links will benefit from a higher prefetch count.

The second thing to notice is that when we have a small number of consumers, adding one more will increase performance (we get more parallellism). And with a tiny prefetch count, increasing consumers even up to a large number has benefits (since each individual consumer spends much of its time starved). But when we have a larger prefetch count, increasing the number of consumers is not so helpful, since even a small number can kept busy enough to max out our queue, but the more consumers we have the more work RabbitMQ has to do to keep track of all of them.

Large queues

All the examples we've looked at so far have one thing in common: very few messages actually get queued. In general we've looked at scenarios where messages get consumed as quickly as they get produced, and thus each queue has an average length of 0.

So what happens whe queues get big? When queues are small(ish) they will reside entirely within memory. Persistent messages will also get written to disc, but they will only get read again if the broker restarts.

But when queues get larger, they will get paged to disc, persistent or not. In this case performance can take a hit as suddenly we need to access the disc to send messages to consumers. So let's run a test: publish a lot of non-persistent messages to a queue, and then consume them all.

Queue load / drain 500k messages

In this small case we can see fairly consistent performance: the messages go into the queue fairly quickly and then come out even more quickly.

Queue load / drain 10M messages

But when we have a larger queue we see that the performance varies a lot more. We see that when loading the queue we initially get a very high throughput, then a pause while some of the queue is paged out to disc, then a more consistent lower throughput. Similarly when draining the queue we see a much lower rate when pulling the messages from disc.

Performance of disc-bound queues is a complex topic - see Matthew's blog post on the subject for some more talk on the subject.

As a co-sponsor we had one of the iPad prizes to award. We decided to allow hacks that used one or more of RabbitMQ, SockJS, or Cloud Foundry. This meant that about half of the twenty-seven hacks were eligible when it came to judging, making the choice rather difficult.

In the end we chose YouChoose, which uses SockJS and is hosted on CloudFoundry. The idea was nifty, and the hack complete and well-executed; and it let one of the judges, who shall remain unnamed, subject the audience to the Titanic theme (until they worked out how to collectively vote it off). Empowering stuff.

(photo by Olly Plumstead)

All the hacks were imaginative and almost all made it to a working demo stage -- which was pretty amazing (or perhaps explained by) considering the lack of sleep some people were suffering on Sunday.

Among our favourites were: Err, microgames with location-based matchmaking (one of the games is "run away as fast as you can"); The Worm, a live popularity chart with optional rageface meter; and Support Net, a site and mobile app for people quitting smoking.

Full coverage of the event is over on GoSquared's blog, including video footage and interviews.

Pusher.com are diversifying

So today I would like to talk about some aspects of RabbitMQ's performance. There are a huge number of variables that feed into the overall level of performance you can get from a RabbitMQ server, and today we're going to try tweaking some of them and seeing what we can see.

The aim of this piece is not to try to convince you that RabbitMQ is the fastest message broker in the world - it often isn't (although we like to think we're still pretty decent) - but to give you some ideas about what sort of performance you can expect in different situations.

All the charts and statistics shown were measured on a PowerEdge R610 with dual Xeon E5530s and 40GB RAM. Largely because it was the fastest machine we had lying around. One major thing that's not ideal is we ran the clients on the same machine as the server - just due to the limited hardware we had available. We used RabbitMQ 2.8.1 (in most cases) and Erlang R15B with HiPE compilation enabled.

By the way, the code to produce all these statistics is available in branch bug24527 of rabbitmq-java-client (although it's currently rather rough) Eventually it will get merged to default, and also become easier to work with. We hope.

Flow control in RabbitMQ 2.8.0+

But first of all I need to introduce a new feature in RabbitMQ 2.8.0+ - internal flow control. RabbitMQ is internally made up of a number of Erlang processes which pass messages to each other. Each process has a mailbox which contains messages it has received and not yet handled. And these mailboxes can grow to an unbounded size.

What this means is that unless the first process to receive data off a network socket is the slowest in the chain, (it's not) then when you have a heavily-loaded RabbitMQ server messages can build up in process mailboxes forever. Or rather, until we run out of memory. Or rather, until the memory alarm goes off. At which point the server will stop accepting new messages while it sorts itself out.

The trouble is, this can take some time. The following chart (the only one in this post made against RabbitMQ 2.7.1) shows a simple process that publishes small messages into the broker as fast as possible, and also consumes them as fast as possible, with acknowledgement, confirms, persistence and so on all switched off. We plot the sending rate, the receiving rate, and the latency (time taken for a sent message to be received), over time. Note that the latency is a logarithmic scale.

Simple 1 -> 1 autoack (2.7.1)

Ouch! That's rather unpleasant. Several things should be obvious:

• The send rate and receive rate fluctuate quite a lot.
• The send rate drops to zero for two minutes (this is the first time the memory alarm went off). In fact the memory alarm goes off again at the end.
• The latency increases steadily (and look at the scale - we show microseconds, but we could just as easily measure it in minutes).

(The small drop in latency around 440s is due to all the messages published before 200s being consumed, and the long gap afterwards.)

Of course, this is only the sort of behaviour you would expect when stressing a server to the limit. But we're benchmarking - we want to do that. And anyway, servers get stressed in production too.

So now let's look at the same experiment conducted against a RabbitMQ 2.8.1 server:

Simple 1 -> 1 autoack (2.8.1)

That looks like a much calmer experience! The send rate, receive rate and latency are all near-constant. The reason is internal flow control. The latency is around 400ms (which is still quite high compared to a less loaded server for reasons I'll discuss in a minute).

These charts don't show memory consumption, but the story is the same - in this circumstance 2.7.1 will eat lots of memory and bounce off the memory alarm threshold, and 2.8.1 will use a fairly constant, fairly low quantity of memory.

Each process in the chain issues credit to the processes that can send messages to it. Processes consume credit as they send messages, and issue more credit as they receive them. When a process runs out of credit it will stop issuing more to its upstream processes. Eventually we reach the process which is reading bytes off a network socket. When that process runs out of credit, it stops reading until it gets more. This is the same as when the memory alarm goes off for the 2.7.1 broker, except that it happens many times per second rather than taking minutes, and we control memory use a lot more.

So where does that 400ms latency come from? Well, there are still messages queueing up at each stage in the pipeline, so it takes a while for a message to get from the beginning to the end. That accounts for some of the latency. However, most of it comes from an invisible "mailbox" in front of the entire server - the TCP buffers provided by the operating system. On Linux the OS will allow up to 8MB of messages to back up in the TCP stack. 8MB doesn't sound like a lot of course, but we're dealing with tiny messages (and each one needs routing decisions, permissions check and so on to be made).

But it's important to remember that we tend to see the worst latency when running at the limit of what we can do. So here's one final chart for this week:

1 -> 1 sending rate attempted vs latency

Note that the horizontal axis is no longer time. We're now showing the results of many runs like the ones above, with each point representing one run.

In the charts above we were running as fast as we can, but here we limit the rate at varying points up to the maximum rate we can achieve. So the yellow line shows rate attempted vs rate achieved - see that it goes most of the way purely 1:1 linearly (when we have spare capacity and so if we try to publish faster we will succeed) and then stops growing as we reach the limit of what we can do.

But look at the latency! With low publishing rates we have latency of considerably less than a millisecond. But this drifts up as the server gets busier. As we stop being able to publish any faster, we hit a wall of latency - the TCP buffers start to fill up and soon messages are taking hundreds of milliseconds to get through them.

So hopefully we've shown how RabbitMQ 2.8.1 offers much more reliable performance when heavily loaded than previous versions, and shown how latency can reach for the skies when your message broker is overloaded. Tune in next time to see how some different ways of using messaging affect performance!

As you may know, WebSockets are a cool new HTML5 technology which allows you to asynchronously send and receive messages. Our compatibility layer - SockJS - emulates it and will work even on old browsers or behind proxies.

WebSockets conceptually are very simple. The API is basically: connect, send and receive. But what if your web-app has many modules and every one wants to be able to send and receive data?

In theory you could open multiple WebSocket connections, one for every module. Although suboptimal (due to the need to handle multiple TCP/IP connections), this approach will work for native WebSockets. But, unfortunately it won't for SockJS due to a technical limitation of HTTP: for some fallbacks transports it is not possible to open more than one connection at a time to a single server.

This problem is real and worth solving. Let me rephrase it:

Assuming you can have only a single connection to a given host, and multiple modules wanting to send and receive data, what do you do?

You need multiplexing: combining data from multiple sources into a single connection. The next question is what API do you use; how do you expose multiplexing in the code?

The Socket.io way

Socket.io has an API that attempts to solve this problem, it calls this 'namespaces'. Here's some example client (browser) code:

var chat = io.connect('http://localhost/chat');
chat.on('connect', function () {
  chat.emit('hi!');
});

var news = io.connect('http://localhost/news');
news.on('news', function () {
  news.emit('woot');
});

I think this API is quite confusing - under the hood Socket.io is opening only a single connection, but reading the code gives us a different story.

The SockJS way

As opposed to Socket.io, SockJS doesn't have any magical API. It looks like a WebSocket object, it behaves like one. Nothing surprising.

So how to solve the multiplexing problem?

It's usually a good idea to avoid inventing new APIs if possible, by using already established ones. Why not present each multiplexed channel as a WebSocket object?

What I'm suggesting is quite simple - you take a real SockJS (or WebSocket) connection, wrap it in a multiplexing layer, and extract any number of fake WebSocket objects out of it. They will be multiplexed internally, but from a module point of view - it will be completely transparent. The module speaks to a WebSocket object as far as it is concerned.

That's it. It's a bit like a magician's hat. You put one WebSocket connection in, you can take any number of fake WebSocket connections out.

This approach is better than what Socket.io proposes - you can create code that just relies on a native WebSocket API. Later on, when the need arises, you can just pass a fake WebSocket object instead of real one. In other words: it composes. Problem solved.

Implementation

If previously in the browser you were using a single SockJS connection, like this:

var sockjs = new SockJS('/echo');

You can modify the client code to:

var real_sockjs = new SockJS('/echo');

var multiplexer = new WebSocketMultiplex(real_sockjs);
var fake_sockjs_1 = multiplexer.channel('ann');
var fake_sockjs_2 = multiplexer.channel('bob');

At this point 'fake' objects will behave identically to a normal SockJS object. You can expect to hear 'open', 'message' and 'close' events.

(The underlying code is about 60 lines of javascript)

Similarly the server side - it normally uses the usual "net.Server" and "Stream" node APIs:

var service = sockjs.createServer();

After a change:

var real_service = sockjs.createServer();

var multiplexer = new multiplex_server.MultiplexServer(real_service);
var fake_service_1 = multiplexer.registerChannel('ann');
var fake_service_2 = multiplexer.registerChannel('bob');

Again 'fake' objects will do the usual thing, they will emit a 'connected' event when a user subscribed to this particular channel arrives.

(The underlying multiplexer code is not very complex either)

If you want to see the multiplexer code in action:

• Server snippet
• Client snippet
• Live code: http://sockjs-multiplex.cloudfoundry.com/

Final thoughts

It's worth emphasising that this approach really does compose. Any module can take a fake WebSocket object and repeat the trick to get more second-layer fake WebSockets objects.

Instead of inventing new API's, just create code that relies on a WebSocket instance passed to the constructor. That'll all you really need to create composable code using WebSockets!

Here at RabbitMQ HQ we spend quite a lot of time arguing. Occasionally, it's about important things, like what messaging really means, and the range of different APIs that can be used to achieve messaging. RabbitMQ and AMQP present a very explicit interface to messaging: you very much have verbs send and receive and you need to think about what your messaging patterns are. There's a lot (of often quite clever stuff) going on under the bonnet but nevertheless, the interface is quite low-level and explicit, which gives a good degree of flexibility. Sometimes though, that style of API is not the most natural fit for the problem you're trying to solve - do you really reach an impasse and think "What I need here is an AMQP-message broker", or do you, from pre-existing knowledge, realise that you could choose to use an AMQP-message broker to solve your current problem?

AtomizeJS exists at the opposite end of the spectrum. There is lots of messaging involved, but you almost never get to see any of it. Instead, you write transactions in JavaScript that modify objects, and those objects are shared between all clients that are connected to the same AtomizeJS server. The API that you're given lets you do slightly more powerful things than you're used to from database transactions, in particular, retry allows you to abort a transaction but then restart it automatically once someone else has changed one of the variables you read. This means you have the observer-pattern, and from that you can then build any explicit messaging patterns you want. In most cases, I doubt you'll be building APIs that say send or receive, instead you'll be building richer data-structures - work queues, shared dictionaries etc. The question to pose then is: is it easier to build these things based on top of a transaction-like API such as offered by AtomizeJS, or on top of an explicit messaging API such as offered by RabbitMQ and AMQP brokers. There is no one solution and horses-for-courses etc, but please leave your thoughts below.

The gain that AtomizeJS provides is not so much in the use of STM from the browser, but the use of STM against a distributed object. This allows you to trivially share state between browsers, modify it safely in intuitive terms, and thus build your applications with little or no application-specific server-side code. Currently, it's a little clunky to use with browsers that don't support bleeding-edge JavaScript features (though I've provided some tooling to try and mitigate this), and everything does work with the latest versions of Chrome, Firefox, IE, Safari, and Opera. Please have a go and let us know what you think!

SockJS version 0.2 has been released:

• https://groups.google.com/group/sockjs/browse_thread/thread/79893c8545c49f06

You can test it in the usual playground:

• http://sockjs.popcnt.org/example-cursors.html
• http://sockjs.cloudfoundry.com/tests-qunit.html

Apart from ton of general updates and few API tweaks, SockJS 0.2 contains two major features:

• Faster connection times - due to a better fallback-detection algorithm.
• Raw websocket api - should make it easier to write command line clients for SockJS.

That means two out of three updates to SockJS protocol I proposed about a month ago are done. The last major feature remaining is binary data support.

Unfortunately the releases rarely go smoothly, but thanks to alert SockJS users the problem was quickly spotted and fixed.

So, happy playing with SockJS!

plugins

Prior to 2.7.0 if you wanted to use a plugin then a .ez file was placed in the plugins directory, and the broker restarted. Any plugin found in this directory was installed on startup.  This meant two things: plugins weren't supplied with the server (even our supported ones), and installing or uninstalling a plugin involved moving files about and ensuring other plugin dependencies were installed. Administration of plugins was unnecessarily messy.

In 2.7.0, we introduced a command rabbitmq-plugins, to enable or disable any plugin in the plugins directory. Simply issue the command rabbitmq-plugins list to see what plugin files are there, and rabbitmq-plugins enable <plugin-name> to use one the next time the broker starts. No files need to be moved around -- they can stay in the plugins directory all the time -- and all the rabbit plugins are now supplied with the server, disabled by default.  The command also understands which plugins depend upon which others, and enables dependencies automatically. Using and managing plugins is now much easier. See the plugins page for more.

In 2.7.1 there is a fix to the consistent-hash-exchange plugin which mis-routed messages when handling multiple exchanges.

connect by URI

All of the rabbit clients (.NET, Java and Erlang) clients accept an amqp URI scheme for connections. This is a convenient one-stop shop allowing the username and password, hostname and port, and virtual host, to be supplied by a single URI or String parameter. For example:

amqp://guest:ghost@rabbit01.coderus.moc:5672/vhost01

See the individual client APIs for details.

Java threads

In Release 2.7.0 the threading structure of the Java client has been significantly redesigned. Before this release there were restrictions in what could be done in the Java client's Consumer callback methods, and also on which application threads can call Channel methods. These have been due to the underlying threading structure of the Java client which shared the channel thread with the callbacks. Locks on the channel and connection objects meant that calling Channel methods directly from the Consumer resulted in deadlocks (with a few exceptions, like acknowledgements). The QueueingConsumer helper class was made available to isolate applications from some of these issues, at the cost of introducing another queue (in the Java client).

With the new threading structure, there are far fewer restrictions upon which application threads can call channel operations, because all Consumer callbacks are executed on threads which are separate from the channel. In fact, it is now possible to configure the connection to manage a pool of threads used specifically for callbacks — preserving the order of execution of these within each channel. Simple client applications can take the default which provides a small pool of callback threads and sophisticated clients can provide their own ExecutorService object, which allows them to create and manage the size and behaviour of the thread pool themselves. QueueingConsumer is now no longer necessary, for everything one would want to do as a result of a Consumer callback can be done in the Consumer method directly, without deadlocks. See the Java API Guide for more.

2.7.1 fixed a few irritations in the Java Client adjustments in 2.7.0: we inadvertently hid some of the API, and this has now been restored. There were also some potential resource leaks which we have now fixed.

performance

A number of small performance improvements have been made to the server in both 2.7.0 and 2.7.1. These have been quite varied in scope, and I can only touch on some of them here.

• In the first place, basic file I/O has been improved by the simple expedient of using lower-level basic file operations. This has allowed certain operations to occur in parallel that were previously serialised through an Erlang process. This results in removing some bottlenecks, and slightly speeds up several areas, including server shutdown.
• An area of intense I/O, interestingly unaffected by the tweak above, is what is known as the 'message store'. This is, unsurprisingly, where messages are stored (stored for various reasons, not only message persistence). Rather than using a conventional database, RabbitMQ manages its own file storage for messages.  (A conventional database has almost the entirely wrong performance profile for queueing -- the most recently used items are likely to be the ones accessed last.)  The messages store is one of the most sophisticated parts of the server because it needs to respond very quickly without gating the rest of the system by the relatively slow I/O operations it performs. It behaves something like a paging system cache, in that while messages are waiting to be written they can potentially be 'stolen' from the write list if they are subsequently read, rewritten or even deleted. The 'overtaking' rules are quite different from those of a paging system, however, and in this release, the organisation was changed to allow some deletes to 'cancel' store requests which hadn't yet occurred. This results in reducing unnecessary writes and therefore higher overall throughput for every queue. Extensive tests have shown an improvement in performance with preserved reliability, even under load.
• There was sub-optimal treatment of a connection with a large number of consumers, especially if they were of low utilisation. There appeared to be an overhead for consumers that were relatively inactive. In 2.7.0 this has been improved, which means that having lots of low-use consumers has far less of an impact on overall performance.
• Deletion of queues with a large number of bindings and exchanges with a large number of bindings took rather longer than we would have liked. This has been speeded up in 2.7.1.

HiPE option

Erlang offers a High Performance compiler (HiPE) for some platforms whereby Erlang modules can be compiled to native code. This compilation does not always produce a faster system, however, and is not supported by all Erlang environments and versions. In 2.7.0 we introduced a configuration option to use HiPE, and a re-compilation is performed automatically at server startup. Not every rabbit module is re-compiled — only those that we have determined may benefit from this treatment. Although this option delays startup by some tens of seconds, it produces significant performance improvements at runtime which may be crucial for some larger rabbit installations.

This option is disabled by default, since it may actually affect behaviour (not that we have detected this), and the performance improvements are not tested on all the environments our users employ. However, if it works for you, go for it. If your Erlang environment does not support HiPE, there is a brief diagnostic message and the option is ignored.

We would be very interested in your experiences with this feature.

re-queued messages

RabbitMQ deals with FIFO queues. If all goes well, the order the messages are put on a queue is the same as the order they are consumed. When a consumer fails, however, some of the messages it received may not have been acknowledged, and in this case these messages are re-queued so that they may be delivered again. In this way messages may appear to be re-ordered. Before this release, there was no guarantee of the order of re-queued messages.

From 2.7.0 the relative order of re-queued messages from a single consumer is preserved. Therefore, if another consumer receives them later, they will be consumed in the same order they originally appeared. Of course, if two or more consumers on the same queue fail, there is no guarantee that messages re-queued by distinct consumers will retain their relative order. But in the majority of cases where order matters, this guarantee should be enough.

high availability problems

A number of fixes for high availability features (mostly introduced in 2.7.0) were included in 2.7.1. These relate to some memory leaks; recovery of master queues; frequent restarting causing HA queues to fail; and promotion of slaves (to master) failing under some circumstances.  The general quality of this code area is high, but the complex nature of the failure scenarios (which this feature is specifically designed to protect against) makes it a fruitful one for bugs to lurk. Nearly all the bugs fixed in 2.7.1 are due to rare or obscure combinations of recovery or restart events, and we confidently expect there to be few surprises left. Of course, High Availability doesn't mean Guaranteed Availability, so there are going to be situations from which we cannot recover.

other small improvements and bug fixes

• If you ran a broker for a very long time it was possible to wrap one of the internal GUIDs (Globally Unique IDentifiers). Clearly this was not intended, and in any case it can't cause a problem in practice can it?  Well it did!  (Wouldn't you know it -- some people run brokers for a very long time!) We've fixed it in 2.7.1.
• The management plugin interface now displays a bit more information about queue lengths and shovel information is presented a little more nicely, plus there were a number of small problems with statistics and HA slave information that are now fixed.
• rabbitmqctl eval <expr> is new (in 2.7.1) to evaluate an arbitrary Erlang expression in the broker node.
• .net client session autoclose could sometimes return an AlreadyClosed exception (it is supposed not to do this).
• The STOMP adapter didn't support reply-to queues properly (they weren't re-usable), and could supply multiple message-id headers on a MESSAGE frame if the SEND frame supplied one. We now check for this latter condition and reject the SEND.
• Some functions used which are no longer in Erlang R15B have been removed (and the code rewritten), so that RabbitMQ should now build and run under the latest Erlang Release. Let us know if not!

thank you for listening

As usual, the rabbit team welcome feedback of your experiences, good or bad. We encourage you to use the rabbitmq-discuss mailing list.

With questions like that it's always good to open a history book and review previous generation of computer geeks with no artistic skills. What were they doing? On consoles with green letters they were playing geeky computer games, MUDs (Multi User Dungeons) were especially popular.

Hey, we can do that!

So here it is, a rough and dirty, hacked together in an afternoon MUD! But it aint a normal MUD, it's a unique one:

• The world isn't exactly large, with five locations and 6 commands in total.
• But it's an in-browser game, using SockJS underneath.
• It's built using Django, and the state is handled using Django ORM.

So, forget the 21st century and dive into an ancient world of dragons, at least for a few minutes:

• http://mud.sockjs.org

If you're interested in technology feel free to take a look at the sources. Also, as the project is using Django ORM you can add new locations using Django Admin (user: guest, pass: guest). Unleash your creativity! Unfortunately there isn't a simple way of restricting Django Admin users, so you can't see what you added. You may also want to take a look at the initial database fixture.

Here's a diagram illustrating the architecture of this demo:

As you can see it's quite simple, and it follows one of the recommended SockJS deployment models. It should be possible to scale it horizontally, although this game is only a toy and we haven't really tested that.