Archive for the ‘New Features’ Category

MQTT Adapter

Wednesday, September 12th, 2012

I've written a plugin for RabbitMQ that adds support for the MQTT 3.1 protocol. MQ Telemetry Transport is a light-weight PUB/SUB protocol designed for resource-constrained devices and limited bandwidth situations, making it ideally suited to sensors and mobile devices. The implementation is a protocol adapter  plugin, allowing MQTT clients to connect to a RabbitMQ broker simultaneously with clients implementing other protocols. We encourage projects that demand the combination of a low-overhead protocol on a robust, scalable broker with high reliability and enterprise features to consider this option. (more…)

Introducing RabbitMQ-Web-Stomp

Monday, May 14th, 2012

For quite a while here, at RabbitMQ headquarters, we were struggling to find a good way to expose messaging in a web browser. In the past we tried many things ranging from the old-and-famous JsonRPC plugin (which basically exposes AMQP via AJAX), to Rabbit-Socks (an attempt to create a generic protocol hub), to the management plugin (which can be used for basic things like sending and receiving messages from the browser).

Over time we've learned that the messaging on the web is very different to what we're used to. None of our attempts really addressed that, and it is likely that messaging on the web will not be a fully solved problem for some time yet.

That said, there is a simple thing RabbitMQ users keep on asking about, and although not perfect, it's far from the worst way do messaging in the browser: exposing STOMP through Websockets.


Some queuing theory: throughput, latency and bandwidth

Friday, May 11th, 2012

You have a queue in Rabbit. You have some clients consuming from that queue. If you don't set a QoS setting at all (basic.qos), then Rabbit will push all the queue's messages to the clients as fast as the network and the clients will allow. The consumers will balloon in memory as they buffer all the messages in their own RAM. The queue may appear empty if you ask Rabbit, but there may be millions of messages unacknowledged as they sit in the clients, ready for processing by the client application. If you add a new consumer, there are no messages left in the queue to be sent to the new consumer. Messages are just being buffered in the existing clients, and may be there for a long time, even if there are other consumers that become available to process such messages sooner. This is rather sub optimal.

So, the default QoS prefetch setting gives clients an unlimited buffer, and that can result in poor behaviour and performance. But what should you set the QoS prefetch buffer size to? The goal is to keep the consumers saturated with work, but to minimise the client's buffer size so that more messages stay in Rabbit's queue and are thus available for new consumers or to just be sent out to consumers as they become free. (more…)

RabbitMQ 2.7.0 and 2.7.1 are released

Tuesday, December 20th, 2011

The previous release of RabbitMQ (2.7.0) brought with it a better way of managing plugins, one-stop URI connecting by clients, thread-safe consumers in the Java client, and a number of performance improvements and bug-fixes. The latest release (2.7.1) is essentially a bug-fix release; though it also makes RabbitMQ compatible with Erlang R15B and enhances some of the management interface. The previous release didn't get a blog post, so I've combined both releases in this one.  (These are my own personal remarks and are NOT binding; errors of commission or omission are entirely my own -- Steve Powell.) (more…)

Performance of Queues: when less is more

Thursday, October 27th, 2011

Since the new persister arrived in RabbitMQ 2.0.0 (yes, it's not so new anymore), Rabbit has had a relatively good story to tell about coping with queues that grow and grow and grow and reach sizes that preclude them from being able to be held in RAM. Rabbit starts writing out messages to disk fairly early on, and continues to do so at a gentle rate so that by the time RAM gets really tight, we've done most of the hard work already and thus avoid sudden bursts of writes. Provided your message rates aren't too high or too bursty, this should all happen without any real impact on any connected clients.

Some recent discussion with a client made us return to what we'd thought was a fairly solved problem and has prompted us to make some changes. (more…)

High Availability in RabbitMQ: solving part of the puzzle

Tuesday, October 25th, 2011

In RabbitMQ 2.6.0 we introduced Highly Available queues. These necessitated a new extension to AMQP, and a fair amount of documentation, but to date, little has been written on how they work. (more…)

rabbitmq-tracing – a UI for the firehose

Friday, September 9th, 2011

While the firehose is quite a cool feature, I always thought that it was a shame we didn't have a simple GUI to go on top and make it accessible to system administrators. So I wrote one. You can download it here.


Federation plugin preview release

Wednesday, June 22nd, 2011

Note: this blog post talks about the federation plugin preview that was released for RabbitMQ 2.5.0. If you're using 2.6.0 or later, federation is part of the main release; get it the same way you would any other plugin.

Another day, another new plugin release :) Today it's federation. If you want to skip this post and just download the plugin, go here. The detailed instructions are here.

The high level goal of federation is to scale out publish / subscribe messaging across WANs and administrative domains.

To do this we introduce the concept of the federation exchange. A federation exchange acts like a normal exchange of a given type (it can emulate the routing logic of any installed exchange type), but also knows how to connect to upstream exchanges (which might in turn themselves be federation exchanges). (more…)

Sender-selected distribution

Wednesday, March 23rd, 2011

RabbitMQ 2.4.0 introduced an extension that allows publishers to specify multiple routing keys in the CC and BCC message headers. The BCC header is removed from the message prior to delivery. Direct and topic exchanges are the only standard exchange types that make use of routing keys, therefore the routing logic of this feature only works with these exchange types.

Why would I want this?

1. Custom routing logic

You would normally resort to an external or custom exchange when routing rules are too complex to be expressed with standard exchanges. CC/BCC headers allow a peer to implement potentially complex routing rules by populating these headers with the matching routes.

Imagine a RabbitMQ broker receiving Java Log4J messages and that we are interested in messages at level SEVERE that arrive outside office-hours. This assumes an AMQP Log4J handler that forwards log messages to a RabbitMQ exchange and a client (perhaps connected to a pager) that retrieves them from a queue. Let us assume that the queue is titled "out-of-hours-emergencies" and declared by the pager client.

The problem is how to selectively route messages satisfying these criteria (severity and time). The Java logging API has sufficient sophistication to perform some selective processing and filtering in the log handler before the messages reach the broker, so the problem could potentially be solved upstream from the broker in simple cases. For the purposes of this example we'll want to manage the routing across all log producers centrally in the  broker.

The log handler could decorate the AMQP messages with information about the log events by placing information in the headers. Messages could then be routed according to those headers with the built-in amq.headers exchange. So the first constraint could potentially be met without resorting to additional features, provided that the event severity appears in a message header. The second constraint of our requirement (only messages received outside office-hours) cannot be satisfied with a built-in exchange in the same way. The built-in exchange types can only perform routing based on the contents of a message, not when it arrives. Even if the messages contained a timestamp, built-in exchanges have no way of matching by inequality.

We can solve this problem by relying on a smart consumer that populates the BCC headers before republishing received messages. The relevant criteria in our example would be "out-of-hours-emergencies", so the smart consumer adds this to the BCC header before republishing severe log messages that arrive out of hours. It could use any information at its disposal to make that determination, including date, time, message contents or information from other sources. Any number of criteria can selectively be added to the BCC header in the same way. A queue with the same name will receive all messages from our smart consumer that republished messages with this string in the BCC header. At this point the pager client retrieves messages from the "out-of-hours-emergencies" queue and pages an operator.

This technique can route messages that are encoded in a domain-specific format. A smart peer with knowledge of the format could unpack the message, populate the BCC header with a relevant field and republish. The smart peer is acting in a similar way as an external exchange.

2. Confidential routing

This is useful in cases where the routing key is a secure token which producers and consumers agree beforehand. Wild-cards make topic exchanges useless in this scenario. Messages published with a routing key set to "topsecret.eyesonly" can be obtained by any consumer that binds with a wild-card "#".

Producers can send messages to arbitrary subsets of consumers by populating the BCC header with the routing keys of the selected recipients. The recipients will have no way of learning the identities of other recipients, because the BCC header is removed from the message prior to delivery.

Routing information may still leak in other ways, such as the Management & Monitoring plugin or the rabbitmqctl administration utility. These will need appropriate protection.

Can't AMQP do this already?

While it's not possible to remove headers, it is possible to obtain some comparable effects using only standard AMQP features.

  • Producers can send multiple messages, each with a different routing key. This wastes network bandwidth and broker resources, because the broker cannot optimise the storage of the duplicate messages.
  • Producers can declare a temporary exchange, with a temporary binding for each intended recipient. This a great deal of effort that needs to be repeated each time the set of recipients changes.

How do I use this?

Be sure to use RabbitMQ version 2.4.0 or later. Any AMQP client can be used. Set the CC or BCC headers to the list of routing keys. The header value must be an AMQP array type, even if it only contains a single value. The message will be routed to all destinations according to the combined routings keys in the CC and BCC headers, as well as the basic.publish method ("routingkey1", "routingkey2" and "routingkey3" in this example).

Java sample code:

1:          BasicProperties props  = new BasicProperties();
2:          Map<String, Object> headers = new HashMap<String, Object>();
3:          List<String> ccList = new ArrayList<String>();
4:          ccList.add("routingkey2");
5:          ccList.add("routingkey3");
6:          headers.put("CC", ccList);
7:          props.setHeaders(headers);
8:          channel.basicPublish(exchange, "routingkey1", props, payload);

What are the interoperability implications?

Any AMQP client can make use of this feature. Producers require nothing more than the ability to set headers in messages.

The use of any RabbitMQ-specific extensions makes it harder to swap RabbitMQ for a different AMQP broker - sender-selected distribution is no exception.

If your application already makes use of headers named CC or BCC then you should use different keys or contact the RabbitMQ team for assistance.

Introducing Publisher Confirms

Thursday, February 10th, 2011

In many messaging scenarios, you must not lose messages.  Since AMQP gives few guarantees regarding message persistence/handling, the traditional way to do this is with transactions, which can be unacceptably slow.  To remedy this problem, we introduce an extension to AMQP in the form of Lightweight Publisher Confirms. (more…)