Federation plugin preview release

June 22nd, 2011 by Simon MacMullen

Note: this blog post talks about the federation plugin preview that was released for RabbitMQ 2.5.0. If you're using 2.6.0 or later, federation is part of the main release; get it the same way you would any other plugin.

Another day, another new plugin release :) Today it's federation. If you want to skip this post and just download the plugin, go here. The detailed instructions are here.

The high level goal of federation is to scale out publish / subscribe messaging across WANs and administrative domains.

To do this we introduce the concept of the federation exchange. A federation exchange acts like a normal exchange of a given type (it can emulate the routing logic of any installed exchange type), but also knows how to connect to upstream exchanges (which might in turn themselves be federation exchanges). Read the rest of this entry »

RabbitMQ 2.5.0 released

June 16th, 2011 by Jerry Kuch

The RabbitMQ team is delighted to announce the release of RabbitMQ 2.5.0.

This release fixes a number of bugs. In particular:

  • recovery has been simplified, improving startup times when many exchanges or bindings exist
  • bindings are recovered between durable queues and non-durable exchanges on restart of individual cluster nodes
  • better performance under high load and memory pressure
  • source compatibility with the new Erlang R14B03 release

New features include:

  • tracing facility for debugging incoming and outgoing messages, (see firehose)
  • improved inbound network performance
  • improved routing performance
  • new rabbitmqctl commands ('report', 'environment', and 'cluster_status')

For details see the release notes.

As always, we welcome any questions, bug reports, and other feedback on this release, as well as general suggestions for features and enhancements in future releases. Mail us via the RabbitMQ discussion list, or directly at [email protected].

Can you hear the drums, Erlando?

May 17th, 2011 by Matthew Sackman

Most of us at RabbitMQ HQ have spend time working in a number of functional languages in addition to Erlang, such as Haskell, Scheme, Lisp, OCaml or others. Whilst there is lots to like about Erlang, such as its VM/Emulator, there are inevitably features that we all miss from other languages. In my case, having spent a couple of years working in Haskell before returning to the RabbitMQ fold, all sorts of features are "missing", such as laziness, type classes, additional infix operators, the ability to specify precedence of functions, fewer parenthesis, partial application, more consistent standard libraries and do-notation. That's a fair list, and it'll take me a while to get around to implementing them all in Erlang, but here are two for starters. Read the rest of this entry »

Very fast and scalable topic routing – part 2

March 28th, 2011 by Vlad Alexandru Ionescu

In our previous blog post we talked about a few approaches to topic routing optimization and described the two more important of these in brief. In this post, we will talk about a few things we tried when implementing the DFA, as well as some performance benchmarking we have done on the trie and the DFA. Read the rest of this entry »

Sender-selected distribution

March 23rd, 2011 by Emile Joubert

RabbitMQ 2.4.0 introduced an extension that allows publishers to specify multiple routing keys in the CC and BCC message headers. The BCC header is removed from the message prior to delivery. Direct and topic exchanges are the only standard exchange types that make use of routing keys, therefore the routing logic of this feature only works with these exchange types.

Why would I want this?

1. Custom routing logic

You would normally resort to an external or custom exchange when routing rules are too complex to be expressed with standard exchanges. CC/BCC headers allow a peer to implement potentially complex routing rules by populating these headers with the matching routes.

Imagine a RabbitMQ broker receiving Java Log4J messages and that we are interested in messages at level SEVERE that arrive outside office-hours. This assumes an AMQP Log4J handler that forwards log messages to a RabbitMQ exchange and a client (perhaps connected to a pager) that retrieves them from a queue. Let us assume that the queue is titled "out-of-hours-emergencies" and declared by the pager client.

The problem is how to selectively route messages satisfying these criteria (severity and time). The Java logging API has sufficient sophistication to perform some selective processing and filtering in the log handler before the messages reach the broker, so the problem could potentially be solved upstream from the broker in simple cases. For the purposes of this example we'll want to manage the routing across all log producers centrally in the  broker.

The log handler could decorate the AMQP messages with information about the log events by placing information in the headers. Messages could then be routed according to those headers with the built-in amq.headers exchange. So the first constraint could potentially be met without resorting to additional features, provided that the event severity appears in a message header. The second constraint of our requirement (only messages received outside office-hours) cannot be satisfied with a built-in exchange in the same way. The built-in exchange types can only perform routing based on the contents of a message, not when it arrives. Even if the messages contained a timestamp, built-in exchanges have no way of matching by inequality.

We can solve this problem by relying on a smart consumer that populates the BCC headers before republishing received messages. The relevant criteria in our example would be "out-of-hours-emergencies", so the smart consumer adds this to the BCC header before republishing severe log messages that arrive out of hours. It could use any information at its disposal to make that determination, including date, time, message contents or information from other sources. Any number of criteria can selectively be added to the BCC header in the same way. A queue with the same name will receive all messages from our smart consumer that republished messages with this string in the BCC header. At this point the pager client retrieves messages from the "out-of-hours-emergencies" queue and pages an operator.

This technique can route messages that are encoded in a domain-specific format. A smart peer with knowledge of the format could unpack the message, populate the BCC header with a relevant field and republish. The smart peer is acting in a similar way as an external exchange.

2. Confidential routing

This is useful in cases where the routing key is a secure token which producers and consumers agree beforehand. Wild-cards make topic exchanges useless in this scenario. Messages published with a routing key set to "topsecret.eyesonly" can be obtained by any consumer that binds with a wild-card "#".

Producers can send messages to arbitrary subsets of consumers by populating the BCC header with the routing keys of the selected recipients. The recipients will have no way of learning the identities of other recipients, because the BCC header is removed from the message prior to delivery.

Routing information may still leak in other ways, such as the Management & Monitoring plugin or the rabbitmqctl administration utility. These will need appropriate protection.

Can't AMQP do this already?

While it's not possible to remove headers, it is possible to obtain some comparable effects using only standard AMQP features.

  • Producers can send multiple messages, each with a different routing key. This wastes network bandwidth and broker resources, because the broker cannot optimise the storage of the duplicate messages.
  • Producers can declare a temporary exchange, with a temporary binding for each intended recipient. This a great deal of effort that needs to be repeated each time the set of recipients changes.

How do I use this?

Be sure to use RabbitMQ version 2.4.0 or later. Any AMQP client can be used. Set the CC or BCC headers to the list of routing keys. The header value must be an AMQP array type, even if it only contains a single value. The message will be routed to all destinations according to the combined routings keys in the CC and BCC headers, as well as the basic.publish method ("routingkey1", "routingkey2" and "routingkey3" in this example).

Java sample code:

1:          BasicProperties props  = new BasicProperties();
2:          Map<String, Object> headers = new HashMap<String, Object>();
3:          List<String> ccList = new ArrayList<String>();
4:          ccList.add("routingkey2");
5:          ccList.add("routingkey3");
6:          headers.put("CC", ccList);
7:          props.setHeaders(headers);
8:          channel.basicPublish(exchange, "routingkey1", props, payload);

What are the interoperability implications?

Any AMQP client can make use of this feature. Producers require nothing more than the ability to set headers in messages.

The use of any RabbitMQ-specific extensions makes it harder to swap RabbitMQ for a different AMQP broker - sender-selected distribution is no exception.

If your application already makes use of headers named CC or BCC then you should use different keys or contact the RabbitMQ team for assistance.

Ruby AMQP Benchmarks

March 1st, 2011 by botanicus

I decided to run some benchmarks of my AMQP encoder/decoder (AMQ Protocol gem) against the old one in the AMQP gem to see whether it performs better or not. So far I did only the most basic optimisations like storing reusable values in constants, nothing special (yet).

I did two sets of benchmarks: CPU time benchmarking using my fork of RBench with support for custom formatters (like writing results into a YAML file) and memory benchmarking using Object.count_objects (Ruby 1.9). Read the rest of this entry »

Introducing Publisher Confirms

February 10th, 2011 by Alexandru Scvorţov

In many messaging scenarios, you must not lose messages.  Since AMQP gives few guarantees regarding message persistence/handling, the traditional way to do this is with transactions, which can be unacceptably slow.  To remedy this problem, we introduce an extension to AMQP in the form of Lightweight Publisher Confirms. Read the rest of this entry »

Who are you? Authentication and authorisation in RabbitMQ 2.3.1

February 7th, 2011 by Simon MacMullen

RabbitMQ 2.3.1 introduces a couple of new plugin mechanisms, allowing you much more control over how users authenticate themselves against Rabbit, and how we determine what they are authorised to do. There are three questions of concern here:

  1. How does the client prove its identity over the wire?
  2. Where do users and authentication information (e.g. password hashes) live?
  3. Where does permission information live?

Question 1 is answered in the case of AMQP by SASL - a simple protocol for pluggable authentication mechanisms that is embedded within AMQP (and various other protocols). SASL lets a client and a server negotiate and use an authentication mechanism, without the "outer" protocol having to know any of the details about how authentication works.

SASL offers a number of "mechanisms". Since the beginning, RabbitMQ has supported the PLAIN mechanism, which basically consists of sending a username and password over the wire in plaintext (of course possibly the whole connection might be protected by SSL). It's also supported the variant AMQPLAIN mechanism (which is conceptually identical to PLAIN but slightly easier to implement if you have an AMQP codec lying around). RabbitMQ 2.3.1 adds a plugin system allowing you to add or configure more mechanisms, and we've written an example plugin which implements the SASL EXTERNAL mechanism. Read the rest of this entry »

RabbitMQ, backing stores, databases and disks

January 20th, 2011 by Matthew Sackman

From time to time, on our mailing list and elsewhere, the idea comes up of using a different backing store within RabbitMQ. The backing store is the bit that's responsible for writing messages to disk (a message can be written to disk for a number of reasons) and it's a fairly frequent suggestion to see what RabbitMQ would look like if its own backing store was replaced with another storage system.

Such a change would permit functionality that is not currently possible, for example out-of-band queue browsing, or distributed storage, but there is a fundamental difference in the nature of data storage and access patterns between a message broker such as RabbitMQ and a generic database. Indeed RabbitMQ deliberately does not store messages in such a database. Read the rest of this entry »

Ruby AMQP 0.7 released!

January 19th, 2011 by botanicus

I'm happy to announce that the AMQP 0.7 is released, as I promised in the previous blog post. So what are the changes?

AMQP 0.7 gem installation.

When you install the AMQP gem, you'll see changes of the current version. (How did I do that? With changelog gem and a bit of gemspec magic.)

Callback for MQ#queue

Synchronous API for Queue.Declare/Queue.Declare-Ok request/response was exposed via asynchronous callback:

channel =
fanout  = channel.fanout(:task_fanout)
channel.queue(:tasks) do |queue, message_count, consumer_count|
  puts "Queue #{} declared!"
  puts "Message count: #{message_count}"
  puts "Consumer count: #{consumer_count}"

Auto-named queues & not rewritting of anonymous entities in MQ#queues and MQ#exchanges

If a queue is declared with an empty name, the broker is supposed to generate random name. In previous versions of the Ruby AMQP this wasn't supported, because the synchronous API (waiting for Queue.Declare-Ok was missing). Not anymore:

channel =
channel.queue("") do |queue|
  puts "Queue with name #{} declared!"

# OUTPUT: Queue with name amq.gen-PfCGdyBA4Sr4rkZg3IN3Kw== declared!

The same should apply for exchanges, but this isn't supported by the current version of RabbitMQ.

Also, in the previous AMQP versions, the MQ#queues, MQ#exchanges and similar was just a hash, hence if given entity was anonymous (the name was nil), and if the collection already included another anonymous instance, then the one which was already in the collection was rewritten.

Callback for MQ::Queue#bind

MQ::Queue#bind can take a callback, as well as MQ#queue now can:

channel =
fanout  = channel.fanout(:task_fanout)
channel.queue(:tasks).bind(fanout) do |queue|
  puts "Queue #{} was bound!"


Thanks to majek, author of the Puka AMQP client for Python, you can use URL instead of option hash as an argument for AMQP.connect and AMQP.start:

# Will resolve to: {vhost: "/", port: 5671, ssl: true}

AMQP.start("amqp://[email protected]:1111/")
# Will resolve to: {user: "botanicus", vhost: "/", host: "localhost", port: 1111, ssl: false}


The default exchange is a direct exchange with an empty name where all the queues are automatically bound (and you can't bind there anything manually). Do not confuse the default exchange with which is only a predefined direct exchange without any "magic" abilities).

Fail if an entity is re-declared with different options

Rather than wait for the server, than if possible we let this fail on the client, so the user gets more descriptive error message:

channel =
channel.queue(:tasks, auto_delete: true)
channel.queue(:tasks, auto_delete: false)

# Exception: There is already an instance called tasks with options
{:queue => :tasks, :nowait => true, :auto_delete => true},
you can't define the same instance with different options ({:queue => :tasks,
:nowait => true, :auto_delete => false})! (MQ::IncompatibleOptionsError)

Don't reconnect if the credentials are invalid

AMQP reconnects automatically if the connection failed. It did try to reconnect even on an error like providing invalid credentials. I changed it to register the reconnect hook after the connection is actually established, so if for whatever reason the connection fails, it won't try to reconnect.

rSpec 2 specs

This is still work in progress, you can check the spec/ directory. Huge thanks to arvicco and michaelklishin for their work on this!


We closed nearly all issues at tmm1/amqp repository. Please do not report any further bugs there, use ruby-amqp/amqp instead.

Friendlier environment for contributors

We use bundler now, so if you want to contribute or just run the tests, just clone the repo, run bundle install and voila, that's it! There's also bin/irb for easier debugging.

Speaking about them, I'd really want to thank all the contributors, their work really helped to get the AMQP gem where it is now. Since the beginning 22 people contributed to the project, and 5 of them have more than 5 commits. Check the CONTRIBUTORS file for more details!

Plans for AMQP 0.8

The next 0.8 release will bring some major API changes: there won't be two separate constants MQ and AMQP, but only the second one. The MQ class will become AMQP::Channel, so we will be compliant with the official AMQP terminology and we also want to introduce support for AMQP 0.9.1 via the AMQ-Protocol gem.


Any comments, ideas? You're always welcome to drop by at Jabber MUC [email protected], and tell us what do you think!