Federation plugin preview release

Note: this blog post talks about the federation plugin preview that was released for RabbitMQ 2.5.0. If you're using 2.6.0 or later, federation is part of the main release; get it the same way you would any other plugin.

Another day, another new plugin release :) Today it's federation. If you want to skip this post and just download the plugin, go here. The detailed instructions are here.

The high level goal of federation is to scale out publish / subscribe messaging across WANs and administrative domains.

To do this we introduce the concept of the federation exchange. A federation exchange acts like a normal exchange of a given type (it can emulate the routing logic of any installed exchange type), but also knows how to connect to upstream exchanges (which might in turn themselves be federation exchanges).

Queues that bind to a federation exchange receive messages that were published to the upstream exchanges (or the upstreams' upstreams) as well as messages that are published locally. In the event of the network going down, messages will be queued upstream.

So how does this differ from clustering?

Federation Clustering
Brokers are logically separate and may have different owners. A cluster forms a single logical broker.
Brokers can run different versions of RabbitMQ and Erlang. Nodes must run the same version of RabbitMQ, and frequently Erlang.
Brokers can be connected via unreliable WAN links. Communication is via AMQP (optionally secured by SSL). Brokers must be connected via reliable LAN links. Communication is via Erlang internode messaging.
Brokers can be connected in whatever topology you arrange. Links can be one- or two-way. All nodes connect to all other nodes in both directions.
Chooses Availability and Partition Tolerance from the CAP theorem. Chooses Consistency and Availability from the CAP theorem.
Some exchanges in a broker may be federated while some may be local. Clustering is all-or-nothing.
A client connecting to any broker can only see queues in that broker. A client connecting to any node can see queues on all nodes.

The README contains detailed instructions for getting things working, but let's look at a simple example.

Suppose we have three brokers, london, paris and newyork. We want a federated topic exchange called stocks (sorry for the predictable example...). We'll look at how the london broker is configured to federate with paris and newyork.

This is what a simple rabbitmq.config file would look like for london:

   [{exchanges, [[{exchange,     "stocks"},
                  {virtual_host, "/"},
                  {type,         "topic"},
                  {durable,      true},
                  {auto_delete,  false},
                  {internal,     false},
                  {upstream_set, "common-upstreams"}]]},
    {upstream_sets, [{"common-upstreams", [[{connection, "newyork"}],
                                           [{connection, "paris"}]]}
    {connections, [{"newyork", [{host, ""}]},
                   {"paris",   [{host, ""}, {protocol, "amqps"},
                                {username, "my-user"}, {password, "secret"}]}
    {local_username, "my-user"}

The rabbitmq_federation application has several options configured.

Firstly we configure an exchange to declare. (You can declare federation exchanges via AMQP - see the README - but since they can require some coordination to set up it is often more convenient to declare them in the configuration file).

Each federated exchange needs the name of an "upstream set" to work with - a set of upstream machines to connect to - so we set one of those up.

Then we list how to make each connection. There are quite a few options here; see the README.

Finally we specify the name of the local user that should be considered to be injecting messages from the upstream brokers.

When we bring the london broker up, we should see messages in the log like:

=INFO REPORT==== 22-Jun-2011::12:16:42 ===
Federation exchange 'stocks' in vhost '/' connected to

=INFO REPORT==== 22-Jun-2011::12:16:43 ===
Federation exchange 'stocks' in vhost '/' connected to

and the exchange will now receive messages published remotely.

In this case we would probably also configure the other brokers to federate with london, but this is not the only way you can do things - for example you can connect brokers in a unidirectional ring, or do massive fanout with brokers in a tree structure.

There are of course limitations, since this is a preview release. The worst is that federation is not compatible with clustering. You shouldn't use the federation plugin in a cluster. This is the first thing we're going to fix.

There are other limitations too: You can't federate headers exchanges. You can't change which machines you federate with unless you restart the broker. There's no status reporting (except for the messages written to the log). Again these will get fixed.

So - is this useful to you? What do you think? (Reminder: download it here). Let us know in comments here, or on the rabbitmq-discuss mailing list.

9 Responses to “Federation plugin preview release”

  1. Russell Says:

    I'm not an expert so I'm wondering...Does this make the shovel plugin redundant?

  2. Simon MacMullen Says:

    Russell: partly. I think a lot of people effectively want federated pub/sub, and are using the shovel to achieve that. Federation will be simpler for them.

    But the shovel lets you do some stuff with shared queues that federation doesn't (yet) - e.g. have multiple shovels consuming round-robin from a single queue and sending to different destinations, and you have a sort of distributed worker queue. I'm not sure if that's actually a popular thing or not, but I can imagine people having reasons to keep using the shovel.

  3. Simone Says:

    A limitation of the shovel plugin is that you can't have a transparent bidirectional shovel, or the entire system goes into an infinite loop, is this scenario tackled by the federation plugin?

  4. Derek Says:

    So, one question I have is with the routing logic. In the README, it states "Bindings are grouped together and bind / unbind commands are sent to the upstream exchange". Am I reading it correctly then that if I have a local federated topic exchange, the upstream bindings will be selected based on the local bindings? In other words, if I bind for a specific (non-pattern) routing key, only those messages will be sent between the upstream broker and the local broker? I'm looking at federating 3000+ sites, so being able to restrict which messages are actually send across the WAN is quite important :)

  5. Simon MacMullen Says:

    Simone: Yes, check the max_hops parameter in the README. Each link must place a limit on the number of times that a message can be forwarded. By default links have a max_hops value of 1, meaning they will not forward messages that have already been forwarded. You can increase that value, but not to infinity.

  6. Simon MacMullen Says:

    Derek: not sure what you mean by "non-pattern" key - you mean one without wildcards? Anyway, yes, the federation plugin will make sure that whether or not you use wildcards only "needed" messages will be sent across the WAN.

  7. Brennan Sellner Says:

    Will any attributes/headers be added to the messages arriving over a federated exchange indicating the upstream source/host? We're considering using this to provide visibility into a number of remote brokers that previously operated independently. We could add data to each of our messages indicating the source site, but it would be much nicer if the federated plugin handled it...

  8. Simon MacMullen Says:

    Brennan: yes, there's an "x-received-from" header.

  9. JBB Says:

    Personally, I like this idea better than Shovel. This is a very important piece of the puzzle for me.