Menu

Very fast and scalable topic routing – part 2

March 28th, 2011 by Vlad Alexandru Ionescu

In our previous blog post we talked about a few approaches to topic routing optimization and described the two more important of these in brief. In this post, we will talk about a few things we tried when implementing the DFA, as well as some performance benchmarking we have done on the trie and the DFA. Read the rest of this entry »

Sender-selected distribution

March 23rd, 2011 by Emile Joubert

RabbitMQ 2.4.0 introduced an extension that allows publishers to specify multiple routing keys in the CC and BCC message headers. The BCC header is removed from the message prior to delivery. Direct and topic exchanges are the only standard exchange types that make use of routing keys, therefore the routing logic of this feature only works with these exchange types.

Why would I want this?

1. Custom routing logic

You would normally resort to an external or custom exchange when routing rules are too complex to be expressed with standard exchanges. CC/BCC headers allow a peer to implement potentially complex routing rules by populating these headers with the matching routes.

Imagine a RabbitMQ broker receiving Java Log4J messages and that we are interested in messages at level SEVERE that arrive outside office-hours. This assumes an AMQP Log4J handler that forwards log messages to a RabbitMQ exchange and a client (perhaps connected to a pager) that retrieves them from a queue. Let us assume that the queue is titled "out-of-hours-emergencies" and declared by the pager client.

The problem is how to selectively route messages satisfying these criteria (severity and time). The Java logging API has sufficient sophistication to perform some selective processing and filtering in the log handler before the messages reach the broker, so the problem could potentially be solved upstream from the broker in simple cases. For the purposes of this example we'll want to manage the routing across all log producers centrally in the  broker.

The log handler could decorate the AMQP messages with information about the log events by placing information in the headers. Messages could then be routed according to those headers with the built-in amq.headers exchange. So the first constraint could potentially be met without resorting to additional features, provided that the event severity appears in a message header. The second constraint of our requirement (only messages received outside office-hours) cannot be satisfied with a built-in exchange in the same way. The built-in exchange types can only perform routing based on the contents of a message, not when it arrives. Even if the messages contained a timestamp, built-in exchanges have no way of matching by inequality.

We can solve this problem by relying on a smart consumer that populates the BCC headers before republishing received messages. The relevant criteria in our example would be "out-of-hours-emergencies", so the smart consumer adds this to the BCC header before republishing severe log messages that arrive out of hours. It could use any information at its disposal to make that determination, including date, time, message contents or information from other sources. Any number of criteria can selectively be added to the BCC header in the same way. A queue with the same name will receive all messages from our smart consumer that republished messages with this string in the BCC header. At this point the pager client retrieves messages from the "out-of-hours-emergencies" queue and pages an operator.

This technique can route messages that are encoded in a domain-specific format. A smart peer with knowledge of the format could unpack the message, populate the BCC header with a relevant field and republish. The smart peer is acting in a similar way as an external exchange.

2. Confidential routing

This is useful in cases where the routing key is a secure token which producers and consumers agree beforehand. Wild-cards make topic exchanges useless in this scenario. Messages published with a routing key set to "topsecret.eyesonly" can be obtained by any consumer that binds with a wild-card "#".

Producers can send messages to arbitrary subsets of consumers by populating the BCC header with the routing keys of the selected recipients. The recipients will have no way of learning the identities of other recipients, because the BCC header is removed from the message prior to delivery.

Routing information may still leak in other ways, such as the Management & Monitoring plugin or the rabbitmqctl administration utility. These will need appropriate protection.

Can't AMQP do this already?

While it's not possible to remove headers, it is possible to obtain some comparable effects using only standard AMQP features.

  • Producers can send multiple messages, each with a different routing key. This wastes network bandwidth and broker resources, because the broker cannot optimise the storage of the duplicate messages.
  • Producers can declare a temporary exchange, with a temporary binding for each intended recipient. This a great deal of effort that needs to be repeated each time the set of recipients changes.

How do I use this?

Be sure to use RabbitMQ version 2.4.0 or later. Any AMQP client can be used. Set the CC or BCC headers to the list of routing keys. The header value must be an AMQP array type, even if it only contains a single value. The message will be routed to all destinations according to the combined routings keys in the CC and BCC headers, as well as the basic.publish method ("routingkey1", "routingkey2" and "routingkey3" in this example).

Java sample code:

1:          BasicProperties props  = new BasicProperties();
2:          Map<String, Object> headers = new HashMap<String, Object>();
3:          List<String> ccList = new ArrayList<String>();
4:          ccList.add("routingkey2");
5:          ccList.add("routingkey3");
6:          headers.put("CC", ccList);
7:          props.setHeaders(headers);
8:          channel.basicPublish(exchange, "routingkey1", props, payload);

What are the interoperability implications?

Any AMQP client can make use of this feature. Producers require nothing more than the ability to set headers in messages.

The use of any RabbitMQ-specific extensions makes it harder to swap RabbitMQ for a different AMQP broker - sender-selected distribution is no exception.

If your application already makes use of headers named CC or BCC then you should use different keys or contact the RabbitMQ team for assistance.

Ruby AMQP Benchmarks

March 1st, 2011 by botanicus

I decided to run some benchmarks of my AMQP encoder/decoder (AMQ Protocol gem) against the old one in the AMQP gem to see whether it performs better or not. So far I did only the most basic optimisations like storing reusable values in constants, nothing special (yet).

I did two sets of benchmarks: CPU time benchmarking using my fork of RBench with support for custom formatters (like writing results into a YAML file) and memory benchmarking using Object.count_objects (Ruby 1.9). Read the rest of this entry »

Introducing Publisher Confirms

February 10th, 2011 by Alexandru Scvorţov

In many messaging scenarios, you must not lose messages.  Since AMQP gives few guarantees regarding message persistence/handling, the traditional way to do this is with transactions, which can be unacceptably slow.  To remedy this problem, we introduce an extension to AMQP in the form of Lightweight Publisher Confirms. Read the rest of this entry »

Who are you? Authentication and authorisation in RabbitMQ 2.3.1

February 7th, 2011 by Simon MacMullen

RabbitMQ 2.3.1 introduces a couple of new plugin mechanisms, allowing you much more control over how users authenticate themselves against Rabbit, and how we determine what they are authorised to do. There are three questions of concern here:

  1. How does the client prove its identity over the wire?
  2. Where do users and authentication information (e.g. password hashes) live?
  3. Where does permission information live?

Question 1 is answered in the case of AMQP by SASL - a simple protocol for pluggable authentication mechanisms that is embedded within AMQP (and various other protocols). SASL lets a client and a server negotiate and use an authentication mechanism, without the "outer" protocol having to know any of the details about how authentication works.

SASL offers a number of "mechanisms". Since the beginning, RabbitMQ has supported the PLAIN mechanism, which basically consists of sending a username and password over the wire in plaintext (of course possibly the whole connection might be protected by SSL). It's also supported the variant AMQPLAIN mechanism (which is conceptually identical to PLAIN but slightly easier to implement if you have an AMQP codec lying around). RabbitMQ 2.3.1 adds a plugin system allowing you to add or configure more mechanisms, and we've written an example plugin which implements the SASL EXTERNAL mechanism. Read the rest of this entry »

RabbitMQ, backing stores, databases and disks

January 20th, 2011 by Matthew Sackman

From time to time, on our mailing list and elsewhere, the idea comes up of using a different backing store within RabbitMQ. The backing store is the bit that's responsible for writing messages to disk (a message can be written to disk for a number of reasons) and it's a fairly frequent suggestion to see what RabbitMQ would look like if its own backing store was replaced with another storage system.

Such a change would permit functionality that is not currently possible, for example out-of-band queue browsing, or distributed storage, but there is a fundamental difference in the nature of data storage and access patterns between a message broker such as RabbitMQ and a generic database. Indeed RabbitMQ deliberately does not store messages in such a database. Read the rest of this entry »

Ruby AMQP 0.7 released!

January 19th, 2011 by botanicus

I'm happy to announce that the AMQP 0.7 is released, as I promised in the previous blog post. So what are the changes?

AMQP 0.7 gem installation.

When you install the AMQP gem, you'll see changes of the current version. (How did I do that? With changelog gem and a bit of gemspec magic.)

Callback for MQ#queue

Synchronous API for Queue.Declare/Queue.Declare-Ok request/response was exposed via asynchronous callback:

channel = MQ.new
fanout  = channel.fanout(:task_fanout)
channel.queue(:tasks) do |queue, message_count, consumer_count|
  puts "Queue #{queue.name} declared!"
  puts "Message count: #{message_count}"
  puts "Consumer count: #{consumer_count}"
end

Auto-named queues & not rewritting of anonymous entities in MQ#queues and MQ#exchanges

If a queue is declared with an empty name, the broker is supposed to generate random name. In previous versions of the Ruby AMQP this wasn't supported, because the synchronous API (waiting for Queue.Declare-Ok was missing). Not anymore:

channel = MQ.new
channel.queue("") do |queue|
  puts "Queue with name #{queue.name} declared!"
end

# OUTPUT: Queue with name amq.gen-PfCGdyBA4Sr4rkZg3IN3Kw== declared!

The same should apply for exchanges, but this isn't supported by the current version of RabbitMQ.

Also, in the previous AMQP versions, the MQ#queues, MQ#exchanges and similar was just a hash, hence if given entity was anonymous (the name was nil), and if the collection already included another anonymous instance, then the one which was already in the collection was rewritten.

Callback for MQ::Queue#bind

MQ::Queue#bind can take a callback, as well as MQ#queue now can:

channel = MQ.new
fanout  = channel.fanout(:task_fanout)
channel.queue(:tasks).bind(fanout) do |queue|
  puts "Queue #{queue.name} was bound!"
end

AMQP URL

Thanks to majek, author of the Puka AMQP client for Python, you can use URL instead of option hash as an argument for AMQP.connect and AMQP.start:

AMQP.start("amqps:/")
# Will resolve to: {vhost: "/", port: 5671, ssl: true}

AMQP.start("amqp://[email protected]:1111/")
# Will resolve to: {user: "botanicus", vhost: "/", host: "localhost", port: 1111, ssl: false}

MQ::Exchange.default

The default exchange is a direct exchange with an empty name where all the queues are automatically bound (and you can't bind there anything manually). Do not confuse the default exchange with amq.direct which is only a predefined direct exchange without any "magic" abilities).

Fail if an entity is re-declared with different options

Rather than wait for the server, than if possible we let this fail on the client, so the user gets more descriptive error message:

channel = MQ.new
channel.queue(:tasks, auto_delete: true)
channel.queue(:tasks, auto_delete: false)

# Exception: There is already an instance called tasks with options
{:queue => :tasks, :nowait => true, :auto_delete => true},
you can't define the same instance with different options ({:queue => :tasks,
:nowait => true, :auto_delete => false})! (MQ::IncompatibleOptionsError)

Don't reconnect if the credentials are invalid

AMQP reconnects automatically if the connection failed. It did try to reconnect even on an error like providing invalid credentials. I changed it to register the reconnect hook after the connection is actually established, so if for whatever reason the connection fails, it won't try to reconnect.

rSpec 2 specs

This is still work in progress, you can check the spec/ directory. Huge thanks to arvicco and michaelklishin for their work on this!

Issues

We closed nearly all issues at tmm1/amqp repository. Please do not report any further bugs there, use ruby-amqp/amqp instead.

Friendlier environment for contributors

We use bundler now, so if you want to contribute or just run the tests, just clone the repo, run bundle install and voila, that's it! There's also bin/irb for easier debugging.

Speaking about them, I'd really want to thank all the contributors, their work really helped to get the AMQP gem where it is now. Since the beginning 22 people contributed to the project, and 5 of them have more than 5 commits. Check the CONTRIBUTORS file for more details!

Plans for AMQP 0.8

The next 0.8 release will bring some major API changes: there won't be two separate constants MQ and AMQP, but only the second one. The MQ class will become AMQP::Channel, so we will be compliant with the official AMQP terminology and we also want to introduce support for AMQP 0.9.1 via the AMQ-Protocol gem.

Links

Any comments, ideas? You're always welcome to drop by at Jabber MUC [email protected], and tell us what do you think!

What’s Going on with the Ruby AMQP Gem?

January 12th, 2011 by botanicus

In the past year development of the AMQP gem was practicaly stagnating, as its original author Aman Gupta (@tmm1) was busy. A lot of bugs stayed unresolved, the code was getting old and out-dated and no new features or documentation were made.

At this point I started to talk with the RabbitMQ guys about possible collaboration on this. Actually originally I contacted VMware when I saw Ezra Zygmuntowicz looking for people to his cloud team, but when I found that VMware recently acquired the RabbitMQ project in London, I got interested. I signed the contract, switched from script/console to Wireshark and the RabbitMQ Tracer and since November I've been happily hacking on the AMQP and AMQ-Protocol gems.

To introduce myself, my name's Jakub Stastny (@botanicus) and I work as a Ruby contractor. I contributed to such projects as RubyGems, Merb and rSpec and I wrote my own framework called Rango, the only Ruby framework with template inheritance. I work with Node.js as well and I created Minitest.js, BDD framework for testing asynchronous code. My other hobbies are photography and travelling.

I asked Aman if I can take over the maintainership over the AMQP gem and he was happy to do so. At this point other two guys, Michael Klishin (michaelklishin) and Ar Vicco (arvicco) showed interest in the development, so we created ruby-amqp organisation at GitHub and forked the original code there, as well as a few other related repositories. The GitHub guys were happy to make our repository to be the main one, instead of just a fork, so since now, everything will be there (except the old issues which are still on tmm1's fork and which we want to solve and close soon).

Soo What's New?

Test Suite

At the beginning, there were barely any tests at all, so it was basically impossible to tell if the changes I made break something or not. So I started to write some. In the later stage, when michaelklishin and arvicco joined the development, we rewrote the few original Bacon specs to rSpec 2 and now arvicco is porting his specs which he happened to write some time ago to the main repository. Arvicco has also written amqp-spec, superset of em-spec for testing the AMQP gem.

AMQP 0.9.1

Currently the gem speaks only AMQP 0.8, which is more than 2 years old version, so probably the most important upcoming feature is support of AMQP 0.9.1. Because this is something what can be beneficial for other clients as well, I decided to create a new library called AMQ-protocol. It's using rabbitmq-codegen as many others client libraries.

One of the main goals of this gem is to be really fast and memory-efficient (not for the sake of memory-efficiency itself, but because the garbage collector of MRI is quite weak). I'm about to create some benchmarks soon to see if the performance is better and how much.

AMQ-Protocol is still work-in-progress. It works, but it still needs some polishing, refactoring and optimizations, as well as documentation and tests.

Other Changes

I fixed a lot of bugs and I merged all the pending pull requests to the main repository. I'm going to write more about the changes once I'll release AMQP 0.7. I released 0.7.pre recently, you can try it by running gem install amqp --pre, which would be greatly appreciated. As the work on the test suite is still in progress now, the release process is kind of russian roulette at the moment.

Backward compatibility

I fixed quite a few bugs and obviously the fixed code is never backward-compatible with the old buggy one. One of the major changes is that MQ#queues (as well as MQ#fanouts etc) is not a hash anymore, but an array-like collection with hash-like behaviour. It does NOT override anonymous instances when another anonymous instance is created (as it used to do before) and it does support server-generated names. So instead of MQ#queues[nil] = <first instance> and then MQ#queues[nil] = <second instance>) it now just adds both instances to the collection and when it receives Queue.Declare-Ok from the server, it updates the name to it.

Future plans

The AMQP gem is very opinionated. If you don't want to use EventMachine, you're out of luck. You might want to use something more low-level like IO.select or just another async library like cool.io. You might not even want to care about the asynchronous code at all.

It'd be great if we could have one really un-opinionated AMQP client library which only job would be to expose low-level API defined by the AMQP protocol without any abstraction like hidding channels etc. Such library would be intended for another library implementators rather than for the end users. AMQP is a complex protocol and because of some design decisions it's pretty hard to design a good and easy-to-use (opinionated) client library for it. So some basic library which doesn't make any assumptions would help others to play around and try to implement their own, opinionated libraries on top of this one without the need to manually implement the hard stuff like encoding/decoding or basic socket communication.

Questions? Ideas? Get in touch!

Are you interested in the AMQP gem development? Do you want to participate or do you have some questions? Feel free to contact me, either by comments under this blog post, or you can drop me an e-mail to [email protected] or drop by to Jabber MUC room at [email protected] where all the current maintainers usually are. And for all the news make sure you are following me on Twitter!

AMQP 1.0 prototyping

December 1st, 2010 by Michael

We have been prototyping support for a new protocol, as is our wont. This one is called "AMQP 1.0 R0", and it is the new issue from the AMQP working group (of which RabbitMQ, and latterly VMware, are a member). The "R0" indicates that it's the first revision of a recommendation. The specification is incomplete: there are many TODOs, and to a large extent it is unproven. Those two facts are part of what prompted this prototyping.

The prototype code is mirrored at github: http://github.com/rabbitmq/rabbitmq-amqp1.0. It is built just the same as all our plugins.

The AMQP 1.0 R0 specification differs from the specification of previous versions of AMQP, in that it does not define a broker model; i.e., it doesn't define exchanges queues and bindings, or their equivalents. The protocol is really only about transferring messages from one agent to another, and then agreeing on what the outcome was. That means it is amenable to bolting on to a message broker implementation, among other uses -- the idea is that one can adapt an existing model to suit.

In our case, the incumbent model is that of AMQP 0-9-1, with some generalisations and extensions (for example, chained bindings). Our target with the prototype is therefore to be able to get something useful done with both 1.0 clients and 0-9-1 clients connected at the same time.

Well, the good news is, we've achieved that. In fact the plugin can be set up to replace Rabbit's usual network listener, and will happily talk to AMQP 0-8, 0-9-1, and 1.0 clients. We did have to do some invention along the way, and there are some parts of the specification that we are conspicuously not implementing. These will be detailed in the README soon.

One large part of the invention is to fill in semantics where the specification is silent. Some of these are detailed in this client-broker protocol work we did for the AMQP working group. We're hoping the prototyping will help fill this out some more.

Next week I'll be taking our prototype to the AMQP 1.0 "Connectathon", where it'll be tested against other implementations of the core protocol (not all of which are open source). Again, this will help to flush out barriers to interoperability in the specification.

Chapter 1: Introduction to Distributed Systems

November 17th, 2010 by jdetreville

RabbitMQ needs more and better documentation. (And who doesn’t?) In particular, we need more and better introductory material that introduces the reader to various basic concepts, explains why they’re important, and motivates him or her to keep reading and learn more about RabbitMQ. Here’s a cut at Chapter 1 of that introduction. Your comments are welcome, and Chapters 2 and 3 will follow soon.

(You probably already know all of this, but a surprising number of people don’t. This introduction is for them.)

The Old Future

Long, long ago, the American science-fiction writer Isaac Asimov imagined a future world in which one single giant computer, “Multivac,” would control all of mankind’s affairs. Information would flow in from people and businesses and governments across the globe, and Multivac would store it and process it, and send exactly the right important new information right back out. All sorts of futuristic questions would pour in from our future selves, and the right futuristic answers would just pour back out. This future was a great place!

And our present-day world isn’t all that different from Asimov’s future, just without all that shininess. We’ve got the Internet, and it connects people and businesses and governments all over the globe, and information flows in, and information flows out, and questions pour in, and answers pour out. We’ve got our Googles and our Amazons and our eBays and our Facebooks, and our lives keep getting better every day. More and better information; more and better storage and processing; more and better answers.

But Asimov was only a lowly Ph.D. chemist turned science-fiction writer, not any sort of real Computer Scientist like we have now, and he never worked out all (or, really, any!) of the technical details of exactly how you’d build that one giant, all-knowing, all-powerful Multivac at the North Pole, and exactly who’d pay for it, and exactly what uses they’d allow, and so on. He left that part for future generations to figure out, if in fact they could. And as time has gone by, it’s also turned out that any one single computer that anyone can buy at the computer shop down the street is still several orders of magnitude too small and too weak to control all of mankind’s affairs. That’s the bad news.

The New Future

The good news, which Asimov didn’t anticipate (ha!), is that computers here in the future are cheap—almost dirt cheap, being made largely of silicon, which is after all just processed dirt. So if any one computer you can buy at the shop (or rent on the cloud from Amazon, or whatever) has a million times too little storage or processing power for what you want to do to or for mankind, just get a million of them and plug them together! (Some assembly required.) Google is close to doing just that—just as soon as it completes its takeover of the North Pole—and everyone else is trying to follow close behind. Google’s got its own computers to execute its plans for the world, and Facebook’s got its own computers and its own plans too, and the CIA too, and your company or organization too, and everyone cooperates and competes in controlling all of mankind’s affairs. Our old centralized computer systems couldn’t possibly grow big enough, so we’re replacing them with shiny new distributed systems that could presumably grow bigger forever. And our lives keep getting better every day.

But getting your million computers (or even just a thousand, or even fewer!) to work together on their assigned tasks isn’t as easy as it might sound to your upper management. One given server computer might crash once a year due to bad hardware or bad software or bad power or bad whatever—and that’s usually being pretty optimistic. If you have only a thousand server computers, one will crash on the average every 9 hours; if you had a million, one would crash about every 30 seconds; if you had a billion—which not even Google has yet—about 30 would crash every second, and good luck getting the remainder not to crash or otherwise go bonkers too! One centralized computer can be either up or down, and that’s it, but a distributed computer system is more likely to be 99% up and 1% down at any moment, and the 1% that’s down keeps shifting around and further confusing the other 99%. Problems in distributed systems are unavoidable, and they can multiply without bound. Welcome to the future!

You may have just a thousand computers so far, or maybe just a hundred, or maybe even just ten or so, but you’re still going to have problems and bad things are still going to happen. Crashes are one obvious cause, but lost messages or misconfigured systems or subtle race conditions all add to the error rate too. If you can’t think of half a dozen more potential problems with large distributed systems, you’ve probably never built or operated one. It would be impossibly hard to build that one giant Multivac at the North Pole, but it might be even harder to figure out exactly how those zillion smaller computers that you buy instead will ever work together. What to do?

Perfect Reliability* (*Not really)

There’s a great saying: If you ever see a computer system described as “reliable,” look for the asterisk and the footnote that says “Not really.” Perfect reliability is impossible to achieve. Put your computers in an expensive data center in California, and one sufficiently large earthquake can knock them all out. Spread them out across a bunch of expensive data centers on different continental plates, and you just need a few more earthquakes (or tsunamis, or whatever) to knock enough of your computers (or network links, or whatever) to render the others useless. Enough natural or man-made catastrophes can ruin anything, and they can happen a lot more often than you might think—especially the man-made ones! That’s the bad news.

The good news is, while you can’t build perfectly reliable systems, you can build systems that are reliable enough, whatever that happens to be. That is, you can build computer systems that are arbitrarily reliable. You can ensure that if enough of your computers are up and connected and working correctly, then the system as a whole will continue to do the right things, and that even if more fail, then the system as a whole still won’t do anything wrong. (It might not do anything at all, but that’s life.) If you want more reliability, you can buy more computers (maybe a lot more) and connect them properly. If you know how.

Cargo Cults and Banks

Unfortunately, much of the time, it seems that our distributed-system needs are growing faster than our expertise. Distributed systems are hard to build and they may never become all that easy. Right now, it’s often all we can do adopt best practices—to look at distributed systems that got it right, and try to figure out why they succeeded, and to try to duplicate their success. It’s a little like running your own cargo cult, but without all the coconuts.

Banks are in many ways an excellent industry to study and perhaps to imitate. Banks (and other financial institutions) can clearly care very much about reliability, and banks have been building pretty large, pretty reliable distributed systems for some while now. Banks today tend to build their reliable distributed systems atop reliable message-queuing systems, and they’ve even developed an open standard for such message-queuing systems, and that’s worked out pretty well for them, and that’s what we’ll look at next.