Menu

Ruby AMQP 0.7 released!

January 19th, 2011 by botanicus

I'm happy to announce that the AMQP 0.7 is released, as I promised in the previous blog post. So what are the changes?

AMQP 0.7 gem installation.

When you install the AMQP gem, you'll see changes of the current version. (How did I do that? With changelog gem and a bit of gemspec magic.)

Callback for MQ#queue

Synchronous API for Queue.Declare/Queue.Declare-Ok request/response was exposed via asynchronous callback:
channel = MQ.new
fanout  = channel.fanout(:taskfanout)
channel.queue(:tasks) do |queue, messagecount, consumercount|
  puts "Queue #{queue.name} declared!"
  puts "Message count: #{messagecount}"
  puts "Consumer count: #{consumer_count}"
end

Auto-named queues & not rewritting of anonymous entities in MQ#queues and MQ#exchanges

If a queue is declared with an empty name, the broker is supposed to generate random name. In previous versions of the Ruby AMQP this wasn't supported, because the synchronous API (waiting for Queue.Declare-Ok was missing). Not anymore:
channel = MQ.new
channel.queue("") do |queue|
  puts "Queue with name #{queue.name} declared!"
end

OUTPUT: Queue with name amq.gen-PfCGdyBA4Sr4rkZg3IN3Kw== declared!

The same should apply for exchanges, but this isn't supported by the current version of RabbitMQ.

Also, in the previous AMQP versions, the MQ#queues, MQ#exchanges and similar was just a hash, hence if given entity was anonymous (the name was nil), and if the collection already included another anonymous instance, then the one which was already in the collection was rewritten.

Callback for MQ::Queue#bind

MQ::Queue#bind can take a callback, as well as MQ#queue now can:
channel = MQ.new
fanout  = channel.fanout(:task_fanout)
channel.queue(:tasks).bind(fanout) do |queue|
  puts "Queue #{queue.name} was bound!"
end

AMQP URL

Thanks to majek, author of the Puka AMQP client for Python, you can use URL instead of option hash as an argument for AMQP.connect and AMQP.start:
AMQP.start("amqps:/")

Will resolve to: {vhost: "/", port: 5671, ssl: true}

AMQP.start("amqp://[email protected]:1111/")

Will resolve to: {user: "botanicus", vhost: "/", host: "localhost", port: 1111, ssl: false}

MQ::Exchange.default

The default exchange is a direct exchange with an empty name where all the queues are automatically bound (and you can't bind there anything manually). Do not confuse the default exchange with amq.direct which is only a predefined direct exchange without any "magic" abilities).

Fail if an entity is re-declared with different options

Rather than wait for the server, than if possible we let this fail on the client, so the user gets more descriptive error message:
channel = MQ.new
channel.queue(:tasks, autodelete: true)
channel.queue(:tasks, autodelete: false)

Exception: There is already an instance called tasks with options

{:queue => :tasks, :nowait => true, :autodelete => true}, you can't define the same instance with different options ({:queue => :tasks, :nowait => true, :autodelete => false})! (MQ::IncompatibleOptionsError)

Don't reconnect if the credentials are invalid

AMQP reconnects automatically if the connection failed. It did try to reconnect even on an error like providing invalid credentials. I changed it to register the reconnect hook after the connection is actually established, so if for whatever reason the connection fails, it won't try to reconnect.

rSpec 2 specs

This is still work in progress, you can check the spec/ directory. Huge thanks to arvicco and michaelklishin for their work on this!

Issues

We closed nearly all issues at tmm1/amqp repository. Please do not report any further bugs there, use ruby-amqp/amqp instead.

Friendlier environment for contributors

We use bundler now, so if you want to contribute or just run the tests, just clone the repo, run bundle install and voila, that's it! There's also bin/irb for easier debugging.

Speaking about them, I'd really want to thank all the contributors, their work really helped to get the AMQP gem where it is now. Since the beginning 22 people contributed to the project, and 5 of them have more than 5 commits. Check the CONTRIBUTORS file for more details!

Plans for AMQP 0.8

The next 0.8 release will bring some major API changes: there won't be two separate constants MQ and AMQP, but only the second one. The MQ class will become AMQP::Channel, so we will be compliant with the official AMQP terminology and we also want to introduce support for AMQP 0.9.1 via the AMQ-Protocol gem.

Links

Any comments, ideas? You're always welcome to drop by at Jabber MUC [email protected], and tell us what do you think!

What’s Going on with the Ruby AMQP Gem?

January 12th, 2011 by botanicus

In the past year development of the AMQP gem was practicaly stagnating, as its original author Aman Gupta (@tmm1) was busy. A lot of bugs stayed unresolved, the code was getting old and out-dated and no new features or documentation were made.

At this point I started to talk with the RabbitMQ guys about possible collaboration on this. Actually originally I contacted VMware when I saw Ezra Zygmuntowicz looking for people to his cloud team, but when I found that VMware recently acquired the RabbitMQ project in London, I got interested. I signed the contract, switched from script/console to Wireshark and the RabbitMQ Tracer and since November I've been happily hacking on the AMQP and AMQ-Protocol gems.

To introduce myself, my name's Jakub Stastny (@botanicus) and I work as a Ruby contractor. I contributed to such projects as RubyGems, Merb and rSpec and I wrote my own framework called Rango, the only Ruby framework with template inheritance. I work with Node.js as well and I created Minitest.js, BDD framework for testing asynchronous code. My other hobbies are photography and travelling.

I asked Aman if I can take over the maintainership over the AMQP gem and he was happy to do so. At this point other two guys, Michael Klishin (michaelklishin) and Ar Vicco (arvicco) showed interest in the development, so we created ruby-amqp organisation at GitHub and forked the original code there, as well as a few other related repositories. The GitHub guys were happy to make our repository to be the main one, instead of just a fork, so since now, everything will be there (except the old issues which are still on tmm1's fork and which we want to solve and close soon).

Soo What's New?

Test Suite

At the beginning, there were barely any tests at all, so it was basically impossible to tell if the changes I made break something or not. So I started to write some. In the later stage, when michaelklishin and arvicco joined the development, we rewrote the few original Bacon specs to rSpec 2 and now arvicco is porting his specs which he happened to write some time ago to the main repository. Arvicco has also written amqp-spec, superset of em-spec for testing the AMQP gem.

AMQP 0.9.1

Currently the gem speaks only AMQP 0.8, which is more than 2 years old version, so probably the most important upcoming feature is support of AMQP 0.9.1. Because this is something what can be beneficial for other clients as well, I decided to create a new library called AMQ-protocol. It's using rabbitmq-codegen as many others client libraries.

One of the main goals of this gem is to be really fast and memory-efficient (not for the sake of memory-efficiency itself, but because the garbage collector of MRI is quite weak). I'm about to create some benchmarks soon to see if the performance is better and how much.

AMQ-Protocol is still work-in-progress. It works, but it still needs some polishing, refactoring and optimizations, as well as documentation and tests.

Other Changes

I fixed a lot of bugs and I merged all the pending pull requests to the main repository. I'm going to write more about the changes once I'll release AMQP 0.7. I released 0.7.pre recently, you can try it by running gem install amqp --pre, which would be greatly appreciated. As the work on the test suite is still in progress now, the release process is kind of russian roulette at the moment.

Backward compatibility

I fixed quite a few bugs and obviously the fixed code is never backward-compatible with the old buggy one. One of the major changes is that MQ#queues (as well as MQ#fanouts etc) is not a hash anymore, but an array-like collection with hash-like behaviour. It does NOT override anonymous instances when another anonymous instance is created (as it used to do before) and it does support server-generated names. So instead of MQ#queues[nil] = <first instance> and then MQ#queues[nil] = <second instance>) it now just adds both instances to the collection and when it receives Queue.Declare-Ok from the server, it updates the name to it.

Future plans

The AMQP gem is very opinionated. If you don't want to use EventMachine, you're out of luck. You might want to use something more low-level like IO.select or just another async library like cool.io. You might not even want to care about the asynchronous code at all.

It'd be great if we could have one really un-opinionated AMQP client library which only job would be to expose low-level API defined by the AMQP protocol without any abstraction like hidding channels etc. Such library would be intended for another library implementators rather than for the end users. AMQP is a complex protocol and because of some design decisions it's pretty hard to design a good and easy-to-use (opinionated) client library for it. So some basic library which doesn't make any assumptions would help others to play around and try to implement their own, opinionated libraries on top of this one without the need to manually implement the hard stuff like encoding/decoding or basic socket communication.

Questions? Ideas? Get in touch!

Are you interested in the AMQP gem development? Do you want to participate or do you have some questions? Feel free to contact me, either by comments under this blog post, or you can drop me an e-mail to [email protected] or drop by to Jabber MUC room at [email protected] where all the current maintainers usually are. And for all the news make sure you are following me on Twitter!

AMQP 1.0 prototyping

December 1st, 2010 by Michael

We have been prototyping support for a new protocol, as is our wont. This one is called "AMQP 1.0 R0", and it is the new issue from the AMQP working group (of which RabbitMQ, and latterly VMware, are a member). The "R0" indicates that it's the first revision of a recommendation. The specification is incomplete: there are many TODOs, and to a large extent it is unproven. Those two facts are part of what prompted this prototyping.

The prototype code is mirrored at github: http://github.com/rabbitmq/rabbitmq-amqp1.0. It is built just the same as all our plugins.

The AMQP 1.0 R0 specification differs from the specification of previous versions of AMQP, in that it does not define a broker model; i.e., it doesn't define exchanges queues and bindings, or their equivalents. The protocol is really only about transferring messages from one agent to another, and then agreeing on what the outcome was. That means it is amenable to bolting on to a message broker implementation, among other uses -- the idea is that one can adapt an existing model to suit.

In our case, the incumbent model is that of AMQP 0-9-1, with some generalisations and extensions (for example, chained bindings). Our target with the prototype is therefore to be able to get something useful done with both 1.0 clients and 0-9-1 clients connected at the same time.

Well, the good news is, we've achieved that. In fact the plugin can be set up to replace Rabbit's usual network listener, and will happily talk to AMQP 0-8, 0-9-1, and 1.0 clients. We did have to do some invention along the way, and there are some parts of the specification that we are conspicuously not implementing. These will be detailed in the README soon.

One large part of the invention is to fill in semantics where the specification is silent. Some of these are detailed in this client-broker protocol work we did for the AMQP working group. We're hoping the prototyping will help fill this out some more.

Next week I'll be taking our prototype to the AMQP 1.0 "Connectathon", where it'll be tested against other implementations of the core protocol (not all of which are open source). Again, this will help to flush out barriers to interoperability in the specification.

Chapter 1: Introduction to Distributed Systems

November 17th, 2010 by jdetreville

RabbitMQ needs more and better documentation. (And who doesn’t?) In particular, we need more and better introductory material that introduces the reader to various basic concepts, explains why they’re important, and motivates him or her to keep reading and learn more about RabbitMQ. Here’s a cut at Chapter 1 of that introduction. Your comments are welcome, and Chapters 2 and 3 will follow soon.

(You probably already know all of this, but a surprising number of people don’t. This introduction is for them.)

The Old Future

Long, long ago, the American science-fiction writer Isaac Asimov imagined a future world in which one single giant computer, “Multivac,” would control all of mankind’s affairs. Information would flow in from people and businesses and governments across the globe, and Multivac would store it and process it, and send exactly the right important new information right back out. All sorts of futuristic questions would pour in from our future selves, and the right futuristic answers would just pour back out. This future was a great place!

And our present-day world isn’t all that different from Asimov’s future, just without all that shininess. We’ve got the Internet, and it connects people and businesses and governments all over the globe, and information flows in, and information flows out, and questions pour in, and answers pour out. We’ve got our Googles and our Amazons and our eBays and our Facebooks, and our lives keep getting better every day. More and better information; more and better storage and processing; more and better answers.

But Asimov was only a lowly Ph.D. chemist turned science-fiction writer, not any sort of real Computer Scientist like we have now, and he never worked out all (or, really, any!) of the technical details of exactly how you’d build that one giant, all-knowing, all-powerful Multivac at the North Pole, and exactly who’d pay for it, and exactly what uses they’d allow, and so on. He left that part for future generations to figure out, if in fact they could. And as time has gone by, it’s also turned out that any one single computer that anyone can buy at the computer shop down the street is still several orders of magnitude too small and too weak to control all of mankind’s affairs. That’s the bad news.

The New Future

The good news, which Asimov didn’t anticipate (ha!), is that computers here in the future are cheap—almost dirt cheap, being made largely of silicon, which is after all just processed dirt. So if any one computer you can buy at the shop (or rent on the cloud from Amazon, or whatever) has a million times too little storage or processing power for what you want to do to or for mankind, just get a million of them and plug them together! (Some assembly required.) Google is close to doing just that—just as soon as it completes its takeover of the North Pole—and everyone else is trying to follow close behind. Google’s got its own computers to execute its plans for the world, and Facebook’s got its own computers and its own plans too, and the CIA too, and your company or organization too, and everyone cooperates and competes in controlling all of mankind’s affairs. Our old centralized computer systems couldn’t possibly grow big enough, so we’re replacing them with shiny new distributed systems that could presumably grow bigger forever. And our lives keep getting better every day.

But getting your million computers (or even just a thousand, or even fewer!) to work together on their assigned tasks isn’t as easy as it might sound to your upper management. One given server computer might crash once a year due to bad hardware or bad software or bad power or bad whatever—and that’s usually being pretty optimistic. If you have only a thousand server computers, one will crash on the average every 9 hours; if you had a million, one would crash about every 30 seconds; if you had a billion—which not even Google has yet—about 30 would crash every second, and good luck getting the remainder not to crash or otherwise go bonkers too! One centralized computer can be either up or down, and that’s it, but a distributed computer system is more likely to be 99% up and 1% down at any moment, and the 1% that’s down keeps shifting around and further confusing the other 99%. Problems in distributed systems are unavoidable, and they can multiply without bound. Welcome to the future!

You may have just a thousand computers so far, or maybe just a hundred, or maybe even just ten or so, but you’re still going to have problems and bad things are still going to happen. Crashes are one obvious cause, but lost messages or misconfigured systems or subtle race conditions all add to the error rate too. If you can’t think of half a dozen more potential problems with large distributed systems, you’ve probably never built or operated one. It would be impossibly hard to build that one giant Multivac at the North Pole, but it might be even harder to figure out exactly how those zillion smaller computers that you buy instead will ever work together. What to do?

Perfect Reliability* (*Not really)

There’s a great saying: If you ever see a computer system described as “reliable,” look for the asterisk and the footnote that says “Not really.” Perfect reliability is impossible to achieve. Put your computers in an expensive data center in California, and one sufficiently large earthquake can knock them all out. Spread them out across a bunch of expensive data centers on different continental plates, and you just need a few more earthquakes (or tsunamis, or whatever) to knock enough of your computers (or network links, or whatever) to render the others useless. Enough natural or man-made catastrophes can ruin anything, and they can happen a lot more often than you might think—especially the man-made ones! That’s the bad news.

The good news is, while you can’t build perfectly reliable systems, you can build systems that are reliable enough, whatever that happens to be. That is, you can build computer systems that are arbitrarily reliable. You can ensure that if enough of your computers are up and connected and working correctly, then the system as a whole will continue to do the right things, and that even if more fail, then the system as a whole still won’t do anything wrong. (It might not do anything at all, but that’s life.) If you want more reliability, you can buy more computers (maybe a lot more) and connect them properly. If you know how.

Cargo Cults and Banks

Unfortunately, much of the time, it seems that our distributed-system needs are growing faster than our expertise. Distributed systems are hard to build and they may never become all that easy. Right now, it’s often all we can do adopt best practices—to look at distributed systems that got it right, and try to figure out why they succeeded, and to try to duplicate their success. It’s a little like running your own cargo cult, but without all the coconuts.

Banks are in many ways an excellent industry to study and perhaps to imitate. Banks (and other financial institutions) can clearly care very much about reliability, and banks have been building pretty large, pretty reliable distributed systems for some while now. Banks today tend to build their reliable distributed systems atop reliable message-queuing systems, and they’ve even developed an open standard for such message-queuing systems, and that’s worked out pretty well for them, and that’s what we’ll look at next.

rabbitmq + node.js = rabbit.js

November 12th, 2010 by Michael

For those who have been away from the internets, node.js is an evented JavaScript engine based on Google's V8. Because it is essentially one big, efficient event loop, it's a natural fit for programs that shuffle data backwards and forwards with little state in-between. And it's fun to program, an opinion apparently lots of people share, because there have been loads of libraries crop up around it.

Among the more impressive of these libraries is Socket.IO. One can combine Socket.IO with node.js's built-in web server to make a websocket server, with a socket abstraction for browsers that degrades to XHR tricks for when there's no websockets. (I would be happy to believe that node.js and Socket.IO were made for us by a benevolent and foresightful precursor race; but of course, they were made by hard-working clever people. Thank you, people!)

Once one has a socket abstraction in the browser, a whole world opens up. Specifically, for our purposes, a whole world of messaging. Since node.js has an AMQP client, we can easily hook it up with RabbitMQ; not only to bridge to other protocols and back-end systems, but also to provide messaging between browsers, and between application servers, and so on.

Following on from the work we've been doing with Martin Sustrik of ZeroMQ, I decided to make a very simple protocol for using on the browser sockets, reflecting the messaging patterns used in ZeroMQ (and thereby in RMQ-0MQ) -- pub/sub, request/reply, and push/pull (or pipeline). I wrote a node.js library that uses RabbitMQ to implement message patterns using its routing and buffering; the bridging then comes for free, since RabbitMQ has a bunch of protocol adapters and clients for various languages.

A brief explanation of the messaging patterns:

Publish/Subscribe is for the situation in which a published message should be delivered to multiple subscribers. In the general case, various kinds of routing can be used to filter the messages for each subscriber. This might be used to broadcast notifications from a backend system to users' browsers, for example.

Request/Reply is for RPC over messaging; requests are distributed round-robin among worker processes, and replies are routed back to the requesting socket. This might be used by browsers to query back-end services; or even for browsers to query each other.

Pipeline is for chaining together processes. Messages are pushed to worker processes in a round-robin, which themselves may push to another stage of processing. This might be used to co-ordinate a workflow among sets of users (or indeed individuals).

Having duly dispensed with ado, here is rabbit.js.

All it needs is a bare-bones RabbitMQ and node.js installed; and, the node-amqp and Socket.IO libraries. Instructions and the locations of these things are in the README. (Do note that you need my fork of node-amqp.)

It also includes a tiny message socket server; that is, a node.js server that accepts socket connections and speaks in length-prefixed messages. Since it's all going through RabbitMQ, you can talk to the browsers hooked up with Socket.IO via a socket. You can also use the in-process pipe server from code running in node.js itself.

All in all, I am surprised how much I could get done with only a handful of lines of code and some technologies that each hit a sweet spot -- node.js for fun network server programming, Socket.IO for magical browser sockets, and RabbitMQ for the no-tears messaging.

Exchange to Exchange bindings

October 19th, 2010 by Matthew Sackman

Arriving in RabbitMQ 2.1.1, is support for bindings between exchanges. This is an extension of the AMQP specification and making use of this feature will (currently) result in your application only functioning with RabbitMQ, and not the myriad of other AMQP 0-9-1 broker implementations out there. However, this extension brings a massive increase to the expressivity and flexibility of routing topologies, and solves some scalability issues at the same time.

Normal bindings allow exchanges to be bound to queues: messages published to an exchange will, provided the various criteria  of the exchange and its bindings are met, pass through the various bindings and be appended to the queue at the end of each binding. That's fine for a lot of use cases, but there's very little flexibility there: it's always just one hop -- the message being published to one exchange, with one set of bindings, and consequently one possible set of destinations. If you need something more flexible then you'd have to resort to publishing the same message multiple times. With exchange-to-exchange bindings, a message published once, can flow through any number of exchanges, with different types, and vastly more sophisticated routing topologies than previously possible.

Read the rest of this entry »

RabbitMQ/0MQ bridge

October 18th, 2010 by Martin Sústrik

Recently, Michael Bridgen and I implemented a bridge to connect the RabbitMQ broker with applications using 0MQ for messaging.

Here it is:

http://github.com/rabbitmq/rmq-0mq

So: What kind of benefit the users can get by using both products in parallel?

In short, there are two different points of view involved here. RabbitMQ users will experience different benefits than users of 0MQ. Those already using both will simply get an easy way to interconnect the two technologies.

This is RabbitMQ blog so let's start with Rabbit-focused point of view.

First and most obviously, using 0MQ as a client for RabbitMQ broker makes sense on the platforms where there's no AMQP client available. Ever fancied speaking to RabbitMQ broker from Lisp? Or -- say -- from Cobol? The bridge may come handy here.

The same applies to OS/HW platforms. Do you need to speak to the broker from a device heavily constrained on memory, with very slow CPU or limited battery life? 0MQ client is pretty concise and very fast, which means that you'll get reasonable performance even on slow chips. Effectiveness also translates to lower power consumption, which in turn makes the battery last longer. The r0mq bridge itself is collocated with the RabbitMQ broker requiring no additional resources on the client.

Another usage of the bridge is much more sophisticated. If you are using RabbitMQ in very simple scenarios you may not even appreciate it. If you are managing a big geographically distributed system, it may become a lifesaver. The bridge can be used to interconnect RabbitMQ brokers into loose federation. Just edit the configuration files for the two brokers and everything will just work. You won't have to care about order of starting up the brokers, managing the reconnections after network disruptions etc. A nice feature is that these federations are truly distributed. There's no need for central governance of the federation. Simply connect your broker to a broker in a different company. That one in turn connects to brokers in yet different companies etc. Ultimately you end up with loose world-wide federation maintained in collaborative fashion by all the participants.

Yet one more advantage you may take advantage of is efficient usage of network resources. Unlike AMQP, 0MQ allows you to split your messaging traffic into logically separate flows. You pass your entertainment video in a separate flow from commands used to keep the aircraft flying. This not only means that you'll never experience head-of-line blocking problems, but also that network-level QoS can be set separately for the entertainment channel and for the steering system. Also, network engineers will appreciate that you can monitor the network traffic flow-by-flow rather than having a big opaque chunk of bandwidth used by "messaging".

Finally, 0MQ is bundled with OpenPGM library which implements a reliable mutlicast protocol called PGM. The r0mq bridge thus allows to multicast messages from RabbitMQ broker to the clients (0MQ clients to be precise -- AMQP has no multicast support). This kind of functionality is extremely useful in scenarios where a lot of identical data is passed to many boxes on the LAN. If a separate copy of each datum is sent to each subscriber, you can easily exceed capacity of your network. With multicast, data is sent once only to all the subscribers thus keeping the bandwidth usage constant even when the number of subscribers grows.

When you looking at r0mq bridge from the other side, you are probably developing at a low level using 0MQ as your network transport.

Using RabbitMQ broker has some obvious uses. The most trivial of them is to use the broker as a bridge to connect 0MQ applications to AMQP, STOMP or XMPP applications.

However, the real use case is to use RabbitMQ as a "device" in 0MQ network. 0MQ comes with few simple precompiled devices. Some hardware can be used as 0MQ device (say IP switch in case of multicast transport). There've been some attempts to create more sophisticated devices but these are in very early stages of development. So, what 0MQ developers are missing is a full-blown, sophisticated and production-ready device.

RabbitMQ broker can serve as such a device. First of all, it has been deployed widely and thus it is stable enough to be used in production environment safely. As for the feature set, it offers much more than anything found in 0MQ world. The two most useful features are persistence and monitoring.

Persistence means that the messages passing through the broker are saved on the disk. When you shut down the broker, if the box crashes because of power outage or a different technical problem, the messages are still available on the disk. When the broker is restarted they will be sent further as if the crash hasn't happened. It is similar to how email works.

Monitoring is asked for maybe even more often than persistence. Once you have a non-trivial system you want to know what each node is doing: How many messages there are stored for a particular feed What's the current throughput? And so on. RabbitMQ can tell you these things through its command-line tools or through the management plugin.

As a conclusion I would like to stress that bridging 0MQ and RabbitMQ is not just that dumb kind of bridge you get when you bridge two incompatible but more or less equivalent products. RabbitMQ and 0MQ are focusing on different aspects of messaging. 0MQ puts much more focus on how the messages are transferred over the wire. RabbitMQ, on the other hand, focuses on how messages are stored, filtered and monitored. By combining the two technologies you can get the best from both worlds.

Enjoy!

Prompt-a-licious

October 2nd, 2010 by Michael

I am setting up my old MacBook, reclaimed from my housemate, to be usable for the programmings.

The first step was to install homebrew. I'm finding it a bit friendlier than macports, which seems to be irretrievably broken on the other MacBook.

After a few more steps (git, mercurial, node, rabbitmq of course), I found myself missing my pretty hg-prompt bash prompt. But I'm working with git much more these days, so I wondered if there was something that could do both.

There is: vcprompt, and what do you know it's in homebrew.

$ brew install vcprompt

To get the pretty prompt, I more or less transcribed what I had from hg-prompt. In .bashrc:

D=$'\e[37;40m'
PINK=$'\e[35;40m'
GREEN=$'\e[32;40m'
ORANGE=$'\e[33;40m'

vc_ps1() {
    vcprompt -f "(%n:${PINK}%b${D}${GREEN}%u%m${D})" 2>/dev/null
}

export PS1='${GREEN}\[email protected]\h${D} in ${ORANGE}\w${D}$(vc_ps1)\n$ '

By the way, if like me you forget which of .bashrc and .bash_profile is for what, this post explains it.

If you want to get fancy, there's a guide to customising the bash prompt on the Arch Linux wiki.

Broker vs Brokerless

September 22nd, 2010 by alexis

The RabbitMQ team has been working with Martin Sustrik to provide code and documentation for using RabbitMQ and ZeroMQ together.  Why is this a good idea?  Because the broker and brokerless approaches are complementary.  We'll be posting more about this as the codebase evolves.  This post is introductory and can be seen as commentary on Ilya Grigorik's excellent introduction to ZeroMQ and the InfoQ summary of Ilya's article.

I like ZeroMQ and think it is useful - of which more below.  But I have seen some brash claims made on its behalf.  This can lead to confusion.

So what is the 'brokerless' model?  In the comments to Ilya's and the InfoQ post, ZeroMQ is compared to SCTP and to JGroups.  These are important technologies and form a helpful starting point for thinking about brokerless messaging patterns.  Let's look at what you might need if you combine messaging (like SCTP) with pubsub groups (like JGroups) to make arbitrary networks using 'brokerless' peers.

Some things you might need in a brokerless network

If you set up a brokerless messaging network, three things that you might need are: discovery, availability and management.

Discovery is the problem of maintaining a roster of peers that a system can send messages to, and who can join this roster.

Availability is the problem of dealing with peers disappearing from time to time.  For example if you have 50 subscribers to a feed, and only 40 of them are available to receive updates, should you keep a copy of their messages until they reappear?  That could mean "for a very long time".   And if you do keep messages and lists of "who has seen what", then where is it best to do this?

This is also a problem when message receivers do not respond quickly.  To quote from Martin Sustrik of ZeroMQ, "You can never differentiate between 'network error and 'no response received'. TCP in no better. You'll have accept that or keep with a single box."

Management is an interesting area for analysis too.  ZeroMQ's model aligns messaging closely with sockets.  This means that, like in TCP, 'any' communication network can be implemented in such a way that it provides some messaging capability.  But, networks can be arbitrarily complex.  For example unless you don't care about it (and you may not) management of "who is connected to who, and who can be connected to who" can get complicated.   This kind of management problem gets more difficult the more you scale.  Models like JGroups usually make this problem go away by making a simplifying assumption, i.e.: everyone in the group talks to everyone else in the group.  Easy 🙂

I am not suggesting that you always need these things.  The ZeroMQ philosophy is to home right in on networking, and this creates focus.  But if you do need them then you might end up implementing them yourself.  Enter the broker...

How can a broker help to solve these problems

Brokers can provide solutions for discovery, availability and management.  They can also form reliable networks, e.g. for email delivery and instant messaging services.

First: what is a 'broker'?  It is both a leader, and an intermediary.

A broker is a leader.  In distributed computing, the problems of management, discovery and availability are typically solved by electing a leader among the set of distributed components.  In the world of "messaging", such a leader is usually known as a "broker".  Stating that in order to be a leader, you need to be a broker, makes it much easier to work out who is the leader, than in a completely brokerless system in which "anyone can lead, but nobody knows how".

A broker is also an intermediary.  For example, instead of having to connect everyone in the group directly, communicators simply connect to the broker (or brokers).  A broker may also be used to solve availability problems such as "offline consumer", by providing persistence and managing recovery on behalf of systems that cannot do it themselves.

Thus, brokers simplify network design by making reasonable assumptions.  Of course, when those assumptions don't hold, you may not want a broker.

Brokers are not 'centralized'

A commonly held misconception about brokers is that they are 'centralized'.  Brokers are NOT necessarily a 'centralized' solution.  Intermediaries can be decentralized.  You can have multiple brokers in a single network in order to increase throughput and availability.  Sometimes these networks of servers are called federations.  Note that individual brokers do not need to be 'highly available' in order to have a redundant network of servers.

This is, for example, how email (SMTP) and XMPP networks work.  Both email and instant messaging are brokered models, and both use multiple brokers in a simple and redundant way.  For example, mail transfer agents provide a delivery and routing network for email.  It would be difficult to come up with a design for this that was completely peer to peer, without reinventing 'special peers' - also known as brokers.

So what model is simplest?

Peer to peer models are not inherently more or less simple than brokered models.  If you do not need discovery, availability, management, or intermediation then it may be simpler to not use them.  But if you need them, it may be simpler to not implement them yourself.

Networks of servers (brokers) are not more or less redundant or decentralized than networks of clients (peers).  Both the broker and brokerless model have their pros and cons in terms of reliability, and other considerations eg latency.

The two models solve different problems.

For example, RabbitMQ and ZeroMQ are complementary.  From a RabbitMQ point of view ZeroMQ is a 'smart client' that can use its buffers like a queue.  That's useful in some cases.  From a ZeroMQ point of view, RabbitMQ is a network device that provides services that you would not necessarily want to have to implement yourself.

We want our customers and users to always have the best toolset available which is why we have provided the Github repo for you to play with.  Thanks again to Martin Sustrik for his work on this.

Watch this space for more on this interesting area of work and discussion.

RabbitMQ on github

September 20th, 2010 by David

We've received quite a few requests recently for us to put the RabbitMQ code on github.

RabbitMQ is open source, and the Mercurial repositories where we work on the code are publicly accessible. But github is rapidly establishing itself as the Facebook of open-source development: It makes it easy to follow projects and participate in their development, all within a slick web-based UI.

So from today, we are mirroring our repositories to github. You can find them at http://github.com/rabbitmq. The repositories on github track our Mercurial repositories with a delay of a few minutes.

The main development of RabbitMQ will continue to take place on Mercurial. Converting our development workflow and infrastructure to git would take a lot of effort that we'd prefer to spend improving RabbitMQ. And besides, members of the team differ in their opinions about the relative merits of hg and git.

If you wish to contribute to RabbitMQ, we are happy to receive changes via github, or Mercurial hosting sites such as bitbucket, or even as old-fashioned patches!