Menu

Broker vs Brokerless

The RabbitMQ team has been working with Martin Sustrik to provide code and documentation for using RabbitMQ and ZeroMQ together.  Why is this a good idea?  Because the broker and brokerless approaches are complementary.  We'll be posting more about this as the codebase evolves.  This post is introductory and can be seen as commentary on Ilya Grigorik's excellent introduction to ZeroMQ and the InfoQ summary of Ilya's article.

I like ZeroMQ and think it is useful - of which more below.  But I have seen some brash claims made on its behalf.  This can lead to confusion.

So what is the 'brokerless' model?  In the comments to Ilya's and the InfoQ post, ZeroMQ is compared to SCTP and to JGroups.  These are important technologies and form a helpful starting point for thinking about brokerless messaging patterns.  Let's look at what you might need if you combine messaging (like SCTP) with pubsub groups (like JGroups) to make arbitrary networks using 'brokerless' peers.

Some things you might need in a brokerless network

If you set up a brokerless messaging network, three things that you might need are: discovery, availability and management.

Discovery is the problem of maintaining a roster of peers that a system can send messages to, and who can join this roster.

Availability is the problem of dealing with peers disappearing from time to time.  For example if you have 50 subscribers to a feed, and only 40 of them are available to receive updates, should you keep a copy of their messages until they reappear?  That could mean "for a very long time".   And if you do keep messages and lists of "who has seen what", then where is it best to do this?

This is also a problem when message receivers do not respond quickly.  To quote from Martin Sustrik of ZeroMQ, "You can never differentiate between 'network error and 'no response received'. TCP in no better. You'll have accept that or keep with a single box."

Management is an interesting area for analysis too.  ZeroMQ's model aligns messaging closely with sockets.  This means that, like in TCP, 'any' communication network can be implemented in such a way that it provides some messaging capability.  But, networks can be arbitrarily complex.  For example unless you don't care about it (and you may not) management of "who is connected to who, and who can be connected to who" can get complicated.   This kind of management problem gets more difficult the more you scale.  Models like JGroups usually make this problem go away by making a simplifying assumption, i.e.: everyone in the group talks to everyone else in the group.  Easy :-)

I am not suggesting that you always need these things.  The ZeroMQ philosophy is to home right in on networking, and this creates focus.  But if you do need them then you might end up implementing them yourself.  Enter the broker...

How can a broker help to solve these problems

Brokers can provide solutions for discovery, availability and management.  They can also form reliable networks, e.g. for email delivery and instant messaging services.

First: what is a 'broker'?  It is both a leader, and an intermediary.

A broker is a leader.  In distributed computing, the problems of management, discovery and availability are typically solved by electing a leader among the set of distributed components.  In the world of "messaging", such a leader is usually known as a "broker".  Stating that in order to be a leader, you need to be a broker, makes it much easier to work out who is the leader, than in a completely brokerless system in which "anyone can lead, but nobody knows how".

A broker is also an intermediary.  For example, instead of having to connect everyone in the group directly, communicators simply connect to the broker (or brokers).  A broker may also be used to solve availability problems such as "offline consumer", by providing persistence and managing recovery on behalf of systems that cannot do it themselves.

Thus, brokers simplify network design by making reasonable assumptions.  Of course, when those assumptions don't hold, you may not want a broker.

Brokers are not 'centralized'

A commonly held misconception about brokers is that they are 'centralized'.  Brokers are NOT necessarily a 'centralized' solution.  Intermediaries can be decentralized.  You can have multiple brokers in a single network in order to increase throughput and availability.  Sometimes these networks of servers are called federations.  Note that individual brokers do not need to be 'highly available' in order to have a redundant network of servers.

This is, for example, how email (SMTP) and XMPP networks work.  Both email and instant messaging are brokered models, and both use multiple brokers in a simple and redundant way.  For example, mail transfer agents provide a delivery and routing network for email.  It would be difficult to come up with a design for this that was completely peer to peer, without reinventing 'special peers' - also known as brokers.

So what model is simplest?

Peer to peer models are not inherently more or less simple than brokered models.  If you do not need discovery, availability, management, or intermediation then it may be simpler to not use them.  But if you need them, it may be simpler to not implement them yourself.

Networks of servers (brokers) are not more or less redundant or decentralized than networks of clients (peers).  Both the broker and brokerless model have their pros and cons in terms of reliability, and other considerations eg latency.

The two models solve different problems.

For example, RabbitMQ and ZeroMQ are complementary.  From a RabbitMQ point of view ZeroMQ is a 'smart client' that can use its buffers like a queue.  That's useful in some cases.  From a ZeroMQ point of view, RabbitMQ is a network device that provides services that you would not necessarily want to have to implement yourself.

We want our customers and users to always have the best toolset available which is why we have provided the Github repo for you to play with.  Thanks again to Martin Sustrik for his work on this.

Watch this space for more on this interesting area of work and discussion.

8 Responses to “Broker vs Brokerless”

  1. Twitter Trackbacks for RabbitMQ » Blog Archive » Broker vs Brokerless - Messaging that just works [rabbitmq.com] on Topsy.com Says:

    [...] RabbitMQ » Blog Archive » Broker vs Brokerless - Messaging that just works rabbitmq.com/blog/2010/09/22/broker-vs-brokerless/ – view page – cached RabbitMQ is a complete and highly reliable enterprise messaging system based on the emerging AMQP standard Tweets about this link [...]

  2. Martin Sustrik Says:

    Nice article! It reminds me of TCP/IP. You can think of it as a simple brokerless technology as long as you have two boxes connected by a crossover cable. In real world, however, you have all kind of smart brokers in the middle. You start up with a simple network switch and ultimately end up with something like this:

    http://www.flickr.com/photos/cisco_pics/4406738473

    Martin

  3. Pieter Hintjens Says:

    Nice article but kind of confused. ØMQ does not mandate a brokerless architecture. As Martin says, for small sets you don't need brokers, and these cases can scale hugely without brokers so long as the topology remains simple (e.g. thousands of subscribers with one publisher). However any large connected set needs intermediates and ØMQ architectures most definitely use these. We call them "devices" and they are essential to any realistic ØMQ application.

    ØMQ devices are just brokers under another name. The key differences are that (a) apart from a few trivial devices, they exist in userspace, so must be developed or recycled, (b) they are highly specific to the messaging pattern, so they can be fast and simple, and (c) they can be embedded as threads in application processes.

    What emerges is that splitting different patterns apart (and this is where ØMQ really wins) makes each simpler to solve. We proved this years ago with Direct Mode in OpenAMQ. It also applies to reliability (easier to solve end-to-end than via a broker).

    So this is really not about broker vs. brokerless but rather about centralized vs. distributed messaging patterns. The two models attempt to solve much the same use cases, but do so in very different ways.

    It will be interesting to see how users given the two tools in a single package will respond. Today it seems clear that getting RabbitMQ as broker is far easier than building your own using ØMQ. AMQP does intermediation nicely enough. But ØMQ brokers are relatively trivial to make and people are making them, and they will eventually cover all the ground that AMQP brokers do.

  4. alexis Says:

    Pieter, thank-you.

    Your point about functionality existing in user space is spot on. A broker that provides (say) a persistent queue, is an application in a pubsub network. So what are the other interesting applications and use cases?

    It would be extremely interesting to see lots of use cases that help developers implement best practices in a wide range of cases. Please do help us to set those out - I think the Github project linked to above is a potential starting point.

    alexis

  5. John O'Hara Says:

    Alexis,
    I think this is a good, balanced description of the various trade offs that are made in designing networks of applications.

    As you say, people will deploy middleware and learn over time which kind suits their needs.

    For most of the use cases I see, brokered technology makes sense, it's just easier to live with and more productive for developers.

  6. Josh Perry Says:

    Does RabbitMQ have anything to cover interprocess messaging? I'm in the process of trying to architect a system that will be using messaging quite heavily. In the customer premise equipment(CPE) there will be a number of modules executing in the same process that will communicate with messages. There will also be a gateway module that subscribes to certain messages on this internal bus to publish them up to a broker in "the cloud"; this module will also inject messages inbound from the broker onto the internal bus.

    At this point I'm kind of thinking of using ZeroMQ as the internal messaging system and RabbitMQ for the broker in the datacenter and for the gateway module in the CPE.

  7. Michael Says:

    @Josh

    You can use certainly use Rabbit as a backplane for co-ordination. You may find ZeroMQ more suited to light-weight IPC. Using Rabbit to bridge into other networks (e.g., your data centre) is also sensible. Have a look at http://www.rabbitmq.com/blog/2010/10/18/rabbitmq0mq-bridge/

  8. What's a good distributed queue? - Quora Says:

    [...] and open source projects out there. A lot will depend on your needs: one size does not fit all: http://www.rabbitmq.com/blog/2010/09/22/broker-vs-brokerless/ If you have questions or need support, the RabbitMQ team and community welcomes questions - the [...]