ZeroMQ =/= Erlang

Recently I saw a tweet saying "ZeroMQ Erlangizes everything!" or some such. While I realise that not everything posted on the web is meant seriously, it does seem there is a stream of similar claims lately that ought to be dammed.

In the article Multi-threading Magic[1], Pieter Hintjens and Martin Sústrik persuasively explain why concurrency is better served by message-passing than by locks and shared memory. And they are fair, I think, in their analysis -- except for the insinuation that using ZeroMQ transforms your chosen programming language into a domestic Erlang.

Mid-way through there is a sleight-of-hand. After mentioning the ingredients of Erlang, the article chooses just one of them -- message passing -- to be the key ingredient, and ignores the rest. But the others are at least as important! Let's look at all of them:

  • Fast process creation/destruction
  • Ability to support » 10 000 concurrent processes with largely unchanged characteristics.

A programming model where processes are lightweight values -- and a good scheduler -- make concurrent programming much easier, in a similar way to garbage collection. It frees you from resource micro-management so you can spend more time reasoning about other things.

  • Fast asynchronous message passing.

This is what ZeroMQ gives you. But it gives you it in a form different to that of Erlang: in Erlang, processes are values and message-passing channels are anonymous; in ZeroMQ, channels are values and processes are anonymous. ZeroMQ is more like Go than Erlang. If you want the actor model (that Erlang is based on), you have to encode it in your language of choice, yourself.

  • Copying message-passing semantics (share-nothing concurrency).

Notably, Erlang enforces this. In other languages, shared memory and the trap of using it (usually unwittingly) doesn't go away.

  • Process monitoring.

Erlang comes with a substantial library, battle-tested over decades, for building highly concurrent, distributed, and fault-tolerant systems. Crucial to this is process monitoring -- notification of process termination. This allows sophisticated process management strategies; in particular, using supervisor hierarchies to firewall core parts of the system from more failure-prone parts of the system.

  • Selective message reception.

You can use poll with ZeroMQ to efficiently use many channels in a single process at once; however, you don't get to block on particular kinds of message, meaning you have to buffer messages you don't want to deal with yet, or keep complex state around.

ZeroMQ is a lovely bit of kit, I would not argue otherwise. My point is that it does not magically give you fool-proof concurrent programming; neither does Erlang, but it's an awful lot further ahead than you may have been led to believe. With ZeroMQ there are still a number of things you have to invent, mimicking Erlang or otherwise.

[1] Multi-threading Magic. It's well worth a read.

21 Responses to “ZeroMQ =/= Erlang”

  1. sustrik Says:

    Agreed. The difference is worth of pointing out. Erlang has a lot of functionality that's 0MQ has not. 0MQ, OTOH, has messaging patterns which are not available in Erlang.

  2. Janko M. Says:

    Very good informative, no pun article. I thought you will be bashing ZMQ since you are from RabbitMQ :).

    People lately like to mix everything that's buzzing and has some smell of concurrency/async like Tornado, Node.js, ZMQ with Erlang.

  3. Ian Barber Says:

    Good post, completely agree. One of the things that struck me as a particularly powerful combination was using ZeroMQ in active mode with pattern matching for selective receive, definitely a feature that just enormously simplifies protocol handling.

  4. Michael Says:

    @Ian -- I'd be careful of using active mode on a socket, it relinquishes any control over back-pressure you might have. It'd be good if the Erlang bindings for zmq had {active, once}. (Maybe the new bindings do?)

  5. Michael Says:

    @sustrik -- Well, in message send it's just process to process; but, at a higher level, genevent gives you pub/sub semantics, and genserver gives you request/reply semantics.

    The load-balancing/fair-queueing of req/rep and push/pull is interesting. Typically you would either rely on the scheduler -- i.e., expect that fair scheduling will even things out; just create a process to do the work as you need to; or, in certain cases, keep a pool of workers and a dispatch process (there's examples, with different motivations, in mochiweb and in rabbit, among other places). I've also seen a gen_lb in the wild, and I think Riak has a mechanism for sharing jobs out among nodes.

  6. Julian Morrison Says:

    As far as selective message reception goes, the dual of (Erlang) "from a pool of messages, specify the types to receive" is (Go/ZMQ) "from a pool of channels, specify the ones to select". One message type per channel.

  7. Bryan Says:

    I'm only passingly familiar with both. Could somebody clarify what this means? It sounds like gibberish to me at the moment.

    "in Erlang, processes are values and message-passing channels are anonymous; in ZeroMQ, channels are values and processes are anonymous"

  8. Michael Says:

    @Julian I am not so sure. How do you wait on "this specific reply"? In Erlang this is (get ready for no code formatting):

    Ref = make_ref(),
    Replier ! {Ref, Request}
       {Ref, Reply} -> ...
  9. Michael Says:

    @Bryan By "values" I mean something that you can e.g., construct and assign. In Erlang you construct processes explicitly, and communicate by naming them:

    P = spawn(fun() -> ... end),
    P ! message

    How the communication happens is implicit.

    With ZeroMQ, the communication channel is explicit -- I can (in Java, say), create and assign a thread, but I don't talk to it by naming it. Instead, I open a socket (and hope it has opened the other end). The communication channel is a value, and for the purpose of communication and often otherwise, the thread is anonymous.

  10. Pieter Hintjens Says:

    Nice blog, Michael. That article is somewhat dated, and the stream of claims you want to damn has long turned into a river of evidence.

    It is really not about the underlying technology, but the user experience. Technology is the basis for the user experience but cannot replace it. All language-centric solutions to software architecture (which form the majority of elite languages over the last decades) focus on technology and forget the user.

    Erlang may be great. I don't know, I've never used it. My knowledge of it is second hand, and told to me by Erlang developers, who consistently love and defend it.

    ZeroMQ is great. I do know, having used it in anger in many diverse projects now. My knowledge of it is based not on its implementation (which I admire but often fight with) but on its end user experience.

    And here is the thing. You give ZeroMQ to random developers who are used to building stand-alone apps, or at best weakly distributed apps. You show them the power, help them design some protocols, and suggest some use cases. And it takes about 1-2 days, and their brains begin (if they are not zombified by too many years of the enterprise) to whisper and jump and shout, and out come ideas. And then, in hours (not days) those ideas turn into running code.

    I've done this over and over. It doesn't require any switch away from known languages, tool-chains, plaforms. Just that little library, and some API layers. I just spent a week doing this, and we built prototypes of various distributed products, on Windows, Linux, Android, Mac OS/X, and HTML5. We worked in C, C++, Python, Java, JavaScript, and Node.js. We were seven, and we spent 5 days, and none in the room except me had tried or read about ZeroMQ before.

    This is the end user experience. Simple, fun, an easy step from a known environment into a distributed future full of potential.

    That is why ZeroMQ is not Erlang, nor any other programming language based vision of the future. No conversion required, you just add one more API to your stack, and BOOM! Your world changes.

    I'd apologise for any simplification of Erlang that helped explain the point we were making about ZeroMQ's approach to multithreading, but then again, no. The point is that learning a new language just to proof some concepts is an impossible barrier for most people, in most cases. Does Erlang run on Android? In the browser? In my existing toolchain? In the brains I have in the room this week?

    And if learning a new language to prove some concepts is impossible, it's finished. Erlang may be amazing (and I'm sure it is) but it's a dead language. As dead as Sanskrit. The future is Python, Java, Javascript, and C (and maybe still C++ though that is another elite language).

    That is my view, anyhow, based on the observation that simplicity always beats functionality.

  11. Abioy Says:

    Nice post! Will zeromq fans come and fight back?

  12. Michael Says:

    @Pieter, I think you are doing a disservice to those you call "users", in particular by dividing the world into "elite languages" and -- well, I suppose the implication is that Python, Java, JavaScript and C are "demotic languages". This is a nonsensical distinction. It's completely plausible that six people and one instructor could spend a week learning Erlang, and in fact, this happens routinely.

    If you spend some time learning Erlang, it is very likely that you will learn a lot about how to write programs even if they aren't in Erlang. This isn't something special and magical about Erlang -- I would say the same of Ruby, Scheme, and Smalltalk for instance. It is not an impossible barrier at all. (I recommend the book "Seven languages in Seven weeks" for those who want a run-up)

  13. Michael Says:

    @Mik -- wordpress appears to have lost your comment, would you repost?

  14. Pieter Hintjens Says:

    It's nonsensical to pretend that all languages are equivalent. There's a reason we're writing in English, not in Dutch. Popularity does matter. If I was to tell you, "here's a great recipe for pancakes, but it only works if you learn Dutch first", you'd rightly mock me. And yet, in a week, you can learn the basics of Dutch.

    So you believe that computer languages teach you about programming, and you like to invest in programming languages. For me, languages are disposable tools, tainted by the bias and preconceptions of their designers, and highly subjective in their view of what software is. Learning all these biases doesn't make you a better designer, just one with more and stronger biases.

    Here's an alternative view of what drives software design:

    And you'll perhaps agree that it's free & open protocols (like AMQP/0.9.1, and ZeroMQ's ZMTP) that are the most powerful tools we have for dividing large problems into pieces that fit our psychological limitations. So please point me to the Erlang wire level protocol so I can build interoperable stacks in other languages.

  15. Michael Says:

    Pieter, your analogy simply doesn't follow. I'd ask you to translate the recipe into English. This is possible because you can read and write both English and Dutch -- in other words, you benefit from knowing both languages.

    Programming languages are equivalent in so far as they are in general Turing complete. However that does not mean they are equally expressive, or as you say, popular. A programming language represents a set of design trade-offs. If you invest in learning programming languages, you learn why those trade-offs were made and how far they are justified. It is important to appreciate these things because it is, after all, programming languages that we use to speak most closely with computers.

    Here is the Erlang distribution protocol and the byte-level format Have fun!

  16. mikhailfranco Says:

    It looks like the Java world is getting their Erlang too:

    Typesafe is a combination of Scala and Akka to give Erlangy goodness to the JVM. Although when Martin Odersky and Jonas Boner are leading, you know it will definitely not be "informally-specified bug-ridden slow".

    Another interesting example of a system that has spent a long time reimplementing Erlang is OpenCog.

    It seems that OpenCog (and Novamente before 2008) have spent 10 years building a distributed conceptual graph in C++ together with many modules for AI processing. They did flirt with the idea of Erlang in 2009, but by then it was too little too late.

    It's hard for me say how much of that time was spent on the infrastructure, as opposed to the AI bits, a guess of a 50/50 split might not be too far out, especially in the early years. But whatever the ratio, time spent rewriting distributed infrastructure is time not spent developing the core AI algorithms.

    I think this supports Pieter's point that people choose what they are familiar with, and what is easiest to integrate with their target audience, even if they waste years doing it.


  17. mikhailfranco Says:

    Erlang has evolved for resilience in the real world based on two principles: - robustness: design to avoid the major causes of complexity and failure (hence functional programming, limit side effects, immutable state, distribution, share nothing, message passing, timeouts, bignums, etc.) - humility: you will not succeed, failure will happen, so design some failure recovery too (hence links, monitors, supervisor hierarchies, OTP design patterns)

    Virding's Rule says:

    "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang."

    The corollary is that even when individual components of a system are of good quality, there will be leaky abstractions and coordination issues. The problem arises when an engineer burdened with the anti-wisdom of the 'Fallacies of Distributed Computing' comes to design a system.

    They will naturally put sub-system and layer boundaries in the wrong places, and the fallacies will become evident through unexpected failure of the contracts across these boundaries. So even if the pieces are well designed, the system as a whole will be prone to failure. There will follow a series of painful rewrites and refactorings as the engineers struggle to manage the competing forces of distributed complexity and robustness. They will eventually rediscover many of the design lessons incorporated within Erlang/OTP. Their system may run faster than Erlang when it is up, but they will never win the integrated metric of being the fastest system that stays running.

    A similar discussion is being played out in the Hadoop world at the moment.

    They are trying to decide how the monolithic architecture can be decomposed into a modular set of interchangeable components provided from different sources. Although, to be fair, they are aiming to support much larger clusters (> 6000) than Erlang can support out-of-the-box. But just think what could be achieved if all that brainpower from Yahoo and beyond was devoted to building hierarchical scalability into Erlang.


  18. Pieter Hintjens Says:

    @Michael, thanks for the pointer to the Erlang protocol. I'll think of some more inaccurate analogies later, but it's great to see other languages being able to call Erlang services. Perhaps Erlang is more than just another language.

  19. mikhailfranco Says:

    I think the best comparison for 0MQ is not Erlang, but DDS, an OMG standard for low-latency pub-sub that uses UDP and TCP, it has a well-specified wire protocol (DDSI/RTPS/CDR), has many language bindings, messages are defined with IDL, and it has layered support for persistent queues, selective receive using a dialect of SQL, and an object layer.

    Historically, the problems with DDS were overly complex APIs, and expensive commercial implementations, but both of those are now solved with the new API bindings and open source licensing (e.g. PrismTech )


  20. Pieter Hintjens Says:

    @mikhailfranco: there are many, many products you can compare with 0MQ. That's kind of a game. Hintjens Fourth Law says "Any messaging product, including 0MQ, is just 0MQ done incorrectly".

    Explaining that "0MQ is like DDS but with a simpler API and lower cost" may be accurate, but isn't helpful, though. Explaining that "0MQ is like Erlang, but for any language" isn't accurate, but it is very helpful, and ironically it works both ways, getting people to explore Erlang who otherwise would never have looked at it.

    Martin wrote a useful backgrounder to 0MQ (it's like writing the script after shooting the movie, but hey...) which helps explain what is going on:

  21. alexis Says:

    @pieter - love it. What are the other laws?

    Here are my laws of middleware:

    Law 1) all middleware fundamentally does the same thing, move shit around

    Law 2) despite 1, for any piece of middleware, there are infinitely many things it does not do

    Law 3) it is the job of the Consultant to list them

    Law 4) [therefore..] middleware is Evil

    I agree that Martin's backgrounder is tops.