rabbitmq-tracing – a UI for the firehose

While the firehose is quite a cool feature, I always thought that it was a shame we didn't have a simple GUI to go on top and make it accessible to system administrators. So I wrote one. You can download it here.

It's an extension to the management plugin, and requires 2.6.0 / 2.6.1. Once you've installed it and restarted Rabbit, you should see another tab labelled "Tracing" in management. (Edit: as of RabbitMQ 3.0.0, "Tracing" is under the "Admin" tab.) From here you can start and stop tracing, and download or delete log files. Hopefully the interface is fairly obvious.

Currently rabbitmq-tracing supports two log formats: text and json. text is designed to be human-readable, and looks like this:

2011-9-9 10:57:24: Message published

Node: rabbit@smacmullen Exchange: direct Routing keys: [<<"5d07bff4-1708-4a5d-87f9-a14177d6681b">>] Properties: [] Payload: Hello world

================================================================================ 2011-9-9 10:57:24: Message received

Node: rabbit@smacmullen Exchange: direct Queue: amq.gen-PJfnaKdg7AmsWmYTUeuApw== Routing keys: [<<"5d07bff4-1708-4a5d-87f9-a14177d6681b">>] Properties: [] Payload: Hello world


json is designed to be machine readable and consists of a file with one JSON structure per line. Note that this means the entire file is not itself a JSON structure, you need to split it by line first. (The reason for this is so that we can treat it like a normal log file and just append to it.)

There are two configuration options:

  • "directory". This controls where the log files go. It defaults to "/var/tmp/rabbitmq-tracing".
  • "username". The name of a user as which to create the tracing queues and bindings.

A complete configuration might look like:

[{rabbitmq_tracing, [{directory, "/my/log/file/location"},
                     {username,  "guest"}]}].

As with any new plugin there are of course a couple of caveats. On my workstation, rabbitmq-tracing can write about 2000 msg/s to a log file. You should be careful using rabbitmq-tracing if you think you're going to capture more messages than this. Of course, any messages that can't be logged are queued until they can be.

Also, the code to serve up the log files over HTTP is pretty dumb: it loads the whole log into memory (unfortunately webmachine doesn't really let us stream files). If you have large log files you may wish to transfer them off the server in some other way.

So, what do you think? Is this useful to you? How could it be extended?

Tags: , , ,

7 Responses to “rabbitmq-tracing – a UI for the firehose”

  1. Joe Weeks Says:

    This is great- very useful.

    Would it be feasible to extend this (or the firehose feature itself) to provide pluggable payload marshaling?

    For example, we're using Protocol Buffers. When decoded, there is often plenty of human-readable data and metadata in our messages. We can decode ourselves after the fact, but it would be much more convenient to write a plug-in and have decoded payloads available to system administrators for download directly from this GUI.

    Do you have any thoughts about whether that sort of feature be a good fit for this or firehose?

    Thanks again for another great plug-in!

  2. Simon MacMullen Says:

    Hey Joe, thanks.

    I don't think such marshalling belongs in the firehose itself, that's supposed to be pretty minimal and unopinionated. It might be a decent fit for this plugin.

    Were you expecting to write such plugins in Erlang or something else?

  3. Joe Weeks Says:

    That makes sense.

    I would have expected to have to write that kind of plugin in Erlang. However, given the weakness of my Erlang-fu, the prospect of writing it in another language, maybe through some JInterface integration, would be appealing.

    That being said, an Erlang API would certainly be sufficient for my use case.

  4. Simon MacMullen Says:

    My fear would be that JInterface would make any deployment fairly horrific. Erlang should be quite easy though. I'll have a think...

  5. Bernhard Weisshuhn Says:

    Just tried the plugin with 2.6.1. As a logged in administrator user with all privileges, adding a trace resulted in a crash report about auth_failure. The full report is here:

    Any idea what could be wrong?

    P.S. The system had the guest-user removed, other than that everything should be pretty normal. Lots of plugins installed though.

  6. Simon MacMullen Says:

    Bernhard: Yes, I was dumb and the plugin assumed that the "guest" user existed. I've stuck an updated version here:

    You will need to specify the user to use for creating queues and bindings with a configuration file fragment like:

    {rabbitmq_tracing, [{username, "admin"}]}

  7. Bernhard Weisshuhn Says:

    ... and it works! Fantastic!

    Thanks a bunch, bkw