"How much memory is my queue using?" That's an easy question to ask, and a somewhat more complicated one to answer. RabbitMQ 3.4 gives you a clearer view of how queues use memory. This blog post talks a bit about that, and also explains queue memory use in general. (more…)
Archive for the ‘HowTo’ Category
In this blog post we are going to address the problem of controlling the access to a particular resource in a distributed system. The technique for solving this problem is well know in computer science, it's called Semaphore and it was invented by Dijkstra in 1965 in his paper called "Cooperating Sequential Processes". We are going to see how to implement it using AMQP's building blocks, like consumers, producers and queues.
Different services in our architecture will require a certain amount of resources for operation, whether these resources are CPUs, RAM or disk space, we need to make sure we have enough of them. If we don't put limits on how many resources our servers are going to use, at some point we will be in trouble. This happens with your database if it runs out of file system space, your media storage if you fill it with images and never move them somewhere else, or your JVM if it runs out of RAM. Even your back up solution will be a problem if you don't have a policy for expiring/deleting old backups. Well, queues are no exception. We have to make sure that our application won't allow the queues to grow for ever. We need to have some strategy in place to delete/evict/migrate old messages.
With RabbitMQ 3.2.0 we introduced Consumer Priorities which not surprisingly allows us to set priorities for our consumers. This provides us with a bit of control over how RabbitMQ will deliver messages to consumers in order to obtain a different kind of scheduling that might be beneficial for our application.
When would you want to use Consumer Priorities in your code?
So we added support for federated queues in RabbitMQ 3.2.0. This blog post explains what they're for and how to use them. (more…)
RabbitMQ is a very extensible message broker, allowing users to extend the server’s functionality by writing plugins. Many of the broker features are even shipped as plugins that come by default with the broker installation: the Management Plugin, or STOMP support, to name just a couple. While that’s pretty cool, the fact that plugins must be written in Erlang is sometimes a challenge. I decided to see if it was possible to write plugins in another language that targeted the Erlang Virtual Machine (EVM), and in this post I’ll share my progress.
In the last couple of months I’ve been paying attention to a new programming language called Elixir that targets the EVM, and in the last week it became immensely popular inside the Erlang community (and other circles) since Joe Armstrong, the father of Erlang, tried the language, and liked it very much. So I said, OK, lets give it a try.
RabbitMQ includes a bunch of cool new features. But in order to implement some of them we needed to change some things. So in this blog post I'm going to list some of those things in case you need to do anything about them. (more…)
You have a queue in Rabbit. You have some clients consuming from that
queue. If you don't set a QoS setting at all (
Rabbit will push all the queue's messages to the clients as fast as
the network and the clients will allow. The consumers will balloon in
memory as they buffer all the messages in their own RAM. The queue may
appear empty if you ask Rabbit, but there may be millions of messages
unacknowledged as they sit in the clients, ready for processing by the
client application. If you add a new consumer, there are no messages
left in the queue to be sent to the new consumer. Messages are just
being buffered in the existing clients, and may be there for a long
time, even if there are other consumers that become available to
process such messages sooner. This is rather sub optimal.
So, the default QoS
prefetch setting gives clients an unlimited
buffer, and that can result in poor behaviour and performance. But
what should you set the QoS
prefetch buffer size to? The goal is to
keep the consumers saturated with work, but to minimise the client's
buffer size so that more messages stay in Rabbit's queue and are thus
available for new consumers or to just be sent out to consumers as
they become free. (more…)
One of the problems we face at the RabbitMQ HQ is that whilst we may know lots about how the broker works, we don't tend to have a large pool of experience of designing applications that use RabbitMQ and which need to work reliably, unattended, for long periods of time. We spend a lot of time answering questions on the mailing list, and we do consultancy work here and there, but in some cases it's as a result of being contacted by users building applications that we're really made to think about long-term behaviour of RabbitMQ. Recently, we've been prompted to think long and hard about the basic performance of queues, and this has lead to some realisations about provisioning Rabbits. (more…)