Flow Control

There are two flow control mechanisms in RabbitMQ. Both work by exerting TCP backpressure on connections that are publishing too fast. They are:

Both mechanisms will temporarily block connections - the server will pause reading from the sockets of connected clients which send content-bearing methods (such as basic.publish) which have been blocked. Connection heartbeat monitoring will be disabled too.

The intent here is to introduce a flow control mechanism that throttles producers but lets consumers continue unaffected. However, since AMQP permits producers and consumers to operate on the same channel, and on different channels of a single connection, this logic is necessarily imperfect. In practice that does not pose any problems for most applications since the throttling is observable merely as a delay. Nevertheless, other design considerations permitting, it is advisable to only use individual AMQP connections for either producing or consuming.

Per-Connection Flow Control

RabbitMQ will block connections which are publishing too quickly for queues to keep up. No configuration is required.

A blocked connection will show a state of blocked in rabbitmqctl and the management plugin HTTP API. Typically such fast-publishing connections will be blocked and unblocked several times per second, so inspecting a connection's state may not show per-connection flow control happening clearly. The last_blocked_by and last_blocked_age fields in rabbitmqctl and the management API can be inspected to determine whether flow control is taking place.

The management plugin web UI unifies this information into a single field, showing flow for connections which have recently been in flow control.

Memory-Based Flow Control

The RabbitMQ server detects the total amount of RAM installed in the computer on startup and when rabbitmqctl set_vm_memory_high_watermark fraction is executed. By default, when the RabbitMQ server uses above 40% of the installed RAM, it raises a memory alarm and blocks all connections. Once the memory alarm has cleared (e.g. due to the server paging messages to disk or delivering them to clients) normal service resumes.

When running RabbitMQ in a cluster, the memory alarm is cluster-wide; if one node goes over the limit then all nodes will block connections.

The default memory threshold is set to 40% of installed RAM. Note that this does not prevent the RabbitMQ server from using more than 40%, it is merely the point at which publishers are throttled. Erlang's garbage collector can, in the worst case, cause double the amount of memory to be used (by default, 80% of RAM). It is strongly recommended that OS swap or page files are enabled.

32-bit architectures tend to impose a per process memory limit of 2GB. Common implementations of 64-bit architectures (i.e. AMD64 and Intel EM64T) permit only a paltry 256TB per process. 64-bit Windows further limits this to 8TB. However, note that even under 64-bit OSes, a 32-bit process frequently only has a maximum address space of 2GB.

Configuring the Memory Threshold

The memory threshold at which the flow control is triggered can be adjusted by editing the configuration file. The example below sets the threshold to the default value of 0.4:

[{rabbit, [{vm_memory_high_watermark, 0.4}]}].

The default value of 0.4 stands for 40% of installed RAM or 40% of available virtual address space, whichever is smaller. E.g. on a 32-bit platform, if you have 4GB of RAM installed, 40% of 4GB is 1.6GB, but 32-bit Windows normally limits processes to 2GB, so the threshold is actually to 40% of 2GB (which is 820MB).

A value of 0 makes the memory alarm go off immediately and thus disables all publishing (this may be useful if you wish to disable publishing globally; use rabbitmqctl set_vm_memory_high_watermark 0). To prevent the memory alarm from going off at all, set some high multiplier such as 100.

The memory limit is appended to the RABBITMQ_NODENAME.log file when the RabbitMQ server starts:

=INFO REPORT==== 29-Oct-2009::15:43:27 ===
Memory limit set to 2048MB.
The memory limit may also be queried using the rabbitmqctl status command.

The threshhold can be changed while the broker is running using the rabbitmqctl set_vm_memory_high_watermark fraction command. This command will take effect until the broker shuts down. The corresponding configuration setting should also be changed when the effects should survive a broker restart. The memory limit may change on systems with hot-swappable RAM when this command is executed without altering the threshhold, due to the fact that the total amount of system RAM is queried.

Limited Address Space

When running RabbitMQ inside a 32 bit Erlang VM in a 64 bit OS, the addressable memory is limited. The server will detect this and log a message like:

=WARNING REPORT==== 19-Dec-2013::11:27:13 ===
Only 2048MB of 12037MB memory usable due to limited address space.
Crashes due to memory exhaustion are possible - see
http://www.rabbitmq.com/memory.html#address-space

The memory alarm system is not perfect. While stopping publishing will usually prevent any further memory from being used, it is quite possible for other things to continue to increase memory use. Normally when this happens and the physical memory is exhausted the OS will start to swap. But when running with a limited address space, running over the limit will cause the VM to crash.

It is therefore strongly recommended that when running on a 64 bit OS you use a 64 bit Erlang VM.

Configuring the Paging Threshold

Before the broker hits the high watermark and blocks publishers, it will attempt to free up memory by instructing queues to page their contents out to disc. Both persistent and transient messages will be paged out (the persistent messages will already be on disc but will be evicted from memory).

By default this starts to happen when the broker is 50% of the way to the high watermark (i.e. with a default high watermark of 0.4, this is when 20% of memory is used). To change this value, modify the vm_memory_high_watermark_paging_ratio configuration from its default value of 0.5. For example:

[{rabbit, [{vm_memory_high_watermark_paging_ratio, 0.75},
           {vm_memory_high_watermark, 0.4}]}].

The above configuration starts paging at 30% of memory used, and blocks publishers at 40%.

It is possible to set vm_memory_high_watermark_paging_ratio to a greater value than 1.0. In this case queues will not page their contents to disc. If this causes the memory alarm to go off, then producers will be blocked as explained above.

Unrecognised platforms

If the RabbitMQ server is unable to recognise your system, it will append a warning to the RABBITMQ_NODENAME.log file. It then assumes than 1GB of RAM is installed:

=WARNING REPORT==== 29-Oct-2009::17:23:44 ===
Unknown total memory size for your OS {unix,magic_homebrew_os}. Assuming memory size is 1024MB.

In this case, the vm_memory_high_watermark configuration value is used to scale the assumed 1GB RAM. With the default value of vm_memory_high_watermark set to 0.4, RabbitMQ's memory threshold is set to 410MB, thus it will throttle producers whenever RabbitMQ is using more than 410MB memory. Thus when RabbitMQ can't recognize your platform, if you actually have 8GB RAM installed and you want RabbitMQ to throttle producers when the server is using above 3GB, set vm_memory_high_watermark to 3.

It is advised you do not set the threshold above 50% of your installed RAM.

Disk-Based Flow Control

RabbitMQ can also block producers when free disk space drops below a certain limit. This is a good idea since even transient messages can be paged to disk at any time, and running out of disk space can cause the server to crash. By default RabbitMQ will block producers, and prevent memory-based messages from being paged to disk, when free disk space drops below 50MB. This will reduce but not eliminate the likelihood of a crash due to disk space being exhausted. In particular, if messages are being paged out rapidly it is possible to run out of disk space and crash in the time between two runs of the disk space monitor. A more conservative approach would therefore be to set the limit to the same as the amount of memory installed on the system (see below).

Global flow control will be triggered if the amount of free disk space drops below a configured limit. The free space of the drive or partition that the broker database uses will be monitored every minute to determine whether the alarm should be raised or cleared. Monitoring will start as soon as the broker starts up, causing an entry in the broker logfile:

=INFO REPORT==== 23-Jun-2012::14:52:41 ===
Disk free limit set to 953MB
Monitoring will be disabled on unrecognised platforms, causing an entry such as the one below:
=WARNING REPORT==== 23-Jun-2012::15:45:29 ===
Disabling disk free space monitoring

When running RabbitMQ in a cluster, the disk alarm is cluster-wide; if one node goes under the limit then all nodes will block connections.

RabbitMQ will periodically check the amount of free disk space. The frequency with which disk space is checked is related to the amount of space at the last check (in order to ensure that the disk alarm goes off in a timely manner when space is exhausted). Normally disk space is checked every 10 seconds, but as the limit is approached the frequency increases. When very near the limit RabbitMQ will check as frequently as 10 times per second. This may have some effect on system load.

Configuring the Disk Free Space Limit

The disk free space limit is configured with the disk_free_limit setting. By default 50MB is required to be free on the database partition. (See the description of file locations for the default location). This configuration file sets the disk free space limit to 1GB:

[{rabbit, [{disk_free_limit, 1000000000}]}].
It is also possible to set a free space limit relative to the RAM in the machine. This configuration file sets the disk free space limit to the same as the amount of RAM on the machine:
[{rabbit, [{disk_free_limit, {mem_relative, 1.0}}]}].

Client Notifications

RabbitMQ provides an extension to AMQP 0-9-1 that allows clients to be notified when a connection is blocked.