Menu

Disk Alarms

When free disk space drops below a configured limit (50MB by default), an alarm will be triggered and all producers will be blocked. The goal is to avoid filling up the entire disk which will lead all write operations on the node to fail and can lead to RabbitMQ termination. To reduce the risk of filling up the disk, all incoming messages are blocked. Transient messages, which aren't persisted, are still paged out to disk when under memory pressure, and will use up the already limited disk space. If the disk alarm is set too low and messages are paged out rapidly, it is possible to run out of disk space and crash RabbitMQ in between disk space checks (at least 10 seconds apart). A more conservative approach would be to set the limit to the same as the amount of memory installed on the system (see the configuration below).

An alarm will be triggered if the amount of free disk space drops below a configured limit. The free space of the drive or partition that the broker database uses will be monitored at least every 10 seconds to determine whether the disk alarm should be raised or cleared. Monitoring will start as soon as the broker starts up, causing an entry in the broker logfile:

=INFO REPORT==== 23-Jun-2012::14:52:41 ===
Disk free limit set to 953MB
Monitoring will be disabled on unrecognised platforms, causing an entry such as the one below:
=WARNING REPORT==== 23-Jun-2012::15:45:29 ===
Disabling disk free space monitoring

When running RabbitMQ in a cluster, the disk alarm is cluster-wide; if one node goes under the limit then all nodes will block incoming messages.

RabbitMQ periodically checks the amount of free disk space. The frequency with which disk space is checked is related to the amount of space at the last check (in order to ensure that the disk alarm goes off in a timely manner when space is exhausted). Normally disk space is checked every 10 seconds, but as the limit is approached the frequency increases. When very near the limit RabbitMQ will check as frequently as 10 times per second. This may have some effect on system load.

When free disk space drops below the configured limit, RabbitMQ will block producers and prevent memory-based messages from being paged to disk. This will reduce the likelihood of a crash due to disk space being exhausted, but will not eliminate it entirely. In particular, if messages are being paged out rapidly it is possible to run out of disk space and crash in the time between two runs of the disk space monitor. A more conservative approach would be to set the limit to the same as the amount of memory installed on the system (see the configuration section below).

Configuring the Disk Free Space Limit

The disk free space limit is configured with the disk_free_limit setting. By default 50MB is required to be free on the database partition (see the description of file locations for the default database location). This configuration file sets the disk free space limit to 1GB:

disk_free_limit.absolute = 1000000000
Or you can use memory units (KB, MB GB etc.) like this:
disk_free_limit.absolute = 1GB
Or, using the classic config format:
[{rabbit, [{disk_free_limit, 1000000000}]}].

It is also possible to set a free space limit relative to the RAM in the machine. This configuration file sets the disk free space limit to the same as the amount of RAM on the machine:

disk_free_limit.relative = 1.0
Or, using the classic config format:
[{rabbit, [{disk_free_limit, {mem_relative, 1.0}}]}].

The limit can be changed while the broker is running using the rabbitmqctl set_disk_free_limit disk_limit command or rabbitmqctl set_disk_free_limit mem_relative fraction command. This command will take effect until the broker shuts down. The corresponding configuration setting should also be changed when the effects should survive a broker restart.