Menu

Logging

Introduction

Logging is a critically important aspect of system observability, together with monitoring. Both developers and operators are advised to consult logs when troubleshooting an issue or assessing the state of the system.

RabbitMQ supports a number of features when it comes to logging.

This guide covers topics such as:

and more.

Log File Location

Prior to 3.7.0 there were two log files: for regular messages and unhandled exceptions. As of 3.7.0 a single log file is used for all messages by default.

Default log file location is covered in the File and Directory Location guide.

You can modify the default location either by using a configuration file, or by setting the RABBITMQ_LOGS environment variable. Effective log file path can be discovered using RabbitMQ management UI or CLI commands such as rabbitmqctl environment.

RABBITMQ_LOGS variable value can be either a file path or a hyphen (-), which means all log messages should be printed to standard output.

The environment variable takes precedence over the configuration file. When in doubt, consider overriding log file location via the config file.

Configuration

As of 3.7.0 RabbitMQ uses the Lager logging library under the hood. The library supports logging to different sources and provides a fair amount of flexibility when it comes to configuration.

RabbitMQ initializes its logging subsystem on node start. See the Configuration guide for a general overview of how RabbitMQ nodes are configured.

Log Outputs

Default RabbitMQ logging configuration will direct log messages to a log file. Standard output is another option available out of the box.

Multiple outputs can be used at the same time. Log entries will be copied to all of them.

Different outputs can have different log levels, for example, the console output can be configured to log all messages including debug information while the file output will log only error and higher severity messages.

Logging to a File

  • log.file: log file path or false to disable the file output. Default value is taken from the RABBITMQ_LOGS environment variable or configuration file.
  • log.file.level: log level for the file output. Default level is info.
  • log.file.rotation.date, log.file.rotation.size, log.file.rotation.count for log file rotation settings.

The following example overiddes log file name:

log.file = rabbit.log

The following example overiddes log file directory:

log.dir = /data/logs/rabbitmq

The following example instructs RabbitMQ to log to a file at the debug level:

log.file.level = debug

Logging to a file can be disabled with

log.file = false

The rest of this guide, rabbitmq.conf.example, and Lager configuration reference cover the list of acceptable log levels and other values.

Classic Config Format

It is possible to configure file logging using the classic configuration format:

[{rabbit, [
        {log, [
            {file, [{file, "/path/to/log/file.log"}, %% log.file
                    {level, info},        %% log.file.info
                    {date, ""},           %% log.file.rotation.date
                    {size, 0},            %% log.file.rotation.size
                    {count, 1}            %% log.file.rotation.count
                    ]}
        ]}
    ]}].

Log Rotation

The broker always appends to the log files, so a complete log history is retained. Log file rotation via Lager is disabled by default. Debian and RPM packages will set up log rotation via logrotate after package installation.

log.file.rotation.date, log.file.rotation.size, log.file.rotation.count settings control log file rotation for the file output.

Periodic Rotation

log.file.rotation.date is used to set up periodic (date and time-based) rotation. It uses the same syntax as newsyslog.conf:

# rotate every night at midnight
log.file.rotation.date = $D0

# keep up to 5 archived log files in addition to the current one
log.file.rotation.count = 5
# rotate every day at 23:00 (11:00 p.m.)
log.file.rotation.date = $D23
# rotate every hour at HH:00
log.file.rotation.date = $H00
# rotate every day at 12:30 (00:30 p.m.)
log.file.rotation.date = $D12H30
# rotate every week on Sunday at 00:00
log.file.rotation.date = $W0D0H0
# rotate every week on Friday at 16:00 (4:00 p.m.)
log.file.rotation.date = $W5D16
# rotate every night at midnight
log.file.rotation.date = $D0

File Size-based Rotation

log.file.rotation.size controls rotation based on the current log file size:

# rotate when the file reaches 10 MiB
log.file.rotation.size = 10485760

# keep up to 5 archived log files in addition to the current one
log.file.rotation.count = 5

Rotation Using Logrotate

On Linux, BSD and other UNIX-like systems, logrotate is an alternative way of log file rotation and compression.

RabbitMQ Debian and RPM packages will set up logrotate to run weekly on files located in default /var/log/rabbitmq directory. Rotation configuration can be found in /etc/logrotate.d/rabbitmq-server.

Logging to Console (Standard Output)

Here are the main settings that control console (standard output) logging:

  • log.console (boolean): set to true to enable console output. Default is false
  • log.console.level: log level for the console output. Default level is info.

To enable console logging, use the following config snippet:

log.console = true

The following example disables console logging

log.console = false

The following example instructs RabbitMQ to use the debug logging level when logging to console:

log.console.level = debug

It is possible to configure console logging using the classic config format:

[{rabbit, [
        {log, [
            {console, [{enabled, true}, %% log.console
                       {level, info}    %% log.console.level
            ]}
        ]}
    ]}].

If you enable console output, the file output will still be enabled by default. To disable the file output, set log.file to false.

Please note that RABBITMQ_LOGS set to - will disable the file output even in log.file is configured.

Logging to Syslog

RabbitMQ logs can be forwarded to a Syslog server via TCP or UDP. UDP is used by default and requires Syslog service configuration. TLS is also supported.

Syslog output has to be explicitly configured:

log.syslog = true

Or, in the classic config format:

[{rabbit, [{log, [
    {syslog, [{enabled, true}]}]}]
}].

Syslog Endpoint Configuration

By default the Syslog logger will send log messages to UDP port 514 using the RFC 3164 protocol. RFC 5424 protocol also can be used.

In order to use UDP the Syslog service must have UDP input configured.

UDP and TCP transports can be used with both RFC 3164 and RFC 5424 protocols. TLS support requires the RFC 5424 protocol.

The following example uses TCP and the RFC 5424 protocol:

log.syslog = true
log.syslog.transport = tcp
log.syslog.protocol = rfc5424

In the classic config format:

[{rabbit, [{log, [{syslog, [{enabled, true}]}]}]},
 {syslog, [{protocol, {tcp, rfc5424}}]}
].

To TLS, a standard set of TLS options must be provided:

log.syslog = true
log.syslog.transport = tls
log.syslog.protocol = rfc5424

log.syslog.ssl_options.cacertfile = /path/to/tls/cacert.pem
log.syslog.ssl_options.certfile = /path/to/tls/cert.pem
log.syslog.ssl_options.keyfile = /path/to/tls/key.pem

In the classic config format:

[{rabbit, [{log, [{syslog, [{enabled, true}]}]}]},
 {syslog, [{protocol, {tls, rfc5424,
                        [{cacertfile,"/path/to/tls/cacert.pem"},
                         {certfile,"/path/to/tls/cert.pem"},
                         {keyfile,"/path/to/tls/key.pem"}]}}]}
].

Syslog service IP address (note: hostnames are not supported) and port can be customised:

log.syslog = true
log.syslog.ip = 10.10.10.10
log.syslog.port = 1514

In the classic config format:

[{rabbit, [{log, [{syslog, [{enabled, true}]}]}]},
 {syslog, [{dest_host, {10, 10, 10, 10}},
           {dest_port, 1514}]}
].

Syslog metadata identity and facility values also can be configured. By default identity will be set to the name part of the node name (for example rabbitmq for [email protected]) and facility will be set to daemon.

To set identity and facility of log messages:

log.syslog = true
log.syslog.identity = my_rabbitmq
log.syslog.facility = user

In the classic config format:

[{rabbit, [{log, [{syslog, [{enabled, true}]}]}]},
 {syslog, [{app_name, "my_rabbitmq"},
           {facility, user}]}
].

Less commonly used Syslog client options can be configured using the advanced config file.

Log Message Categories

RabbitMQ has several categories of messages, which can be logged with different levels or to different files.

The categories replace the rabbit.log_levels configuration setting in versions earlier than 3.7.0.

The categories are:

  • connection: connection lifecycle events for AMQP 0-9-1, AMQP 1.0, MQTT and STOMP.
  • channel: channel logs. Mostly errors and warnings on AMQP 0-9-1 channels.
  • queue: queue logs. Mostly debug messages.
  • mirroring: queue mirroring logs. Queue mirrors status changes: starting/stopping/synchronizing.
  • federation: federation plugin logs.
  • upgrade: verbose upgrade logs. These can be excessive.
  • default: all other log entries. You cannot override file location for this category.

It is possible to configure a different log level or file location for each message category using log.<category>.level and log.<category>.file configuration variables.

By default each category will not filter by level. So if you have an output configured to log debug messages, the debug messages will be printed from all categories, unless a category level is configured.

For example, given debug level in the file output, the following will disable debug logging for connection events:

log.file.level = debug
log.connection.level = info

Or, using the classic configuration format:

[{rabbit,
    [{log,
        [{file, [{level, debug}]},
         {categories,
            [{connection,
                [{level, info}]
            }]
        }]
    }]
}].

To redirect all federation logs to the rabbit_federation.log file, use:

log.federation.file = rabbit_federation.log

Using the classic configuration format:

[{rabbit,
    [{log,
        [{categories,
            [{federation,
                [{file, "rabbit_federation.log"}]
            }]
        }]
    }]
}]

To disable a log type, you can use the none log level. For example, to disable upgrade logs:

log.upgrade.level = none

Using the classic configuration format:

[{rabbit,
    [{log,
        [{categories,
            [{upgrade,
                [{level, none}]
            }]
        }]
    }]
}].

Log Levels

Log levels is another way to filter and tune logging. Each log level has a severity associated with it. More critical messages have lower severity number, while debug has the highest number.

The following log levels are used by RabbitMQ:

Log level Severity
debug 128
info 64
warning 16
error 8
critical 4
none 0

When a message is logged, if the level number is higher than the category level, the message will be dropped and not sent to outputs.

If a category level is not configured, it's messages will always be sent to outputs.

To make the default category log only errors or higher severity messages, use

log.default.level = error

The none level means that no messages will be logged.

Levels can be configured separately for each output. If a message level number is higher than the output level, the message will not be logged.

For example if no outputs are configured to log debug messages, even if you set a category level to debug, the debug messages will not be logged.

Although, if an output is configured to log debug messages, it will get them from all categories, unless a category level is configured.

Enabling Debug Logging

To enable debug messages, you should have a debug output.

For example to log debug messages to a file:

log.file.level = debug

In the classic config format:

[{rabbit, [{log, [
    {file, [{level, debug}]}]
}].

To print log messages to standard out:

log.console.enabled = true
log.console.level = debug

In the classic config format:

[{rabbit, [{log, [
    {console, [{enabled, true},
               {level, debug}]}
    ]}]
}].

To disable debug logging for some categories:

log.file.level = debug

log.connection.level = info
log.channel.level = info

In the classic config format:

[{rabbit, [{log, [
    {file, [{level, debug}]},
    {categories, [
        {connection, [{level, info}]},
        {channel, [{level, info}]}
        ]}
    ]}]
}].

Service Logs

On systemd-based Linux distributions, system service logs can be inspected using journalctl --system

journalctl --system

which requires superuser privileges. Its output can be filtered to narrow it down to RabbitMQ-specific entries:

sudo journalctl --system | grep rabbitmq

Service logs will include standard output and standard error streams of the node. The output of journalctl --system will look similar to this:

Dec 26 11:03:04 localhost rabbitmq-server[968]: ##  ##
Dec 26 11:03:04 localhost rabbitmq-server[968]: ##  ##      RabbitMQ 3.7.9. Copyright (C) 2007-2018 Pivotal Software, Inc.
Dec 26 11:03:04 localhost rabbitmq-server[968]: ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
Dec 26 11:03:04 localhost rabbitmq-server[968]: ######  ##
Dec 26 11:03:04 localhost rabbitmq-server[968]: ##########  Logs: /var/log/rabbitmq/[email protected]
Dec 26 11:03:04 localhost rabbitmq-server[968]: /var/log/rabbitmq/[email protected]_upgrade.log
Dec 26 11:03:04 localhost rabbitmq-server[968]: Starting broker...
Dec 26 11:03:05 localhost rabbitmq-server[968]: systemd unit for activation check: "rabbitmq-server.service"
Dec 26 11:03:06 localhost rabbitmq-server[968]: completed with 6 plugins.

Logged Events

Connection Lifecycle Events

Successful TCP connections that send at least 1 byte of data will be logged. Connections that do not send any data, such as health checks of certain load balancer products, will not be logged.

Here's an example:

2018-11-22 10:44:33.654 [info] <0.620.0> accepting AMQP connection <0.620.0> (127.0.0.1:52771 -> 127.0.0.1:5672)

The entry includes client IP address and port (127.0.0.1:52771) as well as the target IP address and port of the server (127.0.0.1:5672). This information can be useful when troubleshooting client connections.

Once a connection successfully authenticates and is granted access to a virtual host, that is also logged:

2018-11-22 10:44:33.663 [info] <0.620.0> connection <0.620.0> (127.0.0.1:52771 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

The examples above include two values that can be used as connection identifiers in various scenarios: connection name (127.0.0.1:57919 -> 127.0.0.1:5672) and an Erlang process ID of the connection (<0.620.0>). The latter is used by rabbitmqctl and the former is used by the HTTP API.

A connection can be closed cleanly or abnormally. In the former case the client closes AMQP 0-9-1 (or 1.0, or STOMP, or MQTT) connection gracefully using a dedicated library function (method). In the latter case the client closes TCP connection or TCP connection is lost. Both cases will be logged by the broker. щ Below is an example entry for a successfully closed connection:

2018-06-17 06:23:29.855 [info] <0.634.0> closing AMQP connection <0.634.0> (127.0.0.1:58588 -> 127.0.0.1:5672, vhost: '/', user: 'guest')

Prior to RabbitMQ 3.7 the format was different:

=INFO REPORT==== 30-Oct-2017::21:40:32 ===
closing AMQP connection <0.24990.164> (127.0.0.1:57919 -> 127.0.0.1:5672, vhost: '/', user: 'guest')

Abruptly closed connections will be logged as warnings:

2018-06-17 06:28:40.868 [warning] <0.646.0> closing AMQP connection <0.646.0> (127.0.0.1:58667 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection

In the pre-3.7 format:

=WARNING REPORT==== 1-Nov-2017::16:58:58 ===
closing AMQP connection <0.601.0> (127.0.0.1:60471 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection

Abruptly closed connections could be harmless (e.g. a short lived program has naturally terminated and didn't have a chance to close its connection properly) or indicate a genuine issue such as a failed application process or a proxy that eagerly closes TCP connections it considers to be idle.

Upgrading From pre-3.7 Versions

RabbitMQ versions prior to 3.7.0 had a different logging subsystem.

Older installations use two log files: <nodename>.log and <nodename>_sasl.log (<nodename> is [email protected]{hostname} by default).

Where <nodename>.log contains RabbitMQ logs, while <nodename>_sasl.log contains runtime logs, mostly unhandled exceptions.

Starting with 3.7.0 these two files were merged and all errors now can be found in the <nodename>.log file. So RABBITMQ_SASL_LOGS environment variable is not used anymore.

Log levels in pre-3.7.0 versions were configured using the log_levels configuration key. Starting with 3.7 it's been replaced with categories, which are more descriptive and powerful.

If the log_levels key is present in rabbitmq.config file, it should be updated to use categories.

rabbit.log_levels will work in 3.7.0 only if no categories are defined.

Advanced Configuration

This section describes the nitty gritty details of the logging subsystem. Most RabbitMQ installations won't require deep knowledge of this topic or any of the advanced configuration explained here.

Lager Handlers and Sinks

RabbitMQ logging subsystem is built on top of Lager, a powerful logging library with several advanced features. Some of them are accessible via the log handler and sink abstractions.

A sink is an "endpoint" where log entries are written by connections, queues and so on. A handler is stateful entity that consumes log entries and processes them, e.g. writes them to a file, sends them to a log collection endpoint or discards them.

By default RabbitMQ creates one file backend handler and one sink per log category (see above). Changing RabbitMQ log configuration parameters changes log handler used under the hood. The number of sinks used by RabbitMQ is largely fixed, although 3rd party plugins can use custom sinks and with a certain amount of configuration it may be possible e.g. to log those messages separately from the rest.

When RabbitMQ is started with default logging settings, a Lager handler is configured under the hood and it looks like this:

[{lager, [
    {handlers,
       [{lager_file_backend,
            [{file,
                 "/var/log/rabbitmq/log/rabbit.log"},
             {formatter_config,
                 [date," ",time," ",color,"[",severity,"] ",
                  {pid,[]},
                  " ",message,"\n"]},
             {level,info},
             {date,""},
             {size,0}]}]},
    {extra_sinks,
       [{error_logger_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_channel_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_connection_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_mirroring_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_queue_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_federation_lager_event,
            [{handlers, [{lager_forwarder_backend,[lager_event,inherit]}]}]},
        {rabbit_log_upgrade_lager_event,
            [{handlers,
                [{lager_file_backend,
                    [{date,[]},
                     {file,
                           "/var/log/rabbitmq/rabbit_upgrade.log"},
                     {formatter_config,
                        [date," ",time," ",color,"[",severity,
                         "] ",
                         {pid,[]},
                         " ",message,"\n"]},
                     {level,info},
                     {size,0}]}]}]}]}
]}].

Most sinks use lager_forwarder_backend. This backend will redirect all messages with matching level to default lager sink (lager_event). Upgrade messages use a separate sink with its own log file.

For instance, if console logging is enabled with

log.console = true
log.console.level = warning

then generated handlers configuration will look something like this:

[{lager,
    [{handlers,
        [{lager_console_backend,
            [{formatter_config,[date," ", time," ",color,"[",severity,"] ", {pid,[]}, " ",message,"\n"]},
             {level,warning}]},
         {lager_file_backend,
            [{date,[]},
            {file,"/var/folders/cl/jnydxpf92rg76z05m12hlly80000gq/T/rabbitmq-test-instances/rabbit/log/rabbit.log"},
            {formatter_config,[date," ",time," ",color,"[",severity, "] ", {pid,[]}, " ",message,"\n"]},
            {level,info},
            {size,0}]
        }]},
     {extra_sinks,
        [{error_logger_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event, inherit]}]}]},
         {rabbit_log_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                       inherit]}]}]},
         {rabbit_log_channel_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                               inherit]}]}]},
         {rabbit_log_connection_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                                  inherit]}]}]},
         {rabbit_log_mirroring_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                                 inherit]}]}]},
         {rabbit_log_queue_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                             inherit]}]}]},
         {rabbit_log_federation_lager_event,[{handlers,[{lager_forwarder_backend,[lager_event,
                                                                                  inherit]}]}]},
         {rabbit_log_upgrade_lager_event,
            [{handlers,
                [{lager_console_backend,
                    [{formatter_config,[date," ", time," ",color,"[",severity,"] ", {pid,[]}, " ",message,"\n"]},
                     {level,warning}]},
                 {lager_file_backend,
                    [{date,[]},
                     {file,"/var/folders/cl/jnydxpf92rg76z05m12hlly80000gq/T/rabbitmq-test-instances/rabbit/log/rabbit_upgrade.log"},
                     {formatter_config,[date," ", time," ",color,"[",severity,"] ", {pid,[]}, " ",message,"\n"]},
                     {level,info},
                     {size,0}]}]}]
         }]
    }]
}].

In the above example, a new lager_console_backend handler is added to the handlers and upgrade_lager_event sink handlers. Because upgrade category defines a separate file by default, all default handlers are copied to the sink handlers and the file setting is modified.

If a target log file is configured for a category via a log.<category>.file config entry, all log messages in that category will be written to this file only as well as non-file backends.

If having upgrade logs in the default log file is desired, or log files are configured in handlers, category-specific files should be disabled. This is done with

log.upgrade.file = false

or

[{rabbit, [{log, [{categories, [{upgrade, [{file, false}]}]}]}]}].

in the classic configuration format.

You can add any additional handlers to default lager configuration or to sinks by setting handlers to extra_sinks in the lager application config.

Handlers configured in the lager config will be merged with those generated by RabbitMQ. So messages will be logged by your custom handlers and by the generated ones.

If you set a sink with the same name as RabbitMQ category sinks, it's handlers will be merged with the generated category sink. So messages logged in the category will be logged in the configured handlers and redirected to default sink.

You can disable RabbitMQ handlers or sinks using RabbitMQ configuration. For example by setting level to none for handlers and categories.

Custom Handler Examples

To create an additional log file for errors only, create an additional handler with the error level. This has to be done using the advanced config file:

[{lager, [
    {handlers, [
        {lager_file_backend,
            [{file, "rabbit_errors.log"},
             {level,error},
             %% The formatter and rotation config is optional
             {formatter_config,
                 [date," ",time," ",color,"[",severity,"] ",
                  {pid,[]},
                  " ",message,"\n"]},
             {date,""},
             {size,0}]}
    ]}]
}].

To use a custom lager backend and disable RabbitMQ default handlers:

[{lager,
    [{handlers,
        [{lager_custom_backend,
            [{level,info},
             {custom,settings}]}]
    }]},
 {rabbit,
    [{log,
        [{file, [{file, false}]}] %% Disable RabbitMQ file handler
    }]}
].

To log direct Erlang AMQP 0-9-1 client messages to console instead of default output:

[{lager,
    [{extra_sinks,
        [{rabbit_log_connection_lager_event,
            [{handlers,
                [{lager_console_backend, info}]}]}]
    }]},
 {rabbit,
    [{log,
        [{categories,
            [{connection, [{level, none}]}]}] %% Block connection category forwarder
    }]}
].

Getting Help and Providing Feedback

If you have questions about the contents of this guide or any other topic related to RabbitMQ, don't hesitate to ask them on the RabbitMQ mailing list.

Help Us Improve the Docs <3

If you'd like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!