Menu

Securing Cluster (Inter-node) and CLI Tool Communication with TLS (SSL)

Sometimes is desirable to make the Erlang nodes talk to each other using TLS (SSL), and thus make the whole RabbitMQ cluster communication via TLS. To achieve that we need to configure the Erlang distribution mechanism to use TLS. In this document we are going to review the steps to make this possible.

Linux

First we need to create the TLS certificate that's going to be used by the Erlang distribution mechanism. We assume you have done that already, otherwise follow the main TLS guide. Once we have our certificates ready we need to concatenate the server certificate and key into one file, for example, assuming we have the files server_certificate.pem and server_key.pem we can do the following:

cat server_certificate.pem server_key.pem > rabbit.pem

Then we have to tell Erlang where to find the ssl library during startup. We can create a variable like this:

# NOTE: these commands ensure that ERL_SSL_PATH is the first line in
# /etc/rabbitmq/rabbitmq-env.conf and will preserve the existing
# contents of that file if it already exists

erl -noinput -eval 'io:format("ERL_SSL_PATH=~s~n", [filename:dirname(code:which(inet_tls_dist))])' -s init stop > /tmp/ssl-path.txt
cat /tmp/ssl-path.txt /etc/rabbitmq/rabbitmq-env.conf > /tmp/new-rabbitmq-env.conf
mv -f /tmp/new-rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf

First we find where Erlang has the inet_tls_dist library, and then the variable ERL_SSL_PATH is set with the result from that command.

By using the previous information now is time to craft the SERVER_ADDITIONAL_ERL_ARGS environment variable so RabbitMQ is able to start Erlang using TLS for distribution. We do that by setting the proto_dist argument to inet_tls and then telling Erlang what certificate to use (in our case that's the rabbit.pem file we just created). Finally we set secure renegotiation to true. Here is the line that should be added to /etc/rabbitmq/rabbitmq-env.conf following the ERL_SSL_PATH line that the above commands added. Please note that the double quotes must be present:

SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/rabbit.pem \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

Once our initial ("seed") node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl also must use TLS to talk to the node. This means we have to do what we just did for SERVER_ADDITIONAL_ERL_ARGS but this time for the environment variable CTL_ERL_ARGS. Here is the complete /etc/rabbitmq/rabbitmq-env.conf file:

# NOTE: the following path is **system dependent** (will
#       change depending on Erlang version, distribution used
#       installation method used). Please double check it before
#       proceeding.
ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-8.2.4/ebin"

SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/rabbit.pem \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

CTL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/rabbit.pem \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

Now that we have this in place, it's just a matter of starting RabbitMQ as we usually do to get the Erlang distribution to use TLS for internode communication. Just like CLI tools, all other nodes in the cluster that want to join our initial node must have inter-node connection TLS configured using SERVER_ADDITIONAL_ERL_ARGS in their /etc/rabbitmq/rabbitmq-env.conf file.

All nodes and CLI tools must use certificate/key pairs and TLS settings that allow inter-node TCP connections perform TLS handshake and peer verification successfully. For example, certificate/key pairs used by other nodes and CLI tools must be signed by the same CA as the initial node or a different CA that is trusted on the initial node's machine. This is no different from how client and plugin TLS connections work.

It is possible to reuse a single certificate/key pair for all nodes and CLI tools if it uses a wildcard Common Name, e.g. *.rabbitmq.example.local and all host names RabbitMQ nodes plus CLI tools are started on match the wildcard pattern.

Linux (Erlang 20.2 and later)

Starting with version 20.2, Erlang supports the -ssl_dist_optfile argument that allows configuring TLS for distributed Erlang in a file. This greatly simplifies the arguments passed on the command line itself.

Here is a complete /etc/rabbitmq/rabbitmq-env.conf file using this setting. Note that the name of the -ssl_dist_optfile file is not significant, it just must be saved in a location readable by the rabbitmq user:

# NOTE: the following path is **system dependent**
ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-8.2.4/ebin"

SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/ssl_dist.config"

CTL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/ssl_dist.config"

Here is an example /etc/rabbitmq/ssl_dist.config file:

[
  {server, [
    {cacertfile, "/full/path/to/ca_certificate.pem"},
    {certfile, "/full/path/to/server_certificate.pem"},
    {keyfile,  "/full/path/to/server_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]},
  {client, [
    {cacertfile, "/full/path/to/ca_certificate.pem"},
    {certfile, "/full/path/to/client_certificate.pem"},
    {keyfile, "/full/path/to/client_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]}
].

The file contains many of the most common options enabled to fully validate certificates. These options are documented further in the Erlang/OTP documentation: Using TLS for Erlang Distribution as well as in the ssl library documentation.

MacOS

With the MacOS standalone distribution it is necessary to add some extra arguments in order to run the erl command to find the path of Erlang's TLS library. Assuming you are inside the folder where you installed the standalone release, the commands will look like these:

erl -boot releases/3.4.3/start_clean \
-eval 'io:format("ERL_SSL_PATH=~s~n", [filename:dirname(code:which(inet_tls_dist))])' -s init stop
"/path/to/erl/lib/ssl-5.3.5/ebin"
export ERL_SSL_PATH=/path/to/erl/lib/ssl-5.3.5/ebin

The difference is that the path to the erl executable and boot file has to be explicitly specified. In RabbitMQ's case the boot file will be inside the releases folder of our standalone installation.

After executing the previous commands, proceed to create the environment variables as explained above on the Linux section

Windows

There are some minor differences when configuring TLS for distributed Erlang on Windows. First, the command to find the location of the inet_tls_dist module is different due to shell parsing rules:

erl -noinput -eval "io:format(""ERL_SSL_PATH=~s~n"", [filename:dirname(code:which(inet_tls_dist))])" -s init stop

Next, the file containing the custom environment variables is named rabbitmq-env-conf.bat on Windows. This file must be saved to the %AppData%\RabbitMQ directory of the administrative user that installed RabbitMQ.

Here is a complete rabbitmq-env-conf.bat file using the -ssl_dist_opfile setting. Note the use of forward-slash directory delimiters.

@echo off
rem NOTE: If spaces are present in any of these paths,
rem double quotes must be used.

rem NOTE: the following path is **system dependent**.
set SSL_PATH="C:/Program Files/erl10.0.1/lib/ssl-9.0/ebin"

rem NOTE: pre-RabbitMQ 3.7.8 variable names:
set RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-pa %SSL_PATH% ^
    -proto_dist inet_tls ^
    -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/ssl_dist.config

set RABBITMQ_CTL_ERL_ARGS=-pa %SSL_PATH% ^
    -proto_dist inet_tls ^
    -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/ssl_dist.config

rem NOTE: post-RabbitMQ 3.7.8 variable names:
rem set SERVER_ADDITIONAL_ERL_ARGS=...
rem set CTL_ERL_ARGS=...

rem See this PR for details
rem https://github.com/rabbitmq/rabbitmq-server/pull/1666

Below is an example ssl_dist.config file. Note that, as with UNIX-like systems, more TLS options are available to be set if necessary.

[
    {server, [
        {cacertfile, "C:/Path/To/ca_certificate.pem"},
        {certfile, "C:/Path/To/server_certificate.pem"},
        {keyfile, "C:/Path/To/server_key.pem"},
        {secure_renegotiate, true},
        {verify, verify_peer},
        {fail_if_no_peer_cert, true}
    ]},
    {client, [
        {cacertfile, "C:/Path/To/ca_certificate.pem"},
        {certfile, "C:/Path/To/client_certificate.pem"},
        {keyfile, "C:/Path/To/client_key.pem"},
        {secure_renegotiate, true},
        {verify, verify_peer},
        {fail_if_no_peer_cert, true}
    ]}
].

Getting Help and Providing Feedback

If you have questions about the contents of this guide or any other topic related to RabbitMQ, don't hesitate to ask them on the RabbitMQ mailing list.

Help Us Improve the Docs <3

If you'd like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!