RabbitMQ nodes accept connections from clients as well as peer cluster nodes and CLI tools.
The main TLS and Troubleshooting TLS guides explain how to secure client connections with TLS. It may be desired to add a layer of encryption and an extra layer of authentication to the other two kinds of connections. This guide explains how to do that.
Switching inter-node and CLI tool communication requires configuring a few runtime flags. They provide the node with a CA certificate bundle and a certificate/key pair. CLI tools also have to be configured to use a certificate/key pair as TLS-enabled nodes won't accept unencrypted connections from CLI tools and peers.
This guide assumes the reader is familiar with the basics of TLS and peer verification (authentication) covered in the main TLS guide.
It also assumes that you already have a CA certificate bundle and a certificate/key pair generated for every cluster node and every host CLI tools will be use on. In production environments those certificates will often be produced by operators or deployment tools. For development and experimentation, there is a quick way to generate them using OpenSSL and Python.
This guide will reference three files:
Make sure you have them ready before we start.
Configuring a node to communicate over TLS-enabled connections involves a few steps. With supported Erlang versions there are two ways of doing it.
The steps are very similar on all operating systems supported but minor details will be different on Windows due to a different shell language.
Strategy one involves the following steps:
Strategy two is very similar but instead of specifying a set of runtime flags, those options can be specified in a file similar to RabbitMQ's advanced.config file and the runtime will be pointed at that file. Therefore the steps are:
We encourage operators to choose the strategy that works best for their deployment tools of choice.
With both options environment variables are used to pass those options to the runtime. This is best done using rabbitmq-env.conf as explained in the Configuration guide.
Once a node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node. Plain TCP connections will be fail.
Once the certificate/key pair files and configuration are in place the new node can be started. Note that it might be necessary to first stop the node, then deploy the files and configuration, and finally start the node. This is because CLI tools configured to use TLS won't be able to connect to a node that does not expect TLS-enabled CLI tool connections.
For nodes and CLI tools to perform TLS handshake and peer verification successfully, the same peer verification example, certificate/key pairs used by other nodes and CLI tools must be signed by the same certificate authority as the initial node or a different CA that is trusted on all cluster nodes.
This is no different from how peer verification works for client and plugin TLS connections.
It is possible to reuse a single certificate/key pair for all nodes and CLI tools. The certificate can also use a wildcard Subject Alternative Name (SAN) or Common Name (CN) such as *.rabbitmq.example.local that would match every hostname in the cluster.
The first strategy covered in this guide requires node's public and private keys to be combined into a single file. Let's call it a combined keys file. To combined them, simply concatenate the private key file, server_key.pem in the example below, to the end of the public key file, server_certificate.pem, starting with a new line:
cat server_certificate.pem server_key.pem > combined_keys.pem
This can be done using a text editor and not just command line tools such as cat.
Assuming a combined keys file from the section above is ready, next we infer the Erlang TLS library path and export ERL_SSL_PATH in rabbitmq-env.conf to point at it:
# These commands ensure that `ERL_SSL_PATH` is the first line in # /etc/rabbitmq/rabbitmq-env.conf and will preserve the existing # contents of that file if it already exists erl -noinput -eval 'io:format("ERL_SSL_PATH=~s~n", [filename:dirname(code:which(inet_tls_dist))])' -s init stop > /tmp/ssl-path.txt cat /tmp/ssl-path.txt /etc/rabbitmq/rabbitmq-env.conf > /tmp/new-rabbitmq-env.conf mv -f /tmp/new-rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf
This makes it possible for the node to load a module, inet_tls_dist, which is used for encrypted inter-node communication, from the path.
Step number two is telling the runtime to use that module using the -proto_dist inet_tls runtime flag. As with other runtime flags, SERVER_ADDITIONAL_ERL_ARGS is the most convenient and compatible to pass them.
Please note that the double quotes must be used here because the environment variable value is multi-line:
# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path # -proto_dist inet_tls tells the runtime to encrypt inter-node communication # -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime # where to find the combined certificate/key file # -ssl_dist_opt server_password password required if the private key is encrypted # SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile /path/to/combined_keys.pem \ -ssl_dist_opt server_password password
Next step is to build on the previous example and enable secure renegotiation for inter-node TLS connections. While this is optional, it is highly recommended. The same -ssl_dist_opt can be used to enable more TLS-related settings. They won't be covered in this example:
# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path # -proto_dist inet_tls tells the runtime to encrypt inter-node communication # -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime # where to find the combined certificate/key file # -ssl_dist_opt server_password password required if the private key is encrypted # -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true enables an additional TLS setting: secure renegotiation SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile /path/to/combined_keys.pem \ -ssl_dist_opt server_password password \ -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"
Once a node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node. Plain TCP connections will be fail.
This is done very similarly to what the example above does using SERVER_ADDITIONAL_ERL_ARGS but this time the environment variable is RABBITMQ_CTL_ERL_ARGS. It controls runtime flags used by CLI tools.
Here is the complete /etc/rabbitmq/rabbitmq-env.conf file:
# IMPORTANT: # the following path is system dependent (will # change depending on the Erlang version, distribution, # and installation method used). Please double check it before proceeding! ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-9.4/ebin" # -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path # -proto_dist inet_tls tells the runtime to encrypt inter-node communication # -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime # where to find the combined certificate/key file # -ssl_dist_opt server_password password required if the private key is encrypted # -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true enables an additional TLS setting: secure renegotiation SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile /path/to/combined_keys.pem \ -ssl_dist_opt server_password password \ -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true" # Same settings as above but for CLI tools RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH \ -proto_dist inet_tls \ -ssl_dist_opt server_certfile /path/to/combined_keys.pem \ -ssl_dist_opt server_password password \ -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"
Modern Erlang versions support a runtime flag, -ssl_dist_optfile, that can be used to configure TLS for inter-node communication using a single file. This simplifies the arguments passed on the command line itself.
Here is a complete /etc/rabbitmq/rabbitmq-env.conf file using this setting. Note that the name of the -ssl_dist_optfile file is not significant but it must be stored in a location readable by the effective rabbitmq user:
# NOTE: the following path is system dependent and will change between Erlang # versions ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-9.4/ebin" # -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path # -proto_dist inet_tls tells the runtime to encrypt inter-node communication # -ssl_dist_optfile tells the runtime where to find its inter-node TLS configuration file SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config" RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"
Here is an example /etc/rabbitmq/inter_node_tls.config file that uses separate server certificate and private key files, enables peer verification and requires peers to present a certificate:
[ {server, [ {cacertfile, "/full/path/to/ca_certificate.pem"}, {certfile, "/full/path/to/server_certificate.pem"}, {keyfile, "/full/path/to/server_key.pem"}, {password, "password-if-keyfile-is-encrypted"}, {secure_renegotiate, true}, {verify, verify_peer}, {fail_if_no_peer_cert, true} ]}, {client, [ {cacertfile, "/full/path/to/ca_certificate.pem"}, {certfile, "/full/path/to/client_certificate.pem"}, {keyfile, "/full/path/to/client_key.pem"}, {password, "password-if-keyfile-is-encrypted"}, {secure_renegotiate, true}, {verify, verify_peer}, {fail_if_no_peer_cert, true} ]} ].
These options are documented further in the Erlang/OTP documentation.
Both strategies covered above for Linux, macOS and BSD systems can be used on Windows. All fundamentals are the same.
There are, however, some minor differences specific to Windows. First, the command that outputs the location of the inet_tls_dist module is different due to Windows shell parsing rules. it looks like this
erl -noinput -eval "io:format(""ERL_SSL_PATH=~s~n"", [filename:dirname(code:which(inet_tls_dist))])" -s init stop
Next, the file containing the custom environment variables is named rabbitmq-env-conf.bat on Windows. This file must be saved to the %AppData%\RabbitMQ directory of the administrative user that installed RabbitMQ.
Here is a complete rabbitmq-env-conf.bat file using the -ssl_dist_opfile setting (strategy two covered above). Note the use of forward-slash directory delimiters.
@echo off rem NOTE: If spaces are present in any of these paths, rem double quotes must be used. rem NOTE: the following path is **system dependent** and will vary between Erlang versions rem and installation paths set SSL_PATH="C:/Program Files/erl10.0.1/lib/ssl-9.0/ebin" rem -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path rem -proto_dist inet_tls tells the runtime to encrypt inter-node communication rem -ssl_dist_optfile tells the runtime where to find its inter-node TLS configuration file set SERVER_ADDITIONAL_ERL_ARGS=-pa %SSL_PATH% ^ -proto_dist inet_tls ^ -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/inter_node_tls.config rem Same as above but for CLI tools set CTL_ERL_ARGS=-pa %SSL_PATH% ^ -proto_dist inet_tls ^ -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/inter_node_tls.config
Below is an example inter_node_tls.config file. As with other operating systems, more TLS options are available to be set if necessary.
[ {server, [ {cacertfile, "C:/Path/To/ca_certificate.pem"}, {certfile, "C:/Path/To/server_certificate.pem"}, {keyfile, "C:/Path/To/server_key.pem"}, {password, "password-if-keyfile-is-encrypted"}, {secure_renegotiate, true}, {verify, verify_peer}, {fail_if_no_peer_cert, true} ]}, {client, [ {cacertfile, "C:/Path/To/ca_certificate.pem"}, {certfile, "C:/Path/To/client_certificate.pem"}, {keyfile, "C:/Path/To/client_key.pem"}, {password, "password-if-keyfile-is-encrypted"}, {secure_renegotiate, true}, {verify, verify_peer}, {fail_if_no_peer_cert, true} ]} ].
If you have questions about the contents of this guide or any other topic related to RabbitMQ, don't hesitate to ask them on the RabbitMQ mailing list.
If you'd like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!