Menu

Shovel Plugin

Sometimes it is necessary to reliably and continually move messages from a source (e.g. a queue) in one broker to a destination in another broker (e.g. an exchange). The Shovel plugin allows you to configure a number of shovels, which do just that and start automatically when the broker starts.

The source and destination can be on the same broker (typically in different vhosts) or distinct brokers. Since 3.7 they can even use different protocols (AMQP 0.9.1 or AMQP 1.0).

A shovel behaves like a well-written client application, which connects to its source and destination, reads and writes messages, and copes with connection failures. In fact, Shovels use the Erlang AMQP 0-9-1 and Erlang AMQP 1.0 clients under the hood.

The primary advantages of a shovel are:

Loose coupling
A shovel can move messages between brokers (or clusters) in different administrative domains:
  • they may have different users and virtual hosts;
  • they may run on different versions of RabbitMQ and Erlang.
  • they may be different broker products.
WAN-friendly
The Shovel plugin uses client connections to communicate between brokers, and is designed to tolerate intermittent connectivity without message loss.
Highly tailorable
When a shovel connects (either to the source or the destination) it can be configured to perform any number of explicit methods. For example, the source queue need not exist initially, and can be declared on connect (AMQP 0.9.1 only).
Supports multiple protocols
Starting with RabbitMQ 3.7.0 the shovel has support for multiple protocols. Currently implemented are AMQP 0.9.1 and AMQP 1.0. This means it is possible to shovel, e.g. from and AMQP 1.0 broker source to a RabbitMQ destination and vice versa. Further protocols may be implemented in the future.

A comparison between clustering, federated exchanges and shovels is given on the Distributed Messaging page.

What does it do?

The Shovel plugin defines (and runs) an Erlang client application for each shovel.

In essence, a shovel is a simple pump. Each shovel:

  • connects to the source broker and the destination broker,
  • consumes messages from the queue,
  • re-publishes each message to the destination broker (using, by default, the original exchange name and routing_key when applicable).

The shovel configuration allows each of these processes to be tailored.

connects

After connection to a source or a destination broker a series of configured declarations can be issued. For example, on an AMQP 0-9-1 endpoint, queues, exchanges and bindings can be declared.

A shovel will attempt to reconnect to a broker if a failure occurs and multiple brokers can be specified for the source and destination so that another broker may be selected (at random) to reconnect to. A reconnection delay can be specified to avoid flooding the network with reconnection attempts, or to prevent reconnection on failure altogether.

All configured declarations (for that source or destination) are re-issued upon re-connect.

consumes

The Shovel's consumer can acknowledge messages automatically on receipt, after (re-)publication, or after confirmation of its publication from the destination server.

re-publishes

Both the publish method and the message properties can be modified with explicit parameter values.

Getting started

The Shovel plugin is included in the RabbitMQ distribution. To enable it, use rabbitmq-plugins:

rabbitmq-plugins enable rabbitmq_shovel

You may also wish to enable the rabbitmq_shovel_management plugin (see below).

There is no requirement to run the shovel on the same broker (or cluster) as its source or destination; the shovel can be entirely separate.

There are two distinct ways to define shovels: static shovels and dynamic shovels. The differences are as follows:

Static Shovels Dynamic Shovels
Defined in the broker configuration file. Defined using the broker's runtime parameters.
Require a restart of the hosting broker to change. Can be created and deleted at any time.
Slightly more general: any queues, exchanges or bindings can be declared manually at startup. Slightly more opinionated: the queues, exchanges and bindings used by the shovel will be declared automatically.

Note that when using AMQP 1.0 the "nodes" may still need to be created outside of the shovel as the protocol does not include topology creation.

Configuration Format Differences

Versions prior to 3.7.0 use a slightly different configuration format. See: static shovels (legacy) and dynamic shovels (legacy) for details. Static shovel created by the old format will still work in 3.7.0+ but it is recommended to upgrade at earliest convenience.

Authentication and Authorization for Shovels

The plugin uses Erlang AMQP 0-9-1 and Erlang AMQP 1.0 clients under the hood to open connections to its source and/or destination. Just like any other client library connection, a Shovel connection must successfully authenticate with RabbitMQ server and be authorized to access the vhost it is trying to connect to.

Authentication and authorization failures of Shovel connections will be visible in the logs of the node that opens them.

Shovels Between Clusters

It's normally desirable to ensure that shovels are resilient to failure of any node in the source or destination clusters, or the cluster hosting the shovel.

You can ensure that shovels can fail over to different nodes in the source or destination clusters by specifying multiple source and/or destination URIs for each shovel.

Dynamic shovels are automatically defined on all nodes of the hosting cluster on which the shovel plugin is enabled. Each shovel will only start on one node (arbitrarily), but will fail over to another node if that one goes down.

Static shovels should be defined in the configuration file for all nodes of the hosting cluster on which the shovel plugin is enabled. Again each shovel will only start on one node and fail over when needed.

Securing Shovel Connections with TLS

Shovel connections can be secured with TLS. Because Shovel uses client libraries under the hood, it is necessary to both configure the source broker to listen for TLS connections and the Shovel/Erlang client to use TLS.

To configure Shovel to use TLS, one needs to

Just like with "regular" client connections, server's CA should be trusted on the node where Shovel runs, and vice versa.

Monitoring shovels

There are two ways of discovering the status of shovels.

Use shovel management

Shovel status can be reported on the Management plugin user interface by enabling the rabbitmq_shovel_management plugin wherever you have the management plugin enabled.

Information about configured shovels will automatically appear in the management API and UI.

CLI tools

Shovel status can be obtained by direct query of the Shovel plugin app. Issue the following rabbitmqctl command:

rabbitmqctl eval 'rabbit_shovel_status:status().'

This calls the status method in a module of the rabbitmq_shovel plugin, which will return an Erlang list, with one element for each configured shovel.

Each element of the list is a map with several fields fields: {Name, Type, Status, Timestamp}

  • Name is the shovel name,
  • Type is either static or dynamic,
  • Status is the current shovel state,
  • and Timestamp is the time when the shovel entered this state.
  • Timestamp is a local calendar time of the form {{YYYY, MM, DD}, {HH, MM, SS}}.

    Status takes one of three forms:

    • The shovel is starting up, connecting and creating resources:
      'starting'
      
    • The shovel is running normally:
      { 'running', [{'src_protocol', Protocol},
                    {'src_uri', Source},
                    {'dest_protocol', Protocol},
                    {'dest_uri', Destination}]}:
      

      where Source and Destination terms give the respective connection parameters and Protocol is either amqp091 or amqp10

    • The shovel has terminated:
      {'terminated', Reason}
      

      where Reason is an Erlang term that indicates the reason for the termination.