Menu

Configuring Dynamic Shovels

Information about dynamic shovels is stored in the RabbitMQ database, along with users, permissions, queues, etc. Every shovel is defined by a corresponding named runtime parameter.

Parameters can be defined using rabbitmqctl, through the management HTTP API, or (with the rabbitmq_shovel_management plugin installed) through the management plugin's web UI.

A simple example

Here we will consume messages from the queue my-queue on the local broker, and push them out to the queue another-queue on a remote broker.

rabbitmqctl rabbitmqctl set_parameter shovel my-shovel \
'{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "my-queue", \
"dest-protocol": "amqp091", "dest-uri": "amqp://remote-server", \
"dest-queue": "another-queue"}'
rabbitmqctl (Windows) rabbitmqctl set_parameter shovel my-shovel ^
"{""src-protocol"": ""amqp091"", ""src-uri"":""amqp://"", ""src-queue"": ""my-queue"", ^
""dest-protocol"": ""amqp091"", ""dest-uri"": ""amqp://remote-server"", ^
""dest-queue"": ""another-queue""}"
HTTP API PUT /api/parameters/shovel/%2f/my-shovel
{"value":{"src-protocol": "amqp091", "src-uri":  "amqp://",
          "src-queue":  "my-queue",
          "dest-protocol": "amqp091", "dest-uri": "amqp://remote-server",
          "dest-queue": "another-queue"}}
Web UI Navigate to Admin > Shovel Management > Add a new shovel. Enter "my-shovel" next to Name, "amqp://" and "my-queue" next to Source, and "amqp://remote-server" and "another-queue" next Destination Expiry. Click Add shovel.

Configuration reference

Shovel

reconnect-delay
The duration (in seconds) to wait before reconnecting to the brokers after being disconnected at either end. Default is 1.
ack-mode

Determines how the shovel should acknowledge messages. If set to on-confirm (the default), messages are acknowledged to the source broker after they have been confirmed by the destination. This handles network errors and broker failures without losing messages, and is the slowest option.

If set to on-publish, messages are acknowledged to the source broker after they have been published at the destination. This handles network errors without losing messages, but may lose messages in the event of broker failures.

If set to no-ack, message acknowledgements are not used. This is the fastest option, but may lose messages in the event of network or broker failures.

src-protocol

The protocol to use for the source. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below.

dest-protocol

The protocol to use for the destination. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below.

AMQP 0.9.1 Source

src-uri

The AMQP URI(s) for the source. Mandatory. See the AMQP URI reference for information on how RabbitMQ treats AMQP URIs in general, and the query parameter reference for the Erlang client's extensions (including those for SSL and SASL) which are available to the shovel.

Note that this field can either be a string, or a list of strings. If more than one string is provided, the shovel will randomly pick one URI from the list. This can be used to connect to a cluster and ensure the link will eventually find another node in the event that one fails. It's probably not a great idea to use multiple URIs that do not point to the same cluster.

src-prefetch-count
The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000.
src-queue

The queue from which to consume. Either this or src-exchange (but not both) must be set.

If the source queue does not exist on the source broker, it will be declared as a durable queue with no arguments.

src-exchange

The exchange from which to consume. Either this or src-queue (but not both) must be set.

The shovel will declare an exclusive queue and bind it to the named exchange with src-exchange-key before consuming from the queue.

If the source exchange does not exist on the source broker, it will be not declared; the shovel will fail to start.

src-exchange-key

Routing key when using src-exchange

src-delete-after

Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis.

The default is never, meaning the shovel should never delete itself.

If set to queue-length then the shovel will measure the length of the source queue when starting up, and delete itself after it has transfered that many messages.

If set to an integer, then the shovel will transfer that number of messages before deleting itself.

AMQP 0.9.1 Destination

dest-uri

The AMQP URI(s) for the destination. Mandatory. See src-uri above.

dest-queue

The queue to which messages should be published. Either this or dest-exchange (but not both) may be set. If neither is set then messages are republished with their original exchange and routing key.

If the destination queue does not exist on the source broker, it will be declared as a durable queue with no arguments.

dest-exchange

The exchange to which messages should be published. Either this or dest-queue (but not both) may be set.

If the destination exchange does not exist on the source broker, it will be not declared; the shovel will fail to start.

dest-exchange-key

Routing key when using dest-exchange. If this is not set, the original message's routing key will be used.

dest-publish-properties
Properties to overwrite when shovelling messages. Setting headers this way is not currently supported. Default is {}.
dest-add-forward-headers
Whether to add x-shovelled headers to the shovelled messages indicating where they have been shovelled from and to. Default is false.
dest-add-timestamp-header
Whether to add x-shovelled-timestamp headers to the shovelled messages containing timestamp (in seconds since epoch) when message had been shovelled. Default is false.

AMQP 1.0 Source

src-uri

The AMQP URI for the source. Mandatory. AMQP 1.0 URIs implement as subset of what is described in the AMQP URI reference. Direct connections are not supported for AMQP 1.0 endpoints and there is no vhost concept. The set of query parameters it supports are different from AMQP 0.9.1 URI(s):

idle_time_out
heartbeat interval
hostname
The name of the target host. Certain vendors (such as Azure ServiceBus) require this to be set even if it is the same as the host segment in the uri.
sasl
anon, none or plain Defaults to: none. When using plain the user and password segments of the URI need to be set.
cacertfile, certfile, keyfile
Paths to files to use in order to present a client-side SSL certificate to the server. Only of use for the amqps scheme.
verify, fail_if_no_peer_cert
Use to configure verification of the server's SSL certificate. See the SSL guide for details of SSL in RabbitMQ in general. Only of use for the amqps scheme.
src-address

The AMQP 1.0 link address. Mandatory.

src-prefetch-count
The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000.
src-delete-after

Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis.

The default is never, meaning the shovel should never delete itself.

If set to an integer, then the shovel will transfer that number of messages before deleting itself.

AMQP 1.0 Destination

dest-uri

The AMQP URI for the destination. Mandatory. See AMQP 1.0 src-uri above. Mandatory.

dest-address

The AMQP 1.0 link address. Mandatory.

dest-properties
Properties to overwrite when shovelling messages. See AMQP 1.0 spec §3.2.4 for details of all possible properties.
dest-application-properties
Application properties to set when shovelling messages.
dest-add-forward-headers
Whether to add x-shovelled application properties to the shovelled messages indicating where they have been shovelled from and to. Default is false.
dest-add-timestamp-header
Whether to set the creation_time header to the ontaining timestamp (in milliseconds since epoch) when message had been shovelled. Default is false.