This guide focuses on dynamically configured shovels. It assumes familiarity with the key concepts behind the Shovel plugin.
Unlike with static shovels, dynamic shovels are configured using runtime parameters. They can be started and stopped at any time, including programmatically. Dynamic shovels can be used for both transient (one-off) and permanently running workloads.
Information about dynamic shovels is stored in RabbitMQ's schema database, along with users, permissions, queues, etc. They therefore can be exported together with other schema definitions.
Parameters can be defined using rabbitmqctl, through the management HTTP API, or (with the rabbitmq_shovel_management plugin enabled) through the management UI's administrative section.
A shovel is declared with a definition body, which is a JSON object. Some keys are mandatory, others are optional. They control connection parameters, protocol used, message transfer source and destination, data safety protocol features, and more.
Every shovel belongs to a virtual host. Note that a Shovel can consume from and publish to not only a different virtual host but an entirely different cluster, so virtual host selection primarily acts as a way of organising shovels and access to them, much like with the rest of permission in RabbitMQ.
Every shovel is also named. The name is used to identify shovels when inspecting their status, deleting them or restarting them.
In this example we will set up a dynamic shovel that will move messages from the queue "source-queue" in the local RabbitMQ cluster to the queue "target-queue" on a remote RabbitMQ node, using AMQP 0-9-1.
A shovel is declared using the rabbitmqctl set_parameter command with component name shovel, a shovel name and a definition body which is a JSON document:
# my-shovel here is the name of the shovel rabbitmqctl set_parameter shovel my-shovel \ '{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "source-queue", "dest-protocol": "amqp091", "dest-uri": "amqp://remote-server", "dest-queue": "target-queue"}'
On Windows rabbitmqctl is named rabbitmqctl.bat and command line value escaping will be different:
rabbitmqctl.bat set_parameter shovel my-shovel ^ "{""src-protocol"": ""amqp091"", ""src-uri"":""amqp://localhost"", ""src-queue"": ""source-queue"", ^ ""dest-protocol"": ""amqp091"", ""dest-uri"": ""amqp://remote.rabbitmq.local"", ^ ""dest-queue"": ""target-queue""}"
The body in this example includes a few keys:
Key | Description |
src-uri |
Source connection URI. Mandatory. See the AMQP URI reference for information on how RabbitMQ treats AMQP URIs in general, and the query parameter reference for additional query parameters that are recognised by the Shovel and Federation plugins (such as TLS settings). Note that this field can either be a string, or a list of strings. If more than one string is provided, the shovel will randomly pick one URI from the list until one of the endpoints succeeds. |
src-protocol | Protocol to use when connecting to the source. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below. |
src-queue |
Source queue that the shovel will consume from. The queue from which to consume. Either this or src-exchange (but not both) must be set. If the source queue does not exist on the target virtual host, it will be declared as a classic durable queue with no optional arguments. |
dest-uri | Same as src-uri above but for destination connection. |
dest-protocol | Protocol to use when connecting to the destination. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below. |
dest-queue |
The queue to which messages should be published. Either this or dest-exchange (but not both) may be set. If neither is set then messages are republished with their original exchange and routing key. If the destination queue does not exist in the destination virtual host, it it will be declared as a classic durable queue with no optional arguments. |
There are other Shovel definition keys that will be covered later in this guide.
To declare a shovel using the HTTP API, make sure that the management plugin is enabled, then use the following endpoint:
PUT /api/parameters/shovel/{vhost}/{name}
where {vhost} is the virtual host in which the Shovel should be started and {name} is the name of the new shovel. The endpoint requires that the user that invokes it has policymaker privileges (tag).
The request body is a JSON document similar in structure to that described earlier in this guide:
{ "value": { "src-protocol": "amqp091", "src-uri": "amqp://localhost", "src-queue": "source-queue", "dest-protocol": "amqp091", "dest-uri": "amqp://remote.rabbitmq.local", "dest-queue": "destination-queue" } }
Below is an example that uses curl to declare a shovel on a local node using default user credentials. The shovel will transfer messages between two queues, "source-queue" and "destination-queue", in the default virtual host.
Note that this exact command would fail if invoked against a remote node. Please add a new user tagged as policymaker for your own experiments.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X PUT http://localhost:15672/api/parameters/shovel/%2f/my-shovel \ -H "content-type: application/json" \ -d @- <<EOF { "value": { "src-protocol": "amqp091", "src-uri": "amqp://localhost", "src-queue": "source-queue", "dest-protocol": "amqp091", "dest-uri": "amqp://localhost", "dest-queue": "destination-queue" } } EOF
To declare a shovel using the management UI, first make sure that the management plugin is enabled.
Then
Use rabbitmqctl shovel_status to inspect dynamic shovels in a cluster. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.
rabbitmqctl shovel_status --formatter=pretty_table
The output can be formatted as JSON and redirected to a tool such as jq:
rabbitmqctl shovel_status --formatter=json | jq
GET /api/shovels is an endpoint that can be used to list dynamic shovels in a cluster. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/
To inspect shovels in a specific virtual host, use GET /api/shovels/{vhost} {vhost} is the virtual host name. The value must be percent-encoded.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/%2f
To inspect status of a specific shovels, use GET /api/shovels/vhost/{vhost}/{name} {vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel. Both values must be percent-encoded.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/vhost/%2f/my-shovel
A dynamic Shovel can be restarted. Restarting a shovel briefly interrupts its operations and makes it reconnect to both source and destination. When an appropriate acknowledgement mode is used by a shovel, the interruption is safe: any unacknowledged or unconfirmed ("in flight") messages consumed from the source or published to the destination will be automatically requeued when the shovel is stopped, and consumed again after the restart.
Use rabbitmqctl restart_shovel to restart a shovel using its name. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.
rabbitmqctl restart_shovel "my-shovel"
DELETE /api/shovels/vhost/{vhost}/{name}/restart is an endpoint that restarts a dynamic shovel. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.
{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be restarted. Both values must be percent-encoded.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X DELETE http://localhost:15672/api/shovels/vhost/%2f/my-shovel/restart
To delete a Shovel using CLI tools, use rabbitmqctl clear_parameter and pass shovel for component name and the name of the shovel that should be deleted:
rabbitmqctl clear_parameter shovel "my-shovel"
DELETE /api/parameters/shovel/{vhost}/{name} is the endpoint that can be used to delete a shovel.
{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be deleted. Both values must be percent-encoded.
# Note: this user's access is limited to localhost! curl -v -u guest:guest -X DELETE http://localhost:15672/api/parameters/shovel/%2f/my-shovel
There are several Shovel properties that haven't been covered in the above example. They don't change how dynamic shovels work fundamentally, and do not change the declaration process.
Key | Description |
reconnect-delay | The duration (in seconds) to wait before reconnecting to the brokers after being disconnected at either end. Default is 1. |
ack-mode |
Determines how the shovel should acknowledge consumed messages. If set to on-confirm (the default), messages are acknowledged to the source broker after they have been confirmed by the destination. This handles network errors and broker failures without losing messages, and is the slowest option. If set to on-publish, messages are acknowledged to the source broker after they have been published at the destination. This handles network errors without losing messages, but may lose messages in the event of broker failures. If set to no-ack, message acknowledgements are not used. This is the fastest option, but may lose messages in the event of network or broker failures. |
src-delete-after |
Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis. The default is never, meaning the shovel should never delete itself. If set to queue-length then the shovel will measure the length of the source queue when starting up, and delete itself after it has transferred that many messages. If set to an integer, then the shovel will transfer that number of messages before deleting itself. |
src-prefetch-count | The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000. |
src-exchange |
The exchange from which to consume. Either this or src-queue (but not both) must be set. The shovel will declare an exclusive queue and bind it to the named exchange with src-exchange-key before consuming from the queue. If the source exchange does not exist on the source broker, it will be not declared; the shovel will fail to start. |
src-exchange-key | Routing key when using src-exchange. |
dest-exchange |
The exchange to which messages should be published. Either this or dest-queue (but not both) may be set. If the destination exchange does not exist on the destination broker, it will be not declared; the shovel will fail to start. |
dest-exchange-key | Routing key when using dest-exchange. If this is not set, the original message's routing key will be used. |
dest-publish-properties | A map (JSON object) of properties to overwrite when shovelling messages. Setting headers this way is not currently supported. Default is {}. |
dest-add-forward-headers | Whether to add x-shovelled headers to the shovelled messages indicating where they have been shovelled from and to. Default is false. |
dest-add-timestamp-header | Whether to add x-shovelled-timestamp headers to the shovelled messages containing timestamp (in seconds since epoch) when message had been shovelled. Default is false. |
AMQP 1.0 source and destination properties have some differences from their AMQP 0-9-1 counterparts.
Key | Description |
src-uri |
The AMQP URI for the source. Mandatory. AMQP 1.0 URIs implement
as subset of what is described in the AMQP URI reference.
There is no virtual host concept in AMQP 1.0, so URI path
segments are not supported. The set of query parameters it supports are different from AMQP 0.9.1
URI(s):
|
src-address | The AMQP 1.0 link address. Mandatory. |
dest-address | The AMQP 1.0 link address. Mandatory. |
src-prefetch-count | The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000. |
dest-properties | Properties to overwrite when shovelling messages. See AMQP 1.0 spec §3.2.4 for details of all possible properties. |
dest-application-properties | Application properties to set when shovelling messages. |
dest-add-forward-headers | Whether to add x-shovelled application properties to the shovelled messages indicating where they have been shovelled from and to. Default is false. |
dest-add-timestamp-header | Whether to set the creation_time header to the timestamp (in milliseconds since epoch) of the moment when message had been republished. Default is false. |
reconnect-delay | The duration (in seconds) to wait before reconnecting to the brokers after being disconnected at either end. Default is 1. |
ack-mode |
Determines how the shovel should acknowledge consumed messages. If set to on-confirm (the default), messages are acknowledged to the source broker after they have been confirmed by the destination. This handles network errors and broker failures without losing messages, and is the slowest option. If set to on-publish, messages are acknowledged to the source broker after they have been published at the destination. This handles network errors without losing messages, but may lose messages in the event of broker failures. If set to no-ack, message acknowledgements are not used. This is the fastest option, but may lose messages in the event of network or broker failures. |
src-delete-after |
Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis. The default is never, meaning the shovel should never delete itself. If set to queue-length then the shovel will measure the length of the source queue when starting up, and delete itself after it has transferred that many messages. If set to an integer, then the shovel will transfer that number of messages before deleting itself. |
See Monitoring Shovels in the overview Shovel plugin guide.
If you have questions about the contents of this guide or any other topic related to RabbitMQ, don't hesitate to ask them using GitHub Discussions or our community Discord server.
If you'd like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!