• Features
• Get Started
• Support
• Community
• Docs
• Blog

Configuring Dynamic Shovels

Overview

This guide focuses on dynamically configured shovels. It assumes familiarity with the key concepts behind the Shovel plugin.

Unlike with static shovels, dynamic shovels are configured using runtime parameters. They can be started and stopped at any time, including programmatically. Dynamic shovels can be used for both transient (one-off) and permanently running workloads.

Information about dynamic shovels is stored in RabbitMQ's schema database, along with users, permissions, queues, etc. They therefore can be exported together with other schema definitions.

Configuration

Parameters can be defined using rabbitmqctl, through the management HTTP API, or (with the rabbitmq_shovel_management plugin enabled) through the management UI's administrative section.

A shovel is declared with a definition body, which is a JSON object. Some keys are mandatory, others are optional. They control connection parameters, protocol used, message transfer source and destination, data safety protocol features, and more.

Every shovel belongs to a virtual host. Note that a Shovel can consume from and publish to not only a different virtual host but an entirely different cluster, so virtual host selection primarily acts as a way of organising shovels and access to them, much like with the rest of permission in RabbitMQ.

Every shovel is also named. The name is used to identify shovels when inspecting their status, deleting them or restarting them.

Declaring a Dynamic Shovel

In this example we will set up a dynamic shovel that will move messages from the queue "source-queue" in the local RabbitMQ cluster to the queue "target-queue" on a remote RabbitMQ node, using AMQP 0-9-1.

Using CLI Tools

A shovel is declared using the rabbitmqctl set_parameter command with component name shovel, a shovel name and a definition body which is a JSON document:

# my-shovel here is the name of the shovel
rabbitmqctl set_parameter shovel my-shovel \
  '{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "source-queue", "dest-protocol": "amqp091", "dest-uri": "amqp://remote-server", "dest-queue": "target-queue"}'

On Windows rabbitmqctl is named rabbitmqctl.bat and command line value escaping will be different:

rabbitmqctl.bat set_parameter shovel my-shovel ^
  "{""src-protocol"": ""amqp091"", ""src-uri"":""amqp://localhost"", ""src-queue"": ""source-queue"", ^
   ""dest-protocol"": ""amqp091"", ""dest-uri"": ""amqp://remote.rabbitmq.local"", ^
   ""dest-queue"": ""target-queue""}"

The body in this example includes a few keys:

There are other Shovel definition keys that will be covered later in this guide.

Using HTTP API

To declare a shovel using the HTTP API, make sure that the management plugin is enabled, then use the following endpoint:

PUT /api/parameters/shovel/{vhost}/{name}

where {vhost} is the virtual host in which the Shovel should be started and {name} is the name of the new shovel. The endpoint requires that the user that invokes it has policymaker privileges (tag).

The request body is a JSON document similar in structure to that described earlier in this guide:

{
  "value": {
    "src-protocol": "amqp091",
    "src-uri":  "amqp://localhost",
    "src-queue":  "source-queue",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://remote.rabbitmq.local",
    "dest-queue": "destination-queue"
  }
}

Below is an example that uses curl to declare a shovel on a local node using default user credentials. The shovel will transfer messages between two queues, "source-queue" and "destination-queue", in the default virtual host.

Note that this exact command would fail if invoked against a remote node. Please add a new user tagged as policymaker for your own experiments.

# Note: this user's access is limited to localhost!
curl -v -u guest:guest -X PUT http://localhost:15672/api/parameters/shovel/%2f/my-shovel \
                       -H "content-type: application/json" \
                       -d @- <<EOF
{
  "value": {
    "src-protocol": "amqp091",
    "src-uri": "amqp://localhost",
    "src-queue": "source-queue",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://localhost",
    "dest-queue": "destination-queue"
  }
}
EOF

Using Management UI

To declare a shovel using the management UI, first make sure that the management plugin is enabled.

Then

• Navigate to Admin > Shovel Management > Add a new shovel
• Fill out the form with shovel parameters covered earlier in this guide
• Click Add shovel

Inspecting Status of Dynamic Shovels

Using CLI Tools

Use rabbitmqctl shovel_status to inspect dynamic shovels in a cluster. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.

rabbitmqctl shovel_status --formatter=pretty_table

The output can be formatted as JSON and redirected to a tool such as jq:

rabbitmqctl shovel_status --formatter=json | jq

Using HTTP API

GET /api/shovels is an endpoint that can be used to list dynamic shovels in a cluster. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/

To inspect shovels in a specific virtual host, use GET /api/shovels/{vhost} {vhost} is the virtual host name. The value must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/%2f

To inspect status of a specific shovels, use GET /api/shovels/vhost/{vhost}/{name} {vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/vhost/%2f/my-shovel

Using Management UI

• Navigate to Admin > Shovel Status
• Locate the shovel of interest in the table

Restarting a Shovel

A dynamic Shovel can be restarted. Restarting a shovel briefly interrupts its operations and makes it reconnect to both source and destination. When an appropriate acknowledgement mode is used by a shovel, the interruption is safe: any unacknowledged or unconfirmed ("in flight") messages consumed from the source or published to the destination will be automatically requeued when the shovel is stopped, and consumed again after the restart.

Using CLI Tools

Use rabbitmqctl restart_shovel to restart a shovel using its name. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.

rabbitmqctl restart_shovel "my-shovel"

Using HTTP API

DELETE /api/shovels/vhost/{vhost}/{name}/restart is an endpoint that restarts a dynamic shovel. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.

{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be restarted. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X DELETE http://localhost:15672/api/shovels/vhost/%2f/my-shovel/restart

Using Management UI

• Navigate to Admin > Shovel Status
• Locate the shovel of interest in the table
• Click Restart and wait for the next UI refresh

Deleting a Shovel

Using CLI Tools

To delete a Shovel using CLI tools, use rabbitmqctl clear_parameter and pass shovel for component name and the name of the shovel that should be deleted:

rabbitmqctl clear_parameter shovel "my-shovel"

Using HTTP API

DELETE /api/parameters/shovel/{vhost}/{name} is the endpoint that can be used to delete a shovel.

{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be deleted. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
curl -v -u guest:guest -X DELETE http://localhost:15672/api/parameters/shovel/%2f/my-shovel

AMQP 0-9-1 Shovel Definition Reference

There are several Shovel properties that haven't been covered in the above example. They don't change how dynamic shovels work fundamentally, and do not change the declaration process.

AMQP 1.0 Shovel Definition Reference

AMQP 1.0 source and destination properties have some differences from their AMQP 0-9-1 counterparts.

Monitoring Shovels

See Monitoring Shovels in the overview Shovel plugin guide.