Skip to main content
Version: 4.1

Policies

Overview​

While much of the configuration for RabbitMQ lives in the configuration file, some things do not mesh well with the use of a configuration file:

  • If they need to be the same across all nodes in a cluster
  • If they are likely to change at run time

Such values in RabbitMQ are stored as runtime parameters. Policies is one example of how runtime parameter are used.

This guide focuses on policies and operator policies.

Why Policies Exist​

tip

Policies is a declarative mechanism of configuring certain properties of groups of queues, streams, exchanges. They are designed for parameters that can change at runtime.

note

Since queue (stream) type is set at declaration time and cannot be changed, policies cannot be used to configure queue type, by design. To specify queue (stream) type, use optional arguments.

Policies is a declarative, virtual host-scoped mechanism for dynamically [re]configuring optional arguments for queues, exchanges, streams, and some plugins (namely federation).

Policies vs. The Optional Arguments Set by Applications​

Before we explain what policies are and how to use them it would be helpful to explain why they were introduced to RabbitMQ.

In addition to mandatory properties (e.g. durable or exclusive), queues and exchanges in RabbitMQ have optional arguments, sometimes referred to as x-arguments.

Those are provided by clients when they declare queues (exchanges) and control various optional features, such as queue length limit or TTL.

Client-controlled properties in some of the protocols RabbitMQ supports generally work well but they can be inflexible: updating TTL values or mirroring parameters that way required application changes, redeployment and queue re-declaration (which involves deletion). In addition, there is no way to control the extra arguments for groups of queues and exchanges. Policies were introduced to address the above pain points.

A policy matches one or more queues by name (using a regular expression pattern) and appends its definition (a map of optional arguments) to the x-arguments of the matching queues. In other words, it is possible to configure x-arguments for multiple queues at once with a policy, and update them all at once by updating policy definition.

In modern versions of RabbitMQ the set of features which can be controlled by policy is not the same as the set of features which can be controlled by client-provided arguments.

How Policies Work​

Key policy attributes are

  • name: it can be anything but ASCII-based names without spaces are recommended
  • pattern: a regular expression that matches one or more queue (exchange) names. Any regular expression can be used.
  • definition: a set of key/value pairs (think a JSON document) that will be injected into the map of optional arguments of the matching queues and exchanges
  • policy priority used to determine which policy should be applied to a queue or exchange if multiple policies match its name

Policies automatically match against exchanges and queues, and help determine how they behave. Each exchange or queue will have at most one policy matching (see Combining Policy Definitions below), and each policy then injects a set of key-value pairs (policy definition) on to the matching queues (exchanges).

Policies can match only queues of a specific type, all queues, only exchanges, or all queues and exchanges. This is controlled using the apply-to flag when a policy is created.

Policies can change at any time. When a policy definition is updated, its effect on matching exchanges and queues will be reapplied. Usually it happens instantaneously but for very busy queues can take a bit of time (say, a few seconds).

Policies are matched and applied every time an exchange or queue is created, not just when the policy is created.

Policies can be used to configure

and other features.

What Cannot Be Set Using a Policy​

Some optional arguments cannot be configured using a policy. They control various settings that cannot be changed at runtime. Two key examples are:

Those values intentionally cannot be configured by policies: their values are fixed at queue declaration time.

How to Define a Policy​

An example of defining a policy looks like:

rabbitmqctl set_policy federate-me \
    "^federated\." '{"federation-upstream-set":"all"}' \
    --priority 1 \
    --apply-to exchanges
danger

When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use.

This matches the value "all" with the key "federation-upstream-set" for all exchanges with names beginning with "federated.", in the virtual host "/".

The "pattern" argument is a regular expression used to match exchange or queue names.

In the event that more than one policy can match a given exchange or queue, the policy with the greatest priority applies.

The "apply-to" argument can be one of the following:

  • "exchanges", applies to exchanges only
  • "queues", applies to all types of queues, including streams
  • "classic_queues", applies to classic queues only
  • "quorum_queues", applies to quorum queues only
  • "streams", applies to streams only
  • "all", applies to all exchanges and queues (including streams)

The "apply-to" and "priority" settings are optional. The default values are "all" and "0" respectively.

Policy Priorities​

Policy patterns are matched against exchange and queue names to determine what policy (if any) should then inject a set of key-value pairs (the definition of that policy) into the optional arguments of matching queues (exchanges).

At most one policy matches a queue or exchange. Since multiple policies can match a single name, a mechanism is needed to resolve such policy conflicts. This mechanism is called policy priorities. Every policy has a a numeric priority associated with it. This priority can be specified when declaring a policy. If not explicitly provided, the priority of 0 (a very low one) will be used.

important

At most one policy matches a queue or exchange. Matching policies are then sorted by priority and the one with the highest priority will take effect.

Higher values indicate higher priority: a policy with priority 10 will overrule a policy with priority 8, and both will overrule the default priority of 0.

danger

When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use.

Matching policies are then sorted by priority and the one with the highest priority will take effect.

When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use.

Combining Policy Definitions​

In some cases we might want to apply more than one policy definition to a resource. For example we might need a queue to be federated and has message TTL. At most one policy will apply to a resource at any given time, but we can apply multiple definitions in that policy.

A federation policy definition would require an upstream set to be specified, so we would need the federation-upstream-set key in our definition. On the other hand to define some queues as TTL-enabled, we would need the TTL-related keys key to be defined as well for the policy. The policy definition is just a JSON object and can have multiple keys combined in the same policy definition.

Here's an example:

rabbitmqctl set_policy ttl-fed \
    "^tf\." '{"federation-upstream-set":"all", "message-ttl":60000}' \
    --priority 1 \
    --apply-to queues

By doing that all the queues matched by the pattern "^tf\." will have the "federation-upstream-set" and the policy definitions applied to them.

Operator Policies​

Difference From Regular Policies​

Sometimes it is necessary for the operator to enforce certain policies. For example, it may be desirable to force queue TTL but still let other users manage policies. Operator policies allow for that.

Operator policies are much like regular ones but their definitions are used differently. They are merged with regular policy definitions before the result is applied to matching queues.

Because operator policies can unexpectedly change queue attributes and, in turn, application assumptions and semantics, they are limited only to a few arguments:

ClassicQuorumStream
delivery-limit ✓
expires ✓ ✓
max-in-memory-bytes ✓
max-in-memory-length ✓
max-length ✓ ✓
max-length-bytes ✓ ✓ ✓
message-ttl ✓ ✓
target-group-size ✓

Conflict Resolution with Regular Policies​

An operator policy and a regular policy can contain the same keys in their definitions. When it happens, the more conservative value is chosen as effective. For example, if a matching operator policy definition sets max-length to 50 and a matching regular policy definition uses the value of 100, the value of 50 will be used. If, however, regular policy's value was 20, it would be used. Operator policies, therefore, don't just overwrite regular policy values. They enforce limits but try to not override user-provided policies where possible.

ClassicQuorumStream
delivery-limit
  • lesser value
expires
  • lesser value
  • lesser value of the two policies
  • policy precedence over queue arguments
max-in-memory-bytes
  • lesser value
max-in-memory-length
  • lesser value
max-length
  • lesser value
  • lesser value
max-length-bytes
  • lesser value
  • lesser value
  • lesser value of the two policies
  • policy precedence over queue arguments
message-ttl
  • lesser value
  • lesser value
target-group-size
  • greater value

When the same key is provided by both client-provided x-arguments and by a user policy, the former take precedence. The exception to this rule are all stream configuration settings as well as (x-)overflow for quorum queues where the policy takes precedence.

However, if an operator policy is also used, that will take precedence over the client-provided arguments, too. Operator policies are a protection mechanism and override client-provided values and user policy values.

Use operator policies to introduce guardrails for application-controlled parameters related to resource use (e.g. peak disk space usage).

Defining Operator Policies​

Operator policies are defined in a way very similar to regular (user) policies. When rabbitmqctl is used, the command name is set_operator_policy instead of set_policy. In the HTTP API, /api/policies/ in request path becomes /api/operator-policies/:

rabbitmqctl set_operator_policy transient-queue-ttl \
    "^amq\." '{"expires":1800000}' \
    --priority 1 \
    --apply-to queues
danger

When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use.

How to Disable Operator Policy Changes​

Modification of operator policies via the HTTP API and Web UI can be disabled in configuration. This makes operator policies read-only for all users via the HTTP API and Web UI.

management.restrictions.operator_policy_changes.disabled = true

Troubleshooting​

Multiple Policies Have Conflicting Priorities​

To verify this hypothesis, list the policies in the virtual host and see if any policies have a priority equal to the one you are troubleshooting.

rabbitmqctl --vhost "target.vhost" list_policies --formatter=pretty_table

If the results are empty, this means that no policies in the target virtual host have matched the target object (queue, stream, exchange).

rabbitmqadmin v2 additionally can list the policies that match a specific object (if any):

# lists all policies that match a queue named "a.queue" in a virtual host named "target.vhost"
rabbitmqadmin --vhost "target.vhost" policies list_matching_object --name "a.queue" --type "queues"
# lists all policies that match an exchanged named "an.exchange" in a virtual host named "target.vhost"
rabbitmqadmin --vhost "target.vhost" policies list_matching_object --name "an.exchange" --type "exchanges"

Look for any and all policies that have equal priorities. Only one of them will apply to a matching object (queue, stream, exchange), and the selection should be considered non-deterministic (random).

Policy Pattern or Target Type Does Not Match Object's Name or Type​

To verify this hypothesis, list the policies that match the object (e.g. queue) in question using rabbitmqadmin v2.

# lists all policies that match a queue named "a.queue" in a virtual host named "target.vhost"
rabbitmqadmin --vhost "target.vhost" policies list_matching_object --name "a.queue" --type "queues"
# lists all policies that match an exchanged named "an.exchange" in a virtual host named "target.vhost"
rabbitmqadmin --vhost "target.vhost" policies list_matching_object --name "an.exchange" --type "exchanges"

If the results are empty, this means that no policies in the target virtual host have matched the target object (queue, stream, exchange).

Policy Declared in the Wrong Virtual Host​

Policies are virtual host-scoped, that is, belong to a particular virtual host and only apply to the objects (queues, streams, exchanges) in that virtual host.

A policy can be declared in the default virtual host by mistake. To verify this hypothesis, list only the policies in that virtual host.

rabbitmqctl --vhost "target.vhost" list_policies --formatter=pretty_table

If the results are empty, this means that there are no policies defined in the target virtual host.