Which is better: mirrored classic queues or quorum queues? Quorum queues are the much better choice and they will be the only option starting with RabbitMQ version 4.0. This information explains why, the reasons why you should migrate from mirrored classic queues to quorum queues, the ways to handle features during the migration, and includes procedures for some of the migration routes you can take.
You should migrate to mirrored classic queues for the following reasons:
However, before migrating to quorum queues, a few things must be considered:
While quorum queues are a better queue type when compared to mirrored classic queues, they are not 100% compatible feature wise with mirrored classic queues. When you are deciding about whether to migrate from mirrored classic queues to quorum queues, it is recommended to review the quorum queue documentation first, you can review the feature matrix table which provides a comparison of both queue types (mirrored classic queues beside quorum queues)
The level of complexity involved in migrating from mirrored classic queues to quorum queues depends on the features that are currently being used by the mirrored classic queues. Some features require a change in the way queues are being used (refer to Mirrored Classic Queue Features that require Changes in the Way the Queue is Used), while other features simply require removing the feature from the source code or moving it to policy (refer to Mirrored Classic Queue Features that can be removed from Source Code or moved to a Policy).
It is also important to note that migrated applications should be thoroughly tested against quorum queues because the behaviour can differ under the load and in edge cases.
Incompatible features can be either referenced in policies or in the source code. RabbitMQ strictly validates arguments for queue declaration and consumption. Therefore, for migration, you must clean up all information about incompatible features in the source code. For some features, changes in the way that queues are used is required, refer to Mirrored Classic Queue Features that require Changes in the Way the Queue is Used. For other features, it is as simple as just removing corresponding strings from the source code or moving the feature to a policy, refer to Mirrored Classic Queue Features that can be removed from Source Code or moved to a Policy,
The general policies and arguments related to mirroring are:ha-mode, ha-params ha-sync-mode, ha-promote-on-shutdown, ha-promote-on-failure, and queue-master-locator.
There are several migration paths available:
Before deciding which migration method you can use, you must first find the mirrored classic queues and the features they are using.
To find the mirrored classic queues that must be migrated, run the following script (which uses rabbitmqctl to count all the queues across all the virtual hosts as tab-separated values).
Note, the following command uses effective_policy_definition parameters, which are only available since RabbitMQ version 3.10.13/3.11.5. If it's not available, you can use rabbitmqctl from any RabbitMQ version later than 3.10.13/3.11.5, or manually match the policy name to it's definition.
#!/bin/sh printf "%s\t%s\t%s\n" vhost queue_name mirrors for vhost in $(rabbitmqctl -q list_vhosts | tail -n +2) ; do rabbitmqctl -q list_queues -p "$vhost" name durable policy effective_policy_definition arguments mirror_pids type | sed -n '/\t\[[^\t]\+\tclassic$/{s/\t\[[^\t]\+\tclassic$//; p}' | xargs -x -r -L1 -d '\n' printf "%s\t%s\n" "$vhost" done
All mirrored classic queues that include ha-mode in their effective policy definition must be migrated to a different type of queue. All these queues are listed as mirrored classic queues in the Management UI and CLI. Find the policies that apply it by running the following script:
#!/bin/sh printf "%s\t%s\t%s\t%s\t%s\t%s\n" vhost policy_name pattern apply_to definition priority for vhost in $(rabbitmqctl -q list_vhosts | tail -n +2) ; do rabbitmqctl -q list_policies -p "$vhost" | grep 'ha-mode' done
When one or more of the following features are used by mirrored classic queues, straightforward migration to quorum queues is not possible. The way the application interacts with a broker needs to be changed. This information explains how to find whether some of these features are being used in a running system, and the changes you must make to make migration easier.
To find out if a classic mirrored queue uses the "priority" feature, you can check for the x-max-priority string in the list of queues output that is provided by running the command in Finding the Mirrored Classic Queues for Migration or you can also search for the x-max-priority string in the source code. For more information on how the priority is implemented, go to Priority Queue Support.
Priority queues are not created using a policy, therefore no policy changes are required when migrating them. Classic mirrored queues create a separate queue for every priority behind the scenes. To migrate a single mirrored classic queue that uses the "priority" feature, you must create the required number amount of quorum queues. Once the quorum queues are created, adjust the publishing and consumption of these new quorum queues accordingly.
The queue length exceeded with overflow set to reject-publish-dlx is not supported by quorum queues. The reject-publish-dlx value is not supported.
With mirrored classic queues, publishing to a full queue with reject-publish-dlx resulted in RabbitMQ republishing a rejected message to a dead letter exchange. With quorum queues, to apply the same logic, you must change reject-publish-dlx to reject-publish. Then, handle negative acknowledgements: after getting a negative acknowledgement, the application must publish the message again to a different exchange.
To find out if overflow set to reject-publish-dlx is configured for the mirrored classic queues you want to migrate, check for the reject-publish-dlx string in the list of queues output that is provided by running the command in Finding the Mirrored Classic Queues for Migration or you can also search for the reject-publish-dlx string in the source code.
Global QoS prefetch where a channel sets a single prefetch limit for all consumers using that channel is not supported by quorum queues. If this functionality is required, try achieving the same results using alternative methods, for example, one solution might be to use a lower per-consumer QoS (given the known application load pattern).
To find out if this feature is used, run the following command on a running system and check for non-empty output:
rabbitmqctl list_channels pid name global_prefetch_count | sed -n '/\t0$/!p'
A list of channel PIDs that have global QoS turned on are returned. Then, run the following command to map the channel PID to a queue name to verify if it is a mirrored classic queue.
rabbitmqctl list_consumers queue_name channel_pid
Classic mirrored queues consumers can be automatically cancelled when a queue leader fails over. This can cause loss of information about which messages were sent to which consumer, and result in the same messages being sent again (duplicate messages).
Some of the cases for duplicate messages are covered by x-cancel-on-ha-failover and others are not. Most of the cases covered by x-cancel-on-ha-failover do not exist with quorum queues but those that are not covered are still there. Therefore, your application must be able to handle duplicates, which it should be able to do anyway.
The following features are ignored when quorum queues are used. The best way to handle these features is to remove them from the source code, or move them to a policy.
Quorum queues do not support lazy mode (x-queue-mode=lazy).
To migrate mirrored lazy classic queues, remove the x-queue-mode=lazy declaration argument or remove it from the policy if it is set via a policy. For more information about the lazy mode, go to Lazy Queues.
Transient queues are deleted on a node/cluster boot.
The plan is to remove transient queues in future RabbitMQ releases. The only option for transient queues then will be exclusive queues. This only affects the durability of queue definitions. Messages can still be marked transient.
You must make a decision about transient queues before migration, is the content of the queue important enough to get availability guarantees of quorum queues, or is it better to downgrade the transient queue to a classic non-mirrored queue (classic mirrored queues are being removed but classic non-mirrored queues will still be available).
You do not need to complete any migration tasks for exclusive queues. Exclusive queues are not mirrored even if the policy indicates that they are. Also, it is not possible to create an exclusive quorum queue.
For exclusive queues, however, you must decide whether to leave the queue as exclusive or change it to a replicated queue during migration. Be careful not to make exclusive queue declarations with an explicit x-queue-type: quorum argument.
This procedure to migrate from mirrored classic queues to quorum queues is similar to a blue-green cluster upgrade, except you are migrating to a new virtual host on the same RabbitMQ cluster. The steps in the following sections use a new virtual host on the existing cluster to provide an empty namespace to create the new quorum queues using the old queue names.
You will use the Federation Plugin to seamlessly migrate from the old virtual host to the new one.
Important: You can set the default queue type for the new virtual host. Setting it to quorum creates all the queues without an explicit type as quorum queues (except for exclusive, non-durable, or auto-delete queues).
If all incompatible features were cleaned up from the source code, and there is no explicit x-queue-type arguments in the source code, then the same code should work for both the old virtual host with classic mirrored queues and the new virtual host with quorum queues. The only change you need to make is to update the virtual host connection parameters to connect to the new virtual host.
Create the new virtual host with the correct default queue type (quorum) in the existing cluster. The queue type should be selected from the queue type drop down list when the new virtual host is being added via management UI. Alternatively, it can also be created using the CLI interface by specifying the default queue type and adding the permissions. Ensure all required users have access and can connect to the new virtual host by following the steps in the set permissions.
bash rabbitmqctl add_vhost NEW_VHOST --default-queue-type quorum rabbitmqctl set_permissions -p NEW_VHOST USERNAME '.*' '.*' '.*'
A new federation upstream should be created for the NEW_VHOST with the URI pointing to the OLD_VHOST: amqp:///OLD_VHOST. (Note that the default vhost URI is amqp:///%2f).
The federation upstream can be created using the management UI or the CLI:
rabbitmqctl set_parameter federation-upstream quorum-migration-upstream \ --vhost NEW_VHOST \ '{"uri":"amqp:///OLD_VHOST", "trust-user-id":true}'
When this form of URI with an empty hostname is used, there is no need to specify credentials. Connection is only possible within the bounds of a single cluster.
If the user-id in messages is being used for any purpose, it can also be preserved as shown in the previous CLI example.
Export the definitions from the source virtual host to a file. This is available on the Overview page of the management UI (don't forget to select a single virtual host). Alternatively, you can export the definitions using the CLI with the following command:
rabbitmqadmin export -V OLD_VHOST OLD_VHOST.json
Make the following changes to this file before loading it back into the NEW_VHOST:
Now the modified schema can be loaded into the new virtual host from the Management UI or by running the following command from the CLI:
rabbitadmin import -V NEW_VHOST NEW_VHOST.json
Consumers of the migrated queues can now access the new queues by updating the connection parameters to connect to the new virtual host. The federation links start to pull in messages from the original queues.
As with a blue-green cluster, after all consumers are migrated, you might need to also add shovels to move the backlog of the original queues to the new queues more efficiently than federation. For more information, refer to Drain Messages.
Once the original queues are empty (or nearly empty if you do not require full message ordering), the producers should be stopped and reconfigured to use the new queue declarations and virtual host like the consumers, and restarted. Federated exchanges in the old virtual host should also be stopped and equivalent exchanges should be added in the new virtual host. The original queues can be removed once they are empty and no messages are passing through them.
Under sufficient system load, messages from the old virtual host will not be picked up. If message ordering is important, then ordering should be completed in these steps: stop producers, shovel remaining messages to the new virtual host, and start consumers on the new virtual host.
For every non-empty queue in the old virtual host, a shovel needs to be configured. For example:
rabbitmqctl set_parameter shovel migrate-QUEUE_TO_MIGRATE \ '{"src-protocol": "amqp091", "src-uri": "amqp:///OLD_VHOST", "src-queue": "QUEUE_TO_MIGRATE", "dest-protocol": "amqp091", "dest-uri": "amqp:///NEW_VHOST", "dest-queue": "QUEUE_TO_MIGRATE"}'
After the queue is drained, the shovel can be deleted:
rabbitmqctl clear_parameter shovel migrate-QUEUE_TO_MIGRATE
Migrating this way trades uptime so that you can complete the migration in an existing virtual host and cluster.
For each queue (or some group of queues) being migrated, it should be possible to stop all the consumers and producers for the duration of the migration.
All incompatible features should be cleaned up. In addition, every place where queues are being declared, it would be better to make the x-queue-type argument configurable without changing the application code.
If you have questions about the contents of this guide or any other topic related to RabbitMQ, don't hesitate to ask them using GitHub Discussions or our community Discord server.
If you'd like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!