RabbitMQ - Broker Query Language Plugin

Installation

Please see the plugin installation guide for more detailed instructions on plugin installation.

Note that at this stage, the RabbitMQ BQL plugin has not been formally released, and must be built from source before installation. To do this, complete the following steps:
- Setup a RabbitMQ plugin development environment by following the instructions at Plugin Development: Getting Started.
- Once the make operation has completed, the dist/ subdirectory of rabbitmq-bql (within the umbrella) should contain a number of .ez files. Copy amqp_client.ez and rabbitmq-bql.ez into the plugins/ subdirectory of your installed RabbitMQ instance (you will have to create that directory if it does not exist already).
At this point, restart RabbitMQ to pick up the new plugins.
As part of the BQL build process, rabbitmq-bql-client.zip will be produced. This contains the client used for connecting to the BQL plugin. It is self-contained, and can be used on any machine with a functional Erlang runtime. This client should be copied and unzipped on a suitable machine before proceeding. Note that it is perfectly valid to run the client on the same machine as the Rabbit server; however in production scenarios it is recommended that the client be run as a different user to the Rabbit server, simply to prevent mis-use of the Rabbit server account.

Server Configuration

In most scenarios, the BQL server plugin requires no configuration. However, if you've removed the default guest/guest user, then you'll need to configure the plugin to connect with different credentials. The BQL plugin requires a valid user to connect to RabbitMQ internally to subscribe to the bql.query queue.

To set a custom username/password combination for the plugin to connect with, you'll need to add a rabbitmq_bql section to your rabbitmq.config file. The location of this configuration file is documented on the installation page under the section relevant to your platform. A rabbitmq.config with an appropriate section would look like:

[
  {rabbitmq_bql, [
    {user, "bql-user"},
    {password, "bql-password"}
  ]}
].

You'll need to restart RabbitMQ after changing this configuration.

Running the Client

Within the client zip, the bql (or bql.bat) script can be executed. All configuration parameters have defaults that will connect the client to a running Rabbit instance on the same host with default credentials. For example, to run this on a Linux platform running rabbitmq-server, you would execute:

$ ./bql
BQL>

To ensure that your console is able to connect to Rabbit, execute a simple command such as:

BQL> select * from exchanges

You should see output similar to:

------------------------------------------------------------------
| name             | type    | durable | auto_delete | arguments |
------------------------------------------------------------------
| amq.rabbitmq.log | topic   | true    | false       |           |
| amq.match        | headers | true    | false       |           |
| amq.headers      | headers | true    | false       |           |
| amq.topic        | topic   | true    | false       |           |
| amq.direct       | direct  | true    | false       |           |
| amq.fanout       | fanout  | true    | false       |           |
|                  | direct  | true    | false       |           |
| bql.query        | direct  | true    | false       |           |

Configuring the Client

The client supports a number of configuration options to control the AMQP host and credentials used for the connection. This section will discuss the various options, along with examples for their use.

Client Options

The supported options are:

-host - configures the AMQP host to connect to. For example, -host myserver. Defaults to localhost
-port - configures the AMQP port to connect to. For example, -port 1234. Defaults to 5672
-username - configures the username to authenticate with. For example, -username foo. Defaults to guest
-password - configures the password to authenticate with. For example, -password foo. Defaults to guest
-vhost - configures the vhost to connect to an administer. For example, -vhost /myvhost. Defaults to /
-execute - instructs the client to execute the BQL commands in the provided script and exit. For example, -execute mycommands.bql

Option Examples

The following illustrate examples of various connection scenarios for the BQL client and the options that can be used to achieve them.

Connecting to a default-configured Rabbit on the same machine
```
bql
```
Connecting to a default-configured Rabbit on host myhost
```
bql -host myhost
```
Connecting to a Rabbit on host myhost using vhost /myvhost
```
bql -host myhost -vhost /myvhost
```
Connecting to a Rabbit on the same machine with alternate credentials
```
bql -username foo -password fooword
```

Available Commands

BQL supports many different commands, falling broadly into categories of:

Exchange Management
Queue Management
Route Management
Basic Messaging
VHost Management
User Management
Permission Management

The commands available in each category are described in detail below.

General Structure of Commands

Broadly, BQL aims to provide a syntax very similar to SQL. However, it should be noted that it often departs from a pure SQL-92 syntax in order to provide a more relevant experience to the Broker data structure.

Commands are generally in the form of:

BQL> create exchange myexchange;
ok
BQL> create durable queue 'myqueue'
ok
BQL> select name,messages from queues where 'durable'=true order by name
----------------------
| name    | messages |
----------------------
| myqueue | 0        |

There are a number of points to note about these commands:

Single line commands are optionally followed by a semicolon. In BQL scripts however, multiples lines must be deliniated by a semicolon
Names (such as myqueue) are optionally quoted with single-quotes. Similarly for where and order by clauses.
Where keywords are used as names (for example, durable in the where clause), they need to be quoted, otherwise a syntax error will be reported.
Select queries support "where" and "order by" clauses. Group by and joins are not supported. Aggregation predicates are not currently supported (eg, max(messages)).

Exchange Management

Querying

Exchanges can be queried by selecting from the "exchanges" entity. For example,

BQL> select * from exchanges
------------------------------------------------------------------
| name             | type    | durable | auto_delete | arguments |
------------------------------------------------------------------
| amq.rabbitmq.log | topic   | true    | false       |           |
| amq.match        | headers | true    | false       |           |
| amq.headers      | headers | true    | false       |           |
| amq.topic        | topic   | true    | false       |           |
| amq.direct       | direct  | true    | false       |           |
| amq.fanout       | fanout  | true    | false       |           |
|                  | direct  | true    | false       |           |
| bql.query        | direct  | true    | false       |           |

Creation

Exchange creation follows the form:

create [durable] [direct | topic | headers | fanout] exchange <name>

A command in the form:

create exchange myX

will create a non-durable direct exchange. To override any of these defaults, appropriate keywords should be added either to make the exchange durable or to change the exchange type.

Deletion

Exchange deletion follows the form:

drop exchange <name>

No additional flags are required or supported.

Queue Management

Querying

Queues can be queried by selecting from the "queues" entity. For example,

BQL> select * from queues
-----------------------------------------------------------------------------------------------------------
| name      | durable | auto_delete | arguments | pid          | messages_ready | messages_unacknowledged |
-----------------------------------------------------------------------------------------------------------
| myqueue   | true    | false       |           | <4359.299.0> | 0              | 0                       |

Note that in the above sample, a number of columns have been removed due to the excessive length of the line. It is generally recommended that only specific fields are selected for queues in order to maintain readability of the output.

Creation

Queue creation follows the form:

create [durable] queue <name>

A command in the form:

create queue myQ

will create a non-durable queue. To create a durable queue, the durable keyword should be used.

Deletion

Queue deletion follows the form:

drop queue <name>

No additional flags are required or supported.

Route Management

One of the strengths of an AMQP broker is the ability to configure complex custom routing. BQL provides an idea mechanism for allowing dynamic configuration of the routing table.

Querying

Routes can be queried by selecting from the "bindings" entity. For example,

BQL> select * from bindings;
---------------------------------------------------
| exchange_name | queue_name | routing_key | args |
---------------------------------------------------
|               | myqueue    | myqueue     |      |
| amq.direct    | myqueue    | a           |      |

Creation

Route creation follows the form:

create route from <queue-name> to <exchange-name> [when routing_key is <routing-key>]

A command in the form:

create route from myX to myQ

will create a binding from myX to myQ with an empty routing key. If a specific routing key is desired, then a command in the form:

create route from myX to myQ when routing_key is 'key'

should be issued.

Deletion

Route deletion follows the form:

drop route from <queue-name> to <exchange-name> [when routing_key is <routing-key>]

When the routing_key clause is left off, an empty routing key is assumed (similarly to the create command).

Basic Messaging

BQL provides for sending and recieving simple messages from within the console. Note that only text-based messages can currently be operated with within the console at this stage.

Sending Messages

Messages can be sent with commands in the form:

post <message-body> to <exchange-name> [with routing_key <routing-key>]

If a routing-key is not specified, then an empty routing key is attached. An example of sending a message would be:

post 'hello world' to 'amq.direct' with routing_key 'rk'

Receiving Messages

Messages can be retrieved from a queue with commands in the form:

get from <queue-name>

If no messages are available, then the response

empty

will be shown. Otherwise, the body of the message will be shown. For example:

BQL> create queue testqueue;
ok
BQL> create route from 'amq.direct' to testqueue when routing_key is 'something';
ok
BQL> post 'Hello' to 'amq.direct' with routing_key 'something';
ok
BQL> get from testqueue;
<<"Hello">>
BQL> get from testqueue;
empty

Note that get will not block waiting for a message. If no message is available at the time it is called, it will respond with empty. Also, this command removes the retrieved message from the queue - so if you make this call on your application's queue, then the message will no longer be available for the application to consume.

VHost Management

Querying

VHosts can be queried by selecting from the "vhosts" entity. For example,

BQL> select * from vhosts
--------
| name |
--------
| /    |

Creation

VHost creation follows the form:

create vhost <name>

An example of creating a vhost would be:

create vhost '/test'

Deletion

VHost deletion follows the form:

drop vhost <name>

No additional flags are required or supported.

User Management

Querying

Users can be queried by selecting from the "users" entity. For example,

BQL> select * from users
---------
| name  |
---------
| guest |

Creation

User creation follows the form:

create user <name> identified by <password>

An example of creating a user would be:

create user newuser identified by password

Deletion

User deletion follows the form:

drop user <name>

No additional flags are required or supported.

Permission Management

RabbitMQ manages permission by associating a set of regular expressions indicating the configurable, readable and writable objects against a user. For more explanation on this topic, see the Access Control section of the rabbitmqctl(1) manual page.

Querying

Permissions can be queried by selecting from the "permissions" entity. For example,

BQL> select * from permissions
------------------------------------------------------
| username | configure_perm | write_perm | read_perm |
------------------------------------------------------
| guest    | .*             | .*         | .*        |

Granting

Permission grants follow the form:

grant [all|configure|read|write] on <regex> to <username>

An example of granting read access to a queue and write access to an exchange would be:

grant read on myQ to newuser;
grant write on myX to newuser;

Revoking

Permission revocation follows the form:

revoke [all|configure|read|write] from <username>

BQL Scripts

Along with the interactive console, the BQL command line utility also supports an optional first parameter of a script to execute. For example:

$ echo "create durable exchange durableX;" >/tmp/change.bql
$ cd /usr/lib/rabbitmq/lib/rabbitmq_server-1.6.0
$ su - rabbitmq -s /bin/sh -c priv/plugins/rabbitmq-bql/scripts/bql /tmp/change.bql
ok
$ su - rabbitmq -s /bin/sh -c priv/plugins/rabbitmq-bql/scripts/bql
BQL> select * from exchanges where name='durableX';
---------------------------------------------------------
| name     | type   | durable | auto_delete | arguments |
---------------------------------------------------------
| durableX | direct | true    | false       |           |

BQL>

Dumping Broker State

Alongside the bql command, the plugin also provides a utility called bql_dump. This utility will transform current broker state into a replayable BQL script that can be re-applied at a later date.

A sample invocation of this script might be along the lines of:

cd rabbitmq-bql-client
./bql_dump -host myrabbitserver

And would produce output like:

create durable exchange 'testexchange';
create durable queue 'testqueue';
create route from 'testexchange' to 'testqueue' where routing_key is 'key';

Managing Broker Change with BQL

Many typical AMQP applications will declare their own exchanges, queues and bindings upon startup. In these cases, an application upgrade will result in any necessary changes occurring automatically. However, in some deployments, it may be desirable for routing to be managed externally to the applications. For example, you may have an environment where you want to gradually increase the number of sources that an individual queue is fed by. You could either configure your application to support modifying it's bindings; or you could manage this routing in an external manner, similarly to the way that database layouts are maintained externally to the applications with SQL.

Utilising BQL, it is possible to perform the external management scenario in a controlled manner. Similarly to the way that SQL patch scripts are used in many organisational settings, BQL patch scripts can be distributed and version controlled. Test environments can be used to test the application of these scripts, and "reverse patch" scripts can be maintained allowing a failed application installation to be reverted.

Securing BQL Access

Security Principles

In general, BQL follows the general Rabbit security model. This includes managing access to queues and exchanges via regular expressions (as discussed in the access control documentation). However, given the ability for BQL to administer objects outside of the normal protocol remotely, there are a number of additional security concerns that need to be considered. BQL introduces a number of additional security rules - these are discussed in detail in the following section.

Rules

To execute a BQL query, write access is required to the default exchange in the default vhost. Preventing a user from writing to this exchange will prevent them from executing BQL queries.

Objects managed by BQL are separated into two categories. Protocol-manageable objects and non-protocol-manageable objects. As RabbitMQ already has a defined security model for objects that can be managed via the protocol, the BQL console simply follows standard security policy on these. For other objects that normally require tools outside of the AMQP protocol, additional rules have been defined. The categorisation of these object types is as follows:

Protocol Manageable Objects	Non-Protocol Manageable Objects
Exchange	VHost
Queue	User
Binding	Permission
	Connection

For any query operations on protocol manageable objects, the connected user must have wildcard (.*) read privileges on the owning vhost. The BQL client does not attempt to filter results, and instead provides an all-or-nothing restriction model.
For non protocol manageable objects, BQL introduces the idea of treating the default vhost (/) as an administrative vhost. Privileges to access internal broker objects are controlled via access to the administrative vhost.
For any query operations on non protocol manageable objects, wildcard read access is required on the administrative vhost. To update any of these objects, wildcard configure access is required on the administrative vhost.

Security Examples

This section provides examples on various configuration scenarios that can be setup for securing broker access. All examples assume that your broker starts in the initial state of having a single vhost, and admin/password as a "superuser". On a default installation of Rabbit, the superuser would be guest/guest.

Additional user with readonly access to the broker

$ ./bql -username admin -password password
BQL> create user foo identified by fooword
ok
BQL> grant read on '.*' to foo
ok
BQL> exit

$ ./bql -username foo -password fooword
BQL> select * from exchanges
-------------------------------------------------------------------
| name              | type    | durable | auto_delete | arguments |
-------------------------------------------------------------------
| amq.direct        | direct  | true    | false       |           |
| amq.topic         | topic   | true    | false       |           |
| amq.rabbitmq.log  | topic   | true    | false       |           |
| amq.fanout        | fanout  | true    | false       |           |
| amq.headers       | headers | true    | false       |           |
|                   | direct  | true    | false       |           |
| amq.match         | headers | true    | false       |           |

BQL> create user foo2 identified by foo2
{'EXIT',{amqp_error,access_refused,
                    "wildcard access to configure on vhost / refused for user 'foo'",
                    none}}
BQL> exit

Additional user with full BQL access only to a second vhost

$ ./bql -username admin -password password
BQL> create user foo2 identified by fooword2
ok
BQL> grant write on 'amq.default' to foo2
ok
BQL> create vhost '/subhost'
ok
BQL> exit

$ ./bql  -username admin -password password -vhost /subhost
BQL> grant all on '.*' to foo2
ok
BQL> exit

$ ./bql -username foo2 -password fooword2
BQL> select * from exchanges
{'EXIT',{amqp_error,access_refused,
                    "wildcard access to read on vhost / refused for user 'foo2'",
                    none}}
BQL> exit

$ ./bql -username foo2 -password fooword2 -vhost /subhost
BQL> select * from exchanges
-------------------------------------------------------------
| name        | type    | durable | auto_delete | arguments |
-------------------------------------------------------------
| amq.topic   | topic   | true    | false       |           |
| amq.fanout  | fanout  | true    | false       |           |
| amq.direct  | direct  | true    | false       |           |
| amq.headers | headers | true    | false       |           |
|             | direct  | true    | false       |           |
| amq.match   | headers | true    | false       |           |

BQL> select * from users
{'EXIT',{amqp_error,access_refused,
                    "wildcard access to read on vhost / refused for user 'foo2'",
                    none}}

BQL> create queue X
ok

Additional user with access only to second vhost and no BQL access

$ ./bql -username admin -password password
BQL> create user foo3 identified by fooword3
ok
BQL> create vhost '/subhost2'
ok
BQL> exit

$ ./bql  -username admin -password password -vhost /subhost2
BQL> grant all on '.*' to foo3
ok
BQL> exit

$ ./bql -username foo3 -password fooword3 -vhost /subhost2
{"init terminating in do_boot",{{badmatch,{error,{main_reader_died,socket_closed}}},
  [{amqp_connection,start_network_internal,2},{bql_client,connect,5},
   {bql_shell,start,0},{init,start_it,1},{init,start_em,1}]}}

$

Note that this final error is as a result of the BQL client being unable to connect to the administrative vhost. A more clear error is provided in the Rabbit server logs, along the lines of:

exception on TCP connection <0.758.0> from 127.0.0.1:49454
{channel0_error,opening,
                {amqp_error,access_refused,
                            "access to vhost '/' refused for user 'foo3'",
                            'connection.open'}}

Note that the difference between this scenario and the previous is the lack of write access to 'amq.default' in the '/' vhost. Enabling this would subsequently allow the user to use BQL.

Table of Contents

Installation

Server Configuration

Running the Client

Configuring the Client

Client Options

Option Examples

Available Commands

General Structure of Commands

Exchange Management

Querying

Creation

Deletion

Queue Management

Querying

Creation

Deletion

Route Management

Querying

Creation

Deletion

Basic Messaging

Sending Messages

Receiving Messages

VHost Management

Querying

Creation

Deletion

User Management

Querying

Creation

Deletion

Permission Management

Querying

Granting

Revoking

BQL Scripts

Dumping Broker State

Managing Broker Change with BQL

Securing BQL Access

Security Principles

Rules

Security Examples