Clustering

The RabbitMQ Helper is a tool that streamlines that streamlines the RabbitMQ clustering process on Windows. For instructions, see Clustering Guide and Quorum Queues.

For Quorum Queues, ensure you have at least three nodes, or your cluster cannot tolerate a node fault.

Be careful not to decommission a node from your cluster without first removing it from the member list of the Quorum Queues. It hosts replicas using the rabbitmq-queues shrink command. And if adding nodes after Quorum Queues has been configured in Secret Server, you must explicitly add the Quorum Queues to the node by using the rabbitmq-queues grow command. For more details, see RabbitMq Quorum Queue Management documentation.

The Helper does not assist with load-balancing. For details, see Load Balancing.

Clustering Prerequisites

When RabbitMQ is installed on a virtual or physical machine, it is already in a cluster of one node. To add another node to the cluster, you need to install RabbitMQ on a different virtual or physical machine and then join either of the nodes to the other cluster. The cluster name is irrelevant and can be changed later.

  • Ensure that you have at least two RabbitMQ nodes already installed following the appropriate installation process.
  • Ensure that all cluster nodes can resolve each others’ DNS names to the corerct IPs.
  • Install RabbitMQ on N+1 virtual or physical machines.

  • Check the availability of open firewall for cluster ports:

    • Make sure the firewall rule is open for the network type (private/domain only) on each virtual/physical machine where RabbitMQ nodes will be installed.
    • Required TCP ports: 4369, 25672,44002.
  • See RabbitMQ Clustering to find the value for the Erlang cookie of Rabbit Service on the node that you are trying to join.

  • On Erlang versions starting with 20.2, the cookie file locations are:

    • %HOMEDRIVE%\%HOMEPATH%\.erlang.cookie (usually C:\Users\%USERNAME%\erlang.cookie for user %USERNAME%’) if both the HOMEDRIVE and HOMEPATH environment variables are set.

    • %USERPROFILE%\.erlang.cookie, (usually C:\`Users\%USERNAME%\.erlang.cookie') if HOMEDRIVE and HOMEPATH are not set.

  • For the RabbitMQ Windows service: %USERPROFILE%\.erlang.cookie (usually C:\`WINDOWS\system32\config\systemprofile'). The cookie file used by the Windows service account and the user running CLI tools must be synchronized.

  • On Erlang versions prior to 20.2 (e.g. 19.3 or 20.1), the cookie file locations are: %HOMEDRIVE%%HOMEPATH%.erlang.cookie (usually C:\`Users%USERNAME%.\erlang.cookie for user %USERNAME%') if both the HOMEDRIVE and HOMEPATH environment variables are set,%USERPROFILE%.erlang.cookie (usually C:\`Users\%USERNAME%.\erlang.cookie') if HOMEDRIVE and HOMEPATH are not both set.

  • For the RabbitMQ Windows service: %WINDIR%\Users\%USER%.\erlang.cookie (usually C:Users\<UserName>.\erlang.cookie) The cookie file used by the Windows service account and the user running CLI tools must be synchronized.

Usually, the cookie location is the following: C:\WINDOWS\system32\config\systemprofile. That cookie value is the shared secret for the cluster, so all nodes joining the cluster must use this shared secret. By retrieving the value from the cookie in the above node's location, try to join and use it with the Set ErlangCookieFileCommand on another node. The other node will have its Erlang cookie set to match the first node. Clustering commands join the other node to the first node and then bring the first node into the mirror.

Joining a Cluster Using the RabbitMQ Helper

To join a cluster using the RabbitMQ Helper, run the commands listed below.

  1. Set-ErlangCookieFileCommand—updates the Erlang Cookie file with values given by the user. If two or more nodes want to communicate, they must have the same shared secret called the Erlang cookie. For details, see the Prerequisites section above.

    You can update the existing erlang cookie file values to the nodes that need to be joined for clustering. Alternatively, use the same command with an arbitrary cookie on all nodes (if you want to use a new value or cannot look the existing one up).

  2. Join-RabbitMqCluster—joins the other node in a cluster.

Joining a Cluster Powershell Example

Copy 
#on the node which you are joining into the target node/cluster
#Use the value for cookie from the target node, collected in the prerequisite steps.
# <cluster-cookie-value> can be found iside the Erlang Cookie File
Set-ErlangCookieFileCommand -CookieContent cluster-cookie-value> -Verbose
#using the CookieSet and FirewallConfigured will prevent the helper for prompting. Only use if you have actually already set the cluster cookie and you have configured your firewall
#OTHERHOSTNAME is CASE SENSITIVE and is the exact hostname of a machine that you are trying to join this machine into a cluster with.
Join-RabbitMqCluster -StrictHostname OTHERHOSTNAME -CookieSet - FirewallConfigured -Verbose

Clustering a Rabbit Node Without the Helper

Prerequisites

  • Ensure that RabbitMQ is installed and running on each machine you want to include in the cluster

  • Each node must have a unique name. By default, it is rabbit@hostname. You can change this value using the RABBITMQ_NODENAME environment variable

To cluster a Rabbit node in an alternative way:

  1. Copy the .erlang.cookie from one node to all nodes in the system profile (usually C:\Windows\System32\config\systemprofile) and the user profile running rabbitmqctl.bat.

  2. Restart the RabbitMQ nodes.

  3. Go to the RabbitMQ Bin Directory and run the following reset commands:

    1. - rabbitmqctl stop_app

    2. - rabbitmqctl reset

    An example of the RabbitMQ Bin Directory: - C:\Program Files\RabbitMQ Server\rabbitmq_server-xx.xx.xx\sbin.

  4. Join the cluster by running the - rabbitmqctl join_cluster rabbit@OTHERHOSTNAME command.

  5. Start the RabbitMQ nodes by running the - rabbitmqct start_app command.

  6. Verify the cluster by running the rabbitmqctl cluster_statuscommand.

Establishing Cluster Policy

Clusters do not mirror queues unless configured to do so via policies. The Helper creates unbalanced policies by default. Balanced policies require the execution of the Set-MinMasterToClusterPolicy command, which distributes queues to all nodes in the cluster.

Non-Balanced policy

Even though the node is in a two-node cluster, all queues have it as primary. Therefore, the other nodes are not helping with the load even though they may have mirrors of those queues.

Balanced policy (Min-Master)

The queues are distributed inside the cluster because the node is in a two-node cluster. Almost half use one node as their primary and vice versa.

Creating a New Cluster Policy Using the Helper

To create a new cluster policy, run the New-RabbitMqClusterPolicy -Name <policy_name> -Pattern <pattern_string> command in PowerShell command prompt.

The command consists of the following elements:

  • New-RabbitMqClusterPolicy argument—creates an unbalanced cluster policy

  • ‘-Name’— argument sets the policy name. It must be unique

  • The ‘-Pattern’ argument sets a pattern to match at the beginning of the queue name. If the pattern is matched, the queue is replicated. For example, “^ActiveNonSslRabbitMq:” would match a queue that begins with ‘ActiveNonSslRabbitMq:’ and would replicate that queue across the cluster

An example of setting up a cluster policy in the Powershell prompt:

$admincred = Get-Credential -Message “Enter the administrative user RabbitMq user username and password”;

New-RabbitMqClusterPolicy -Name cluster-test-all -Pattern “^ActiveNonSslRabbitMq:” -AdminCredential $admincred.

You can create a policy with a custom sync batch size. The default is 400 for New-RabbitMqClusterPolicy because Delinea products have a worst-case scenario for messages to be 256KB. When a sync message is generated, 256* 400 = 100MB. A larger sync message can cause fragmentation if there is latency or network connection drops between cluster nodes.

An example of a synch batch size policy:

New-RabbitMqClusterPolicy -Name cluster-test-all -Pattern “^ActiveNonSslRabbitMq:” -AdminCredential $admincred -SyncBatchSize 100.

For a policy with a custom replica count batch size, the default value for New-RabbitMqClusterPolicy is two because anything higher strains the cluster. Two replicas mean one master and one mirror.

An example of a replica count batch size policy:

New-RabbitMqClusterPolicy -Name cluster-test-all -Pattern “^ActiveNonSslRabbitMq:” -AdminCredential $admincred -QueueReplicaCount 3.

For a policy with automatic sync mode, the default value for New-RabbitMqClusterPolicy is manual to avoid forcing a queue to synchronize when a new mirror joins automatically.

An example of an automatic sync mode policy:

New-RabbitMqClusterPolicy -Name cluster-test-all -Pattern “^ActiveNonSslRabbitMq:” -AdminCredential $admincred -AutomaticSyncMode.

Establishing Policy without the Helper

The mentioned policy is not ideal and is not balanced. Use the RabbitMQ Helper if possible.

An example of a ctl command:

rabbitmqctl set_policy cluster-test-all "\^cluster\-test:" "{""ha-mode"":""all""}".

Commands:

  • rabbitmqctl—the base command to interact with the RabbitMQ server

  • set_policy—this subcommand defines a new policy for managing queues

Policy Definition Commands:

  • cluster-test-all—this is the chosen name for the policy. It helps identify its purpose

  • "\^cluster\-test:"—defines the pattern that the policy applies to.

Command elements description:

  • ^—matches the beginning of the queue name

  • cluster-test—matches the literal string "cluster-test"

  • ::—matches a colon, potentially separating a category from the queue name (optional here)

  • .*—matches any character sequence zero or more times (to capture anything after "cluster-test:")

  • '{"ha-mode"":""all""}'—defines the policy configuration as a JSON string enclosed in single quotes.

Command elements description:

  • "ha-mode"—this property specifies the High Availability (HA) mode for the queues

  • "all"—this value sets the HA mode to "all," which means the queue will be mirrored to all nodes in the RabbitMQ cluster

Leaving a Cluster

The Helper uses reset to leave the clusters. This removes the node from any cluster it belongs to, all data from the management database (such as configured users and hosts), and all persistent messages.

Removing a Node from a RabbitMQ Cluster Using the Helper

To reset the RabbitMQ node to the initial condition, run the Reset-RabbitMqNodeCommand.