Configure RabbitMQ failover cluster

RabbitMQ is a universal bus to be used to exchange messages between the applications. There are other solutions like Kafka, AWS SQS, Google Pubsub, etc. If you are here then I suppose you already know what RabbitMQ is and I will not go deep into details.

This article describes the installation and configuration of the RabbitMQ in CentOS v7.

The key trick in RabbitMQ is in fact that there is no such thing as global cluster. The master role is being dedicated to one of the ervers within the scope of a single event in the queue. The outage of one server doesn’t cuse the downtime and you don’t need to care about doing something manually when it is back.

Nex’t I’ll provide step by step instructions how to configure failover cluster for rabbitMQ that consists of 3 servers.

Run on all servers

Enable EPEL

rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm

Install needed libraries:

yum install -y vim net-tools wget git zip unip openssh-server openssh logrotate socat

Install Erlang.

There are 3 options

• Install Eplang as is from Epel - I was getting errrors when one of the servers was falling out of cluster.сервера из кластера
• Download and install the latest Erlang from developers - RabbitMQ could not get installed with this one.
• Compile Erlang, that is optimized for Rabbit and includes only required components is a log way but it is the most reliable

You’ll need docker to build the Erlang rpm package. It is more efficient to build it locally and than copy to all the servers.

git clone https://github.com/rabbitmq/erlang-rpm.git  
сd erlang-rpm/docker/  
sed -i 's/-i\ -t/-tty/g' build-rpm-in-docker.sh  
bash build-image-and-rpm.sh latest

Upon completion the rpm files will be located in the following folder:

build-dir-latest/RPMS/x86_64/

Install Erlang

yum install -y build-dir-latest/RPMS/x86_64/erlang-19.2.0-1.el7.centos.x86_64.rpm  
yum install -y build-dir-latest/RPMS/x86_64/erlang-debuginfo-19.2.0-1.el7.centos.x86_64.rpm

Install RabbitMQ

The latest version of RabbitMQ as of today is v3.6.6.

wget -O /tmp/rabbitmq-server-3.6.6-1.el7.noarch.rpm https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm -no-check-certificate  
rpm -import http://www.rabbitmq.com/rabbitmq-release-signing-key.asc  
yum install -y /tmp/rabbitmq-server-3.6.6-1.el7.noarch.rpm

RabbitMQ has a nice web UI that can be very helpfull in every day usage. It is disabled by default. Run the following in order to enabpe it:

rabbitmq-plugins enable rabbitmq_management

RabbitMQ will prompt to be restarted and after it the UI will be available on the 15672 port. Use the server IP address to access it http://%server_ip_address%:15672/

Standard user:

• login: guest
• password: guest

On the first page you’ll see the state of the cluster (there is only one server in it so far)

The following can be used to get the data from the cli:

rabbitmqctl cluster_status

Sample output:

[[email protected] /]# rabbitmqctl cluster_status
Cluster status of node [email protected] ...
[{nodes,[{disc,[[email protected]]}]},
 {running_nodes,[[email protected]]},
 {cluster_name,<<"[email protected]">>},
 {partitions,[]},
 {alarms,[{[email protected],[]}]}]

Connect all servers into a cluster

First make sure that the value of the Erlang cookie is identical on all servers (you can copy it from the active server to all others) The value can be read at /var/lib/rabbitmq/.erlang.cookie

This file has to be owned by rabbitmq user and have 600 permissions on all servers

RabbitMQ daemon needs to be restarted after the changes

Next select any server to be the first one in cluster. You can use the current one. In my example it is [email protected]. Run the following on all servers in the cluster:

rabbitmqctl stop_app  
rabbitmqctl reset  
rabbitmqctl join_cluster [email protected]
rabbitmqctl start_app

After this the output of the cluster_status and web UI on all servers will desplay tthe list of all servers in your cluster You can use HA-Proxy to distribute the load between the servers i the cluster

The server-sde of the configuration is finished at this step

Examples to create users, hosts and queues

You can use cli to create users, hosts and message queues. Since all the commands are to be executed in bash you can create a set of variables that will repeat from server to server

user=username
password=password
vhost=name_of_the_vhost
queue_name=name_of_the_queue
exchange=the_name_of_exchange

Create vhost:

rabbitmqadmin declare vhost name=${vhost}

Create a user to access the vhost

rabbitmqadmin declare user name=${user} password=${password} tags=administrator  
rabbitmqadmin declare permission vhost=${vhost} user=${user} configure=`.\*` write=`.\*` read=`.*`

Create exchange buffer:

rabbitmqadmin -V ${vhost} -u ${user} -p ${password} declare exchange name=${exchange} type=fanout

Create message queue:

rabbitmqadmin -V ${vhost} -u ${user} -p ${password} declare queue name=${queue_name} durable=true

Create exchange buffer and a queue:

rabbitmqadmin -V ${vhost} -u ${user} -p ${password} declare binding source=${exchange} destination_type=queue destination=${queue_name}

Define HA policies for the cluster:

rabbitmqadmin -V ${vhost} -u ${user} -p ${password} declare policy name=ha_all pattern=`.*` definition='{`ha-mode`:`all`,`ha-sync-mode`:`automatic`}' apply-to=all

Usefull links:

• rabbitmqctl manual
• WEB api manual
• Use Ha-Proxy for RabbitMQ