Skip to content

RabbitMQ

Tutorial: Deploying RabbitMQ with Docker Compose

Step 1: Install Docker and Docker Compose

Make sure Docker and Docker Compose are installed on your machine. You can download and install them from the official Docker site: Docker and Docker Compose.

Step 2: Create a Docker Compose File

Create a file named docker-compose.yml and paste the provided Docker Compose configuration.

version: '3'

services:

  rabbitmq:
    image: rabbitmq:3.12-management-alpine
    container_name: 'rabbitmq_api_client'
    volumes:
      - ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
      - RABBITMQ_CONFIG_FILE=/etc/rabbitmq/rabbitmq.conf
    ports:
      - 5672:5672
      - 15672:15672

Understanding the Docker Compose Configuration for RabbitMQ

The provided Docker Compose configuration creates a Docker container named rabbitmq_api_client using the Docker image rabbitmq:3.12-management-alpine. The container is configured to use port 5672 for AMQP connections and port 15672 for the RabbitMQ management interface. The container is also configured to use a custom RabbitMQ configuration file.

Some details about the configuration:

  • Volume: Mounts a volume, linking the local ./rabbitmq/rabbitmq.conf file to the /etc/rabbitmq/rabbitmq.conf file in the container. This allows copying the custom RabbitMQ configuration file into the container.
  • RABBITMQ_DEFAULT_USER: Sets the default username for RabbitMQ to connect to the RabbitMQ management interface.
  • RABBITMQ_DEFAULT_PASS: Sets the default password for RabbitMQ to connect to the RabbitMQ management interface.
  • RABBITMQ_CONFIG_FILE: Sets the path of the custom RabbitMQ configuration file within the container.

Step 3: Create a RabbitMQ Configuration File

Create a directory named rabbitmq in the same location as your docker-compose.yml file. Inside the rabbitmq directory, create a file named rabbitmq.conf. This file can be used to customize RabbitMQ settings.

Step 4: Configure RabbitMQ with a Custom Configuration File (Optional)

Open the rabbitmq.conf file on your machine and add custom configurations if needed. For example:

# rabbitmq.conf
vm_memory_high_watermark.absolute = 2GB

This sets RabbitMQ's high memory watermark limit to 2 GB. If you wish to add more configuration parameters, you can find the complete list of configuration parameters in the official RabbitMQ documentation: Configuration.

Step 5: Deploy RabbitMQ with Docker Compose

Open a terminal, navigate to the directory containing your docker-compose.yml file, and execute the following command:

docker-compose up

This command will download the RabbitMQ Docker image, create a Docker container named rabbitmq_api_client based on the provided configuration, and start the RabbitMQ server in the background.

Upon launch, you should see in the logs whether your configuration file has been recognized:

rabbitmq      |   Config file(s): /etc/rabbitmq/rabbitmq.conf
rabbitmq      |                   /etc/rabbitmq/conf.d/10-defaults.conf
rabbitmq      | 
rabbitmq      |   Starting broker...2023-11-29 07:36:10.216117+00:00 [info] <0.230.0> 
rabbitmq      |  node           : rabbit@b0b0b0b0b0b0
rabbitmq      |  home dir       : /var/lib/rabbitmq
rabbitmq      |  config file(s) : /etc/rabbitmq/rabbitmq.conf

Step 6: Access the RabbitMQ Management Interface

Open your web browser and go to http://localhost:15672/. Log in with the username user and password password as specified in the Docker Compose file.

Step 7: Connect to RabbitMQ via AMQP

You can now connect to RabbitMQ using the AMQP protocol on localhost:5672 with the credentials provided in the Docker Compose file.

Congratulations! You have successfully deployed RabbitMQ using Docker Compose. You can further customize the configuration by modifying the docker-compose.yml and rabbitmq.conf files according to your specific needs.

How to Save RabbitMQ Messages in PostgreSQL with Python?

RabbitMQ, a powerful means of communication between applications and connected objects, facilitates the efficient transfer of messages, but it does not provide a built-in mechanism for long-term storage. To overcome this limitation, integration with a robust database such as PostgreSQL is essential.

For instance, when leveraging AMQP or MQTT protocols for data transmission from connected objects, such as monitoring room temperature or collecting energy consumption from a device, there is often a pressing need to retain this information for future analysis. RabbitMQ does not inherently provide this data persistence feature, but by intelligently synchronizing your message flow with PostgreSQL, you can create a comprehensive solution that enables not only real-time communication but also durable data storage.

In this tutorial, we will show you how to save RabbitMQ messages in PostgreSQL using Python.

Prerequisites for Saving RabbitMQ Messages in PostgreSQL

To follow this tutorial, you need the following:

  • a RabbitMQ server
  • a PostgreSQL server

Launching RabbitMQ Server and PostgreSQL Server

In our example, we will use Docker to launch RabbitMQ and PostgreSQL. Below is the docker-compose.yml file that defines the RabbitMQ and PostgreSQL services.

version: "3"

services:
  rabbitmq:
    image: rabbitmq:3.12-management-alpine
    container_name: rabbit_mq_to_db
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
      - RABBITMQ_DEFAULT_VHOST=default_vhost
    ports:
      - 5672:5672
      - 15672:15672

  postgres:
    image: postgres:16-alpine
    container_name: postgres_db
    hostname: postgres
    restart: always
    environment:
      - POSTGRES_DB=postgres_db
      - POSTGRES_USER=postgres_user
      - POSTGRES_PASSWORD=password
    volumes:
      - ./docker-entrypoint-initdb.d/:/docker-entrypoint-initdb.d
    ports:
      - "5432:5432"

Description of the docker-compose.yml file:

This Docker Compose file defines two services, RabbitMQ and PostgreSQL, along with their respective configurations to create and orchestrate Docker containers that interact with each other. Here is a detailed explanation of the content:

RabbitMQ Service:

  • rabbitmq: Service name.
  • image: Uses RabbitMQ version 3.12 image with the management plugin enabled, allowing access to the RabbitMQ admin interface at http://localhost:15672. Connect using credentials user and password.
  • container_name: Name of the container created for this service.
  • environment: Defines environment variables for RabbitMQ, including username, password, and virtual host (vhost).
  • ports: Maps ports 5672 and 15672 of the RabbitMQ container to the same ports on the host.

PostgreSQL Service:

  • postgres: Service name.
  • image: Uses PostgreSQL version 16 image with Alpine Linux.
  • container_name: Name of the container created for this service.
  • hostname: Hostname of the PostgreSQL container.
  • restart: Container's automatic restart policy (set to "always").
  • environment: Defines environment variables for PostgreSQL, including the database name, username, and password.
  • volumes: Mounts the local directory ./docker-entrypoint-initdb.d/ into the PostgreSQL container's database initialization directory, allowing execution of SQL initialization scripts.
  • ports: Maps port 5432 of the PostgreSQL container to the same port on the host.

To launch RabbitMQ and PostgreSQL, execute the following command:

docker-compose up

Create Python Scripts to Save RabbitMQ Messages in PostgreSQL

Now that RabbitMQ and PostgreSQL are launched, we will create the Python scripts. We will need two scripts:

  • A RabbitMQ publisher to send messages with Python
  • A RabbitMQ consumer to receive messages with Python and save them in PostgreSQL

We will also need the pika library, which is a Python implementation of the AMQP 0-9-1 protocol to connect to RabbitMQ.

Finally, we will need the psycopg2 library to connect to PostgreSQL and interact with the database.

RabbitMQ Publisher to Send Messages with Python

The publisher is a Python script that connects to RabbitMQ and sends messages to a queue.

To connect to RabbitMQ, we will use the pika library. To install it, execute the following command:

pip install pika

Following code is the publisher:

import random

import pika
from time import sleep

# Connection to RabbitMQ
url = 'amqp://user:password@localhost:5672/default_vhost'
params = pika.URLParameters(url)
connection = pika.BlockingConnection(params)
channel = connection.channel()

# Creation of the 'temperature' queue
channel.queue_declare('temperature')

# Creation of the 'temperature_routing_key' route that links the 'temperature' queue to the 'amq.direct' exchange
channel.queue_bind('temperature', 'amq.direct', 'temperature_routing_key')
while True:
    sleep(3)
    # Send a message to the 'temperature' queue
    channel.basic_publish('amq.direct', 'temperature_routing_key', body=str(random.uniform(0, 100)))

The RabbitMQ connection URL is amqp://user:password@localhost:5672/default_vhost where:

  • user is the username
  • password is the password
  • localhost is the RabbitMQ server's IP address
  • 5672 is the RabbitMQ port
  • default_vhost is the RabbitMQ vhost (virtual host)

RabbitMQ Consumer to Receive Messages with Python and Save them in PostgreSQL

The consumer is a Python script that connects to RabbitMQ, reads messages from a queue, and saves them in PostgreSQL.

Here is a simple example of a consumer that reads messages from the temperature queue and saves them in the temperature table in PostgreSQL:

import pika
import psycopg2

# Connexion to PostgreSQL database
connection_sql = psycopg2.connect(database="postgres_db", user="postgres_user", password="password", host="localhost", port="5432")
cursor = connection_sql.cursor()

# Definition of the callback function that will be called when a message is received in the 'temperature' queue
def callback(ch, method, properties, body):
    # Conversion of the message to string
    body = body.decode()
    # Insertion of the message in the 'temperature' table
    cursor.execute("INSERT INTO temperature (value) VALUES (%s)", (body,))
    connection_sql.commit()

# Connection to RabbitMQ
url = 'amqp://user:password@localhost:5672/default_vhost'
params = pika.URLParameters(url)
connection = pika.BlockingConnection(params)
channel = connection.channel()
channel.basic_consume('temperature', callback, auto_ack=True)
channel.start_consuming()
channel.close()
connection.close()

Complete Python Source Code

You can find the complete source code for this tutorial on Github.

It is slightly different from what is presented in this tutorial as it includes the file for creating the temperature table in PostgreSQL.

It uses classes to connect to PostgreSQL and for the RabbitMQ consumer.

Tutorial on RabbitMQClient: Python Client for RabbitMQ

Introduction

RabbitMQ is a powerful message broker that facilitates communication between distributed systems. Its robust and flexible architecture makes it a popular choice for implementing reliable messaging patterns. In this tutorial, we will explore the capabilities of RabbitMQClient, a Python library designed to interact with RabbitMQ servers through its API.

Why RabbitMQ?

Before diving into RabbitMQClient, let's briefly understand why RabbitMQ is a crucial component in modern software architectures. RabbitMQ enables asynchronous communication between different components of a system, providing a scalable and decoupled approach to message handling. This results in improved system reliability, performance, and maintainability.

What is RabbitMQClient?

RabbitMQClient is a Python library that simplifies interaction with RabbitMQ servers by providing a convenient API. It abstracts the complexities of direct communication with RabbitMQ's HTTP API, allowing developers to manage vhosts, queues, exchanges, and more using Pythonic syntax.

Throughout this tutorial, we will cover the installation and configuration of RabbitMQClient, explore its various features, and learn how to perform common operations on RabbitMQ servers. Whether you are a beginner or an experienced developer, this tutorial will guide you through the fundamentals of RabbitMQClient, enabling you to incorporate it into your messaging infrastructure.

Now, let's proceed to the installation and configuration of RabbitMQClient.

Installation

To get started with RabbitMQClient, follow these steps to install the library and set up your Python environment.

1. Install RabbitMQClient

You can install RabbitMQClient using the Python package manager, pip. Open your terminal or command prompt and run the following command:

pip install rabbitmq-api-client

2. Verify the Installation

After installation, you can verify that RabbitMQClient is installed correctly by running the following command:

pip show rabbitmq-api-client

This command should display information about the installed package, confirming a successful installation.

3. Import RabbitMQClient into Your Python Code

You can now start using RabbitMQClient in your Python scripts or projects. Import it at the beginning of your Python code:

from rabbitmq_api_client import RabbitMQClient

4. Provide RabbitMQ Server Details

Create an instance of the RabbitMQClient class by providing the base URL of your RabbitMQ server and authentication details (username and password):

from rabbitmq_api_client import RabbitMQClient

rabbitmq_client = RabbitMQClient(base_url="http://votre-serveur-rabbitmq", 
                                 username="votre-nom-utilisateur", 
                                 password="votre-mot-de-passe")

With these steps, you have successfully installed RabbitMQClient and are ready to explore its features. In the following sections, we will delve into the configuration of RabbitMQClient and its usage to interact with RabbitMQ servers.

Example of Using RabbitMQClient to Create a Queue

from rabbitmq_api_client import RabbitMQClient
from rabbitmq_api_client.schemas import CreateQueue

rabbitmq_client = RabbitMQClient(base_url="http://votre-serveur-rabbitmq", 
                                 username="votre-nom-utilisateur", 
                                 password="votre-mot-de-passe")

queue_schema = CreateQueue(name="queue_name", auto_delete=False, durable=True) 
rabbitmq_client.create_queue(vhost="votre-vhost", 
                             queue=queue_schema)

In this example, the queue will not be deleted when the last consumer disconnects because the auto_delete parameter is set to False. Also, the queue and messages will not be deleted when the RabbitMQ server restarts because the durable parameter is set to True.