Deploy Apache Kafka using Ansible-playbook on Ubuntu 22.04 LTS

Introduction

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation. It is designed to handle real-time data feeds and provides a distributed, fault-tolerant, and scalable publish-subscribe messaging system.

Prerequisites

• Up and running ubuntu 22.04 LTS machine.
• Basic knowledge in Linux commands.
• Internet connectivity.
• Ansible should be installed.

We can use the given link to install Ansible on Ubuntu 22.04 LTS Linux machine.

In this post, We will show you how to deploy Apache Kafka using Ansible-play on ubuntu 22.04 LTS Linux machine.

Step 1: Create an Inventory File

Create an inventory file (e.g., inventory.ini) with the target server’s IP address:

[kafka]
your_kafka_server_ip ansible_ssh_user=your_ssh_user

[zookeeper]
your_zookeeper_server_ip ansible_ssh_user=your_ssh_user

Replace your_kafka_server_ip, your_zookeeper_server_ip, and your_ssh_user with the appropriate values.

Step 2: Create the Ansible Playbook

Deploying Apache Kafka using Ansible involves several steps, including the installation of Java, setting up Zookeeper (which is often used with Kafka for coordination), and deploying Kafka itself. Below is an example Ansible playbook to deploy Apache Kafka on Ubuntu 22.04 LTS.

Create a playbook file (e.g., deploy_kafka.yml):

sudo nano deploy_kafka.yml

Paste the following ansible tasks.

---
- name: Deploy Apache Kafka on Ubuntu 22.04
  hosts: kafka
  become: true
  vars:
    kafka_version: "2.8.0"  # Adjust to the desired Kafka version

  tasks:
    - name: Install Java
      apt:
        name: openjdk-11-jdk
        state: present

    - name: Download and extract Kafka
      get_url:
        url: "https://downloads.apache.org/kafka/{{ kafka_version }}/kafka_{{ kafka_version }}.tgz"
        dest: "/opt/"
      become: yes

    - name: Extract Kafka archive
      ansible.builtin.unarchive:
        src: "/opt/kafka_{{ kafka_version }}.tgz"
        dest: "/opt/"
        remote_src: yes
      become: yes

    - name: Create a symbolic link to Kafka installation
      ansible.builtin.file:
        src: "/opt/kafka_{{ kafka_version }}"
        dest: "/opt/kafka"
        state: link
      become: yes

    - name: Configure server.properties
      template:
        src: server.properties.j2
        dest: "/opt/kafka/config/server.properties"
      become: yes

    - name: Start Zookeeper
      ansible.builtin.command:
        cmd: "/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties"
      become: yes

    - name: Start Kafka Server
      ansible.builtin.command:
        cmd: "/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties"
      become: yes

Step 3: Create the Kafka Configuration Template

Create a template file for Kafka configuration (server.properties.j2):

# /opt/kafka/config/server.properties

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

# The port the socket server listens on
listeners=PLAINTEXT://:9092

# Log Basics
log.dirs=/tmp/kafka-logs

This is a minimal configuration. Adjust it according to your needs.

Step 4: Run the Ansible Playbook

Run the playbook using the following command:

• Installs Java.
• Downloads and extracts Kafka.
• Creates a symbolic link to the Kafka installation.
• Configures the server.properties file.
• Starts Zookeeper and Kafka servers.

Remember to customize the playbook and configuration according to your specific requirements and security considerations.

Conclusion

We have successfully deployed Apache Kafka using Ansible playbook on ubuntu 22.04 LTS machine, If you still have questions, please post them in the comments section below.