Module: Kafka
Module:
Kafka module is currently available in Pigsty Pro as a Beta Preview.
Installation | Configuration | Administration | Playbook | Monitoring | Parameter
Overview
Kafka module is currently available in Pigsty Pro as a Beta Preview.
Installation
Kafka module is currently available in Pigsty Pro as a Beta Preview.
You can install Kafka and its Java dependencies on the specified node using the following command if you are using the open-source version of Pigsty.
./node.yml -t node_install -e '{"node_repo_modules":"infra","node_packages":["kafka"]}'
Kafka requires a Java runtime, so you need to install an available JDK when installing Kafka (OpenJDK 17 is used by default, but other JDKs and versions, such as 8 and 11, can also be used).
# EL7 (use jdk 11 instead)
./node.yml -t node_install -e '{"node_repo_modules":"node","node_packages":["java-11-openjdk-headless"]}'
# EL8 / EL9 (use OpenJDK 17)
./node.yml -t node_install -e '{"node_repo_modules":"node","node_packages":["java-17-openjdk-headless"]}'
# Debian / Ubuntu (use OpenJDK 17)
./node.yml -t node_install -e '{"node_repo_modules":"node","node_packages":["openjdk-17-jdk"]}'
Configuration
Single node Kafka configuration example, please note that in Pigsty single machine deployment mode. the Kafka Peer 9093 port is already occupied by AlertManager, it is recommended to use other ports, such as (9095).
kf-main:
hosts:
10.10.10.10: { kafka_seq: 1, kafka_role: controller }
vars:
kafka_cluster: kf-main
kafka_data: /data/kafka
kafka_peer_port: 9095 # 9093 is already hold by alertmanager
3-node Kraft mode Kafka cluster configuration example:
kf-test:
hosts:
10.10.10.11: { kafka_seq: 1, kafka_role: controller }
10.10.10.12: { kafka_seq: 2, kafka_role: controller }
10.10.10.13: { kafka_seq: 3, kafka_role: controller }
vars:
kafka_cluster: kf-test
Administration
Create Kafka clusters with kafka.yml playbook:
./kafka.yml -l kf-main
./kafka.yml -l kf-test
Create a topic named test:
kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
Here the --replication-factor 1 means each data will be replicated once, and --partitions 1 means only one partition will be created.
Use the following command to view the list of Topics in Kafka:
kafka-topics.sh --bootstrap-server localhost:9092 --list
Use the built-in Kafka producer to send messages to the test Topic:
kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
>haha
>xixi
>hoho
>hello
>world
> ^D
Use the built-in Kafka consumer to read messages from the test Topic:
kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092
Playbook
Pigsty provides 1 playbook related to the Kafka module, which are used to manage the Kafka cluster:
kafka.yml
The kafka.yml playbook for deploying Kafka KRaft mode cluster contains the following subtasks:
kafka-id : generate kafka instance identity
kafka_clean : remove existing kafka instance (DANGEROUS)
kafka_user : create os user kafka
kafka_pkg : install kafka rpm/deb packages
kafka_link : create symlink to /usr/kafka
kafka_path : add kafka bin path to /etc/profile.d
kafka_svc : install kafka systemd service
kafka_dir : create kafka data & conf dir
kafka_config : generate kafka config file
kafka_boot : bootstrap kafka cluster
kafka_launch : launch kafka service
kafka_exporter : launch kafka exporter
kafka_register : register kafka service to prometheus
Monitoring
Pigsty has provided two monitoring panels related to the KAFKA module:
KAFKA Overview shows the overall monitoring metrics of the Kafka cluster.
KAFKA Instance show the monitoring metrics details of a single Kafka instance.
Parameters
Available parameters for Kafka module:
#kafka_cluster: #CLUSTER # kafka cluster name, required identity parameter
#kafka_role: controller #INSTANCE # kafka role, controller, broker, or controller-only
#kafka_seq: 0 #INSTANCE # kafka instance seq number, required identity parameter
kafka_clean: false # cleanup kafka during init? false by default
kafka_data: /data/kafka # kafka data directory, `/data/kafka` by default
kafka_version: 3.8.0 # kafka version string
scala_version: 2.13 # kafka binary scala version
kafka_port: 9092 # kafka broker listen port
kafka_peer_port: 9093 # kafka broker peer listen port, 9093 by default (conflict with alertmanager)
kafka_exporter_port: 9308 # kafka exporter listen port, 9308 by default
kafka_parameters: # kafka parameters to be added to server.properties
num.network.threads: 3
num.io.threads: 8
socket.send.buffer.bytes: 102400
socket.receive.buffer.bytes: 102400
socket.request.max.bytes: 104857600
num.partitions: 1
num.recovery.threads.per.data.dir: 1
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.retention.hours: 168
log.segment.bytes: 1073741824
log.retention.check.interval.ms: 300000
#log.retention.bytes: 1073741824
#log.flush.interval.ms: 1000
#log.flush.interval.messages: 10000
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.