Kafka Cluster Troubleshooting

This skill helps diagnose and fix issues with Kafka brokers, ZooKeeper ensemble, and related services.

When to Use This Skill

• Kafka broker fails to start or crashes
• ZooKeeper connection issues
• Consumer/producer connectivity problems
• TLS/SSL certificate errors
• Cluster coordination failures
• JMX Exporter not exposing metrics
• Prometheus targets not UP
• Grafana dashboards blank or missing data
• Nginx 502/504 gateway errors
• Traffic generator not producing messages

Container Access

IMPORTANT: Use docker exec instead of molecule login for non-interactive troubleshooting.

molecule login requires an interactive shell which is not suitable for automated troubleshooting.

# Get container names
docker ps --format "table {{.Names}}\t{{.Status}}"

# Execute commands in containers
docker exec <container_name> <command>

Log File Locations

ZooKeeper Logs

# ZooKeeper logs (when installed via tarball)
docker exec zk1 cat /opt/zookeeper/logs/zookeeper-root-server-zk1.out
docker exec zk1 tail -50 /opt/zookeeper/logs/zookeeper-root-server-zk1.out

# ZooKeeper logs (systemd journal)
docker exec zk1 journalctl -u zookeeper -n 50 --no-pager

# Check ZooKeeper status
docker exec zk1 /opt/zookeeper/bin/zkServer.sh status

Kafka Broker Logs

# Kafka server logs (when installed via tarball)
docker exec kafka1 cat /opt/kafka/logs/server.log
docker exec kafka1 tail -100 /opt/kafka/logs/server.log

# Kafka logs (systemd journal)
docker exec kafka1 journalctl -u kafka -n 50 --no-pager

# Check if Kafka is responding
docker exec kafka1 /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

System Logs

# General system journal
docker exec kafka1 journalctl -xe --no-pager | tail -50

Common Diagnostic Commands

ZooKeeper Health

# Check ZooKeeper mode (leader/follower)
docker exec zk1 /opt/zookeeper/bin/zkServer.sh status

# Test ZooKeeper connectivity with 4-letter commands
docker exec zk1 bash -c 'echo ruok | nc localhost 2181'
docker exec zk1 bash -c 'echo stat | nc localhost 2181'

# List ZooKeeper znodes
docker exec zk1 /opt/zookeeper/bin/zkCli.sh -server localhost:2181 ls /

Kafka Health

# List topics
docker exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describe cluster
docker exec kafka1 /opt/kafka/bin/kafka-metadata.sh --snapshot /var/kafka-logs/__cluster_metadata-0/00000000000000000000.log --command "describe"

# Check broker API versions
docker exec kafka1 /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Check consumer groups
docker exec kafka1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Network/Port Verification

# Check listening ports
docker exec kafka1 ss -tlnp
docker exec zk1 ss -tlnp

# Test ZooKeeper port from Kafka broker
docker exec kafka1 bash -c 'nc -zv zk1 2181'

Process Status

# Check Java processes
docker exec kafka1 jps -l
docker exec zk1 jps -l

# Check systemd service status
docker exec kafka1 systemctl status kafka
docker exec zk1 systemctl status zookeeper

Common Issues and Solutions

Issue: ZooKeeper Won't Form Quorum

Symptoms: zkServer.sh status shows "Error contacting service"

Check:

# Verify myid file
docker exec zk1 cat /var/lib/zookeeper/myid

# Check zoo.cfg has correct server entries
docker exec zk1 cat /opt/zookeeper/conf/zoo.cfg

Issue: Kafka Cannot Connect to ZooKeeper

Symptoms: Kafka logs show "Connection refused" to ZooKeeper

Check:

# Test connectivity from Kafka to each ZK node
docker exec kafka1 bash -c 'for zk in zk1 zk2 zk3; do echo -n "$zk: "; nc -zv $zk 2181 2>&1; done'

# Verify zookeeper.connect in server.properties
docker exec kafka1 grep zookeeper.connect /opt/kafka/config/server.properties

Issue: TLS/SSL Certificate Errors

Symptoms: SSL handshake failures in logs

Check:

# Verify keystore exists
docker exec kafka1 ls -la /opt/kafka/config/*.jks

# Check certificate validity
docker exec kafka1 keytool -list -v -keystore /opt/kafka/config/kafka.keystore.jks -storepass <password> | grep -A2 "Valid from"

Observability Stack Troubleshooting

JMX Exporter Not Exposing Metrics

Symptoms: curl kafka1:7071/metrics times out or returns nothing

# Verify Java agent loaded in Kafka JVM args
docker exec kafka1 jps -v | grep jmx_prometheus

# Check Kafka service logs for JMX startup errors
docker exec kafka1 journalctl -u kafka -n 100 --no-pager | grep -i jmx

# Confirm JMX Exporter JAR exists
docker exec kafka1 ls -la /opt/jmx-exporter/

# Confirm config file exists
docker exec kafka1 cat /opt/jmx-exporter/kafka.yml

Fix: Ensure kafka_jmx_exporter_enabled: true in group_vars/all.yml. The JMX Exporter is a Java agent — the flag must be set before Kafka starts. Re-run molecule converge to redeploy.

---

Prometheus Targets Not UP

Symptoms: Prometheus UI shows targets in DOWN or UNKNOWN state

# Check Prometheus config
docker exec monitoring1 cat /etc/prometheus/prometheus.yml

# Manually test scrape reachability from monitoring1
docker exec monitoring1 curl -s http://kafka1:7071/metrics | head -5
docker exec monitoring1 curl -s http://zk1:7072/metrics | head -5
docker exec monitoring1 curl -s http://localhost:9308/metrics | head -5
docker exec monitoring1 curl -s http://kafka1:9100/metrics | head -5

# Check Prometheus service logs
docker exec monitoring1 journalctl -u prometheus -n 50 --no-pager

---

Grafana Dashboards Blank

Symptoms: Grafana loads but panels show "No data" or "Error"

# Verify datasource is configured
docker exec monitoring1 curl -s -u admin:admin http://localhost:3000/api/datasources

# Check Prometheus is actually reachable from Grafana (same host)
docker exec monitoring1 curl -s http://localhost:9090/api/v1/query?query=up

# Verify dashboard JSON files were copied
docker exec monitoring1 ls -la /var/lib/grafana/dashboards/

# Check provisioning config
docker exec monitoring1 cat /etc/grafana/provisioning/dashboards/kafka-dashboards.yml
docker exec monitoring1 cat /etc/grafana/provisioning/datasources/prometheus.yml

# Grafana logs
docker exec monitoring1 journalctl -u grafana-server -n 50 --no-pager

Common cause: Prometheus job names in dashboards must match scrape config. Check that Prometheus job kafka-jmx (not kafka_jmx) matches the {job=...} filter in dashboard queries.

---

Nginx Gateway Errors

Symptoms: 502 Bad Gateway or SSL errors

# Test nginx config
docker exec nginx1 nginx -t

# Check nginx error log
docker exec nginx1 cat /var/log/nginx/error.log | tail -30

# Verify TLS cert exists on nginx1
docker exec nginx1 ls -la /opt/kafka-ca/nginx1.crt /opt/kafka-ca/nginx1.key

# Test upstream connectivity from nginx1
docker exec nginx1 curl -s http://monitoring1:3000/api/health
docker exec nginx1 curl -s http://kafkaui1:8080/api/clusters

Fix: If cert is missing, check that nginx group was included in prepare.yml cert generation play.

---

Traffic Generator Issues

Symptoms: traffic-generator service fails or exits; no messages in Kafka topics

# Check service status and recent logs
docker exec traffic1 systemctl status traffic-generator
docker exec traffic1 journalctl -u traffic-generator -n 50 --no-pager

# Manually test kafka-python is installed
docker exec traffic1 python3 -c "import kafka; print(kafka.__version__)"

# Manually test Kafka connectivity from traffic1
docker exec traffic1 python3 -c "
from kafka import KafkaProducer
p = KafkaProducer(bootstrap_servers=['kafka1:9092'])
print('connected')
p.close()
"

# Check if topics were auto-created by producer
docker exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

---

Filtering Ansible Output

When running Molecule commands, always filter to avoid context window bloat:

# Show only failures and summary
molecule converge 2>&1 | grep -E "(FAILED|fatal:|ERROR|PLAY RECAP|Scenario)" -A 15

# Show more context around failures
molecule converge 2>&1 | grep -B 5 -A 25 "fatal:"