一、docker 部署kafka单节点

1.1安装docker

可以参考这篇CentOS 7安装docker并配置镜像加速

1.2 运行zookeeper

docker run -d --name zookeeper -p 2181:2181 swr.cn-east-3.myhuaweicloud.com/srebro/middleware/zookeeper:3.4.13

1.3 运行kafka（注意修改zookeeper，kafka地址）

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=172.16.10.157:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 swr.cn-east-3.myhuaweicloud.com/srebro/middleware/kafka:2.13-2.8.1

1.4 登录kafka容器，创建topic

docker exec -it kafka bash
cd /opt/kafka_2.13-2.8.1/bin
./kafka-topics.sh --zookeeper 172.16.10.37:2181 --create --topic mytestTopic1 --replication-factor 1 --partitions 3

1.5 登录kafka容器，启动一个生产者

docker exec -it kafka bash
cd /opt/kafka_2.13-2.8.1/bin
./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytestTopic1

1.6 新开一个终端窗口，登录kafka容器，启动一个消费者

docker exec -it kafka bash
cd /opt/kafka_2.13-2.8.1/bin
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytestTopic1

1.7 模拟在生产者上，测试几条数据，观察消费者上有没有收到数据

image-20231115121440341

1.8 kafka map 管理工具

A beautiful, concise and powerful kafka web management tool. 一个美观简洁且强大的kafka web管理工具。

  • 官方帮助手册： https://github.com/dushixiang/kafka-map/blob/master/README-zh_CN.md
docker run -d \
    -p 8080:8080 \
    -v /opt/kafka-map/data:/usr/local/kafka-map/data \
    -e DEFAULT_USERNAME=admin \
    -e DEFAULT_PASSWORD=admin \
    --name kafka-map \
    --restart always dushixiang/kafka-map:latest

image-20231115122024832

image-20231115122106331

image-20231115122220324

二、docker-compose 部署kafka单节点

2.1 安装docker-compose

可以参考这篇Cenotos7 安装docker-compose

2.2 编排docker-compose文件

注意先要创建docker 单独的网络

创建自定义网络 ： docker network create -d bridge --subnet "192.168.10.0/24" --gateway "192.168.10.1" srebro.cn

version: '3'
services:
  zookeeper:
    image: swr.cn-east-3.myhuaweicloud.com/srebro/middleware/zookeeper:3.4.13
    networks:
      - srebro.cn
    volumes:
       - /home/application/Middleware/zookeeper-kafka/zookeeper/data:/data
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181
    restart: 'unless-stopped'
  
  kafka:
    image: swr.cn-east-3.myhuaweicloud.com/srebro/middleware/kafka:2.13-2.8.1
    container_name: kafka
    networks:
      - srebro.cn
    depends_on:
      - zookeeper
    ports: 
      - 9092:9092
    volumes:
      - /home/application/Middleware/zookeeper-kafka/kafka/data:/kafka
    environment:
      KAFKA_CREATE_TOPICS: "test"
      KAFKA_BROKER_NO: 0
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.16.10.157:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms16M"
    restart: 'unless-stopped'
  
  kafka-map: 
    image: dushixiang/kafka-map:latest
    networks:
      - srebro.cn
    volumes:
      - "./kafka-map/data:/usr/local/kafka-map/data"  
    ports:
      - 9006:8080
    environment:
      DEFAULT_USERNAME: admin
      DEFAULT_PASSWORD: 123456
    depends_on:
      - zookeeper
      - kafka
    restart: 'unless-stopped'


networks:
  srebro.cn:
    external: true

三、常见问题

3.1 如果zookeeper 容器无法正常运行，报错：

zookeeper  | library initialization failed - unable to allocate file descriptor table - out of memory/usr/bin/start-zk.sh: line 4:    12 Aborted                 (core dumped) /opt/zookeeper-3.4.13/bin/zkServer.sh start-foreground

无法分配文件描述符,一般是因为 操作系统层面和 docker 需要配置Limit

(1)、操作系统层面优化：

cat >> /etc/security/limits.conf << eof
root soft nofile 65535
root hard nofile 65535
root soft nproc 65535
root hard nproc 65535
root soft core unlimited
root hard core unlimited
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
* soft core unlimited
* hard core unlimited
eof


进入到/etc/security/limits.d/下
删除  *-nproc.conf
否则生效的为/etc/security/limits.d/下 文件的配置

cd /etc/security/limits.d/
rm -rf  *-nproc.conf

(2)、Docker层面优化：

vim /etc/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=65535
LimitNPROC=65535
LimitCORE=65535
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

四、参考

  • https://www.jianshu.com/p/dff7ac094c36
  • https://github.com/dushixiang/kafka-map/tree/master