docker搭建kafka集群的方法实现

目录

一、原生Docker命令二、镜像选择三、集群规划四、Zookeeper集群安装五、Kafka集群安装

一、原生Docker命令

1. 删除所有dangling数据卷（即无用的Volume，僵尸文件）

1. docker volume rm $(docker volume ls -qf dangling=true)

复制代码 2. 删除所有dangling镜像（即无tag的镜像）

1. docker rmi $(docker images | grep “^<none>” | awk “{print $3}”

复制代码 3. 删除所有关闭的容器

1. docker ps -a | grep Exit | cut -d ‘ ‘ -f 1 | xargs docker rm

复制代码
二、镜像选择

环境为M1版本的mbp:

Zookeeper采用zookeeperKafka采用wurstmeister/kafkaKafka-Manager采用scjtqs/kafka-managerMysql采用mysql/mysql-server

三、集群规划

1. 新建docker网络

1. docker network create docker-net –subnet 172.20.10.0/16

复制代码

1. docker network ls

复制代码 2. 集群规划

四、Zookeeper集群安装

1. 新建文件zk-docker-compose.yml

1. version: ‘3.4’
3. services:
4.   zook1:
5.     image: zookeeper:latest
6.     restart: always
7.     hostname: zook1
8.     container_name: zook1 #容器名称，方便在rancher中显示有意义的名称
9.     ports:
10.     – 2183:2181 #将本容器的zookeeper默认端口号映X去
11.     volumes: # 挂载数据卷
12.     – “/Users/konsy/Development/volume/zkcluster/zook1/data:/data”
13.     – “/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog”
14.     – “/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs”
15.     environment:
16.         ZOO_MY_ID: 1  #即是zookeeper的节点值，也是kafka的brokerid值
17.         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
18.     networks:
19.         docker-net:
20.             ipv4_address: 172.20.10.11
22.   zook2:   
23.     image: zookeeper:latest
24.     restart: always
25.     hostname: zook2
26.     container_name: zook2 #容器名称，方便在rancher中显示有意义的名称
27.     ports:
28.     – 2184:2181 #将本容器的zookeeper默认端口号映X去
29.     volumes:
30.     – “/Users/konsy/Development/volume/zkcluster/zook2/data:/data”
31.     – “/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog”
32.     – “/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs”
33.     environment:
34.         ZOO_MY_ID: 2  #即是zookeeper的节点值，也是kafka的brokerid值
35.         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
36.     networks:
37.         docker-net:
38.             ipv4_address: 172.20.10.12
39.             
40.   zook3:   
41.     image: zookeeper:latest
42.     restart: always
43.     hostname: zook3
44.     container_name: zook3 #容器名称，方便在rancher中显示有意义的名称
45.     ports:
46.     – 2185:2181 #将本容器的zookeeper默认端口号映X去
47.     volumes:
48.     – “/Users/konsy/Development/volume/zkcluster/zook3/data:/data”
49.     – “/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog”
50.     – “/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs”
51.     environment:
52.         ZOO_MY_ID: 3  #即是zookeeper的节点值，也是kafka的brokerid值
53.         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
54.     networks:
55.         docker-net:
56.             ipv4_address: 172.20.10.13
57. networks:
58.   docker-net:
59.     external:
60.       name: docker-net

复制代码 2. 执行脚本部署zookeeper至Docker:

1. docker compose -f ./zk-docker-compose.yml up -d

复制代码
五、Kafka集群安装

1. 新建文件kafka-docker-compose.yml

1. version: ‘2’
3. services:
4.   kafka1:
5.     image: docker.io/wurstmeister/kafka
6.     restart: always
7.     hostname: kafka1
8.     container_name: kafka1
9.     ports:
10.       – 9093:9093
11.       – 9193:9193
12.     environment:
13.       KAFKA_BROKER_ID: 1
14.       KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193
15.       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
16.       SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193
17.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
18.       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
19.       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
20.       ALLOW_PLAINTEXT_LISTENER : ‘yes’
21.       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
22.     volumes:
23.       – /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
24.       – /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
25.     external_links:
26.       – zook1
27.       – zook2
28.       – zook3
29.     networks:
30.       docker-net:
31.         ipv4_address: 172.20.10.14
33.   kafka2:
34.     image: docker.io/wurstmeister/kafka
35.     restart: always
36.     hostname: kafka2
37.     container_name: kafka2
38.     ports:
39.       – 9094:9094
40.       – 9194:9194
41.     environment:
42.       KAFKA_BROKER_ID: 2
43.       KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194
44.       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
45.       KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194
46.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
47.       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
48.       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
49.       ALLOW_PLAINTEXT_LISTENER : ‘yes’
50.       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
51.     volumes:
52.       – /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka
53.       – /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka
54.     external_links:
55.       – zook1
56.       – zook2
57.       – zook3
58.     networks:
59.       docker-net:
60.         ipv4_address: 172.20.10.15
62.   kafka3:
63.     image: docker.io/wurstmeister/kafka
64.     restart: always
65.     hostname: kafka3
66.     container_name: kafka3
67.     ports:
68.       – 9095:9095
69.       – 9195:9195
70.     environment:
71.       KAFKA_BROKER_ID: 3
72.       KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195
73.       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
74.       KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195
75.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
76.       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
77.       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
78.       ALLOW_PLAINTEXT_LISTENER : ‘yes’
79.       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据
80.     volumes:
81.       – /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka
82.       – /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka
83.     external_links:
84.       – zook1
85.       – zook2
86.       – zook3
87.     networks:
88.       docker-net:
89.         ipv4_address: 172.20.10.16
90. networks:
91.   docker-net:
92.     external:
93.       name: docker-net

复制代码 2. 执行脚本部署kafka至Docker:

1. docker compose -f ./kafka-docker-compose.yml up -d

复制代码 3. listeners 和 advertised.listeners

listeners: 学名叫监听器，其实就是告诉外部连接者要通过什么协议访问指定主机名和端口开放的 Kafka 服务。advertised.listeners：和 listeners 相比多了个 advertised。Advertised 的含义表示宣称的、公布的，就是说这组监听器是 Broker 用于对外发布的。

比如说：

1.    listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094
2.    advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公网 ip>:端口
3.    kafka_listener_security_protocol_map: “INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT”
4.    kafka_inter_broker_listener_name: “INSIDE”

复制代码 advertised_listeners 监听器会注册在 zookeeper 中；

当我们对 172.17.0.10:9092 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 INSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口；

同理，当我们对 <公网 ip>:端口 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 OUTSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口 172.17.0.10:9094；

总结：advertised_listeners 是对外暴露的服务端口，真正建立连接用的是 listeners。

4.安装kafka-manager

4.1 新建文件kafka-manager-docker-compose.yml

1. version: ‘2’
3. services:
4.   kafka-manager:
5.     image: scjtqs/kafka-manager:latest
6.     restart: always
7.     hostname: kafka-manager
8.     container_name: kafka-manager
9.     ports:
10.       – 9000:9000
11.     external_links:  # 连接本compose文件以外的container
12.       – zook1
13.       – zook2
14.       – zook3
15.       – kafka1
16.       – kafka2
17.       – kafka3
18.     environment:
19.       ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181
20.       KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095
21.       APPLICATION_SECRET: letmein
22.       KM_ARGS: -Djava.net.preferIPv4Stack=true
23.     networks:
24.       docker-net:
25.         ipv4_address: 172.20.10.10
26. networks:
27.   docker-net:
28.     external:
29.       name: docker-net

复制代码 4.2 执行脚本部署kafka-manager至Docker:

1. docker compose -f ./kafka-manager-docker-compose.yml up -d

复制代码 4.3 配置Cluster

5. 测试kafka

5.1连接容器

进入kafka容器的命令行：

1. docker exec -ti kafka /bin/bash

复制代码 进入kafka所在目录：

1. cd opt/kafka_2.13-2.8.1/

复制代码 5.2 创建Replication为2，Partition为2的topic

1. bin/kafka-topics.sh –create –zookeeper zook1:2181 –replication-factor 2 –partitions 2 –topic partopic

复制代码 5.3 查看topic的状态

在kafka容器中的 opt/kafka_2.12-1.1.0/ 目录下输入

1. bin/kafka-topics.sh –describe –zookeeper zook1:2181 –topic partopic

复制代码 到此这篇关于docker搭建kafka集群的方法实现的文章就介绍到这了,更多相关docker搭建kafka集群内容请搜索软件技术网以前的文章或继续浏览下面的相关文章希望大家以后多多支持软件技术网！