docker搭建kafka集群的方法实现
目录
-
一、原生Docker命令二、镜像选择三、集群规划四、Zookeeper集群安装五、Kafka集群安装
一、原生Docker命令
1. 删除所有dangling数据卷(即无用的Volume,僵尸文件)
- docker volume rm $(docker volume ls -qf dangling=true)
复制代码 2. 删除所有dangling镜像(即无tag的镜像)
- docker rmi $(docker images | grep “^<none>” | awk “{print $3}”
复制代码 3. 删除所有关闭的容器
- docker ps -a | grep Exit | cut -d ‘ ‘ -f 1 | xargs docker rm
复制代码
二、镜像选择
环境为M1版本的mbp:
-
Zookeeper采用zookeeperKafka采用wurstmeister/kafkaKafka-Manager采用scjtqs/kafka-managerMysql采用mysql/mysql-server
三、集群规划
1. 新建docker网络
- docker network create docker-net –subnet 172.20.10.0/16
复制代码
- docker network ls
复制代码 2. 集群规划
hostname | Ip addr | port | listener |
zook1 | 172.20.10.11 | 2184:2181 | |
zook2 | 172.20.10.12 | 2185:2181 | |
zook3 | 172.20.10.13 | 2186:2181 | |
kafka1 | 172.20.10.14 | 内部9092:9092,外部9192:9192 | kafka1 |
kafka2 | 172.20.10.15 | 内部9093:9093,外部9193:9193 | kafka2 |
Kafka3 | 172.20.10.16 | 内部9094:9094,外部9194:9194 | Kafka3 |
本机(宿主机Mbp) | 172.20.10.2 | ||
kafka manager | 172.20.10.10 | 9000:9000 |
四、Zookeeper集群安装
1. 新建文件zk-docker-compose.yml
- version: ‘3.4’
- services:
- zook1:
- image: zookeeper:latest
- restart: always
- hostname: zook1
- container_name: zook1 #容器名称,方便在rancher中显示有意义的名称
- ports:
- – 2183:2181 #将本容器的zookeeper默认端口号映X去
- volumes: # 挂载数据卷
- – “/Users/konsy/Development/volume/zkcluster/zook1/data:/data”
- – “/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog”
- – “/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs”
- environment:
- ZOO_MY_ID: 1 #即是zookeeper的节点值,也是kafka的brokerid值
- ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
- networks:
- docker-net:
- ipv4_address: 172.20.10.11
- zook2:
- image: zookeeper:latest
- restart: always
- hostname: zook2
- container_name: zook2 #容器名称,方便在rancher中显示有意义的名称
- ports:
- – 2184:2181 #将本容器的zookeeper默认端口号映X去
- volumes:
- – “/Users/konsy/Development/volume/zkcluster/zook2/data:/data”
- – “/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog”
- – “/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs”
- environment:
- ZOO_MY_ID: 2 #即是zookeeper的节点值,也是kafka的brokerid值
- ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
- networks:
- docker-net:
- ipv4_address: 172.20.10.12
- zook3:
- image: zookeeper:latest
- restart: always
- hostname: zook3
- container_name: zook3 #容器名称,方便在rancher中显示有意义的名称
- ports:
- – 2185:2181 #将本容器的zookeeper默认端口号映X去
- volumes:
- – “/Users/konsy/Development/volume/zkcluster/zook3/data:/data”
- – “/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog”
- – “/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs”
- environment:
- ZOO_MY_ID: 3 #即是zookeeper的节点值,也是kafka的brokerid值
- ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
- networks:
- docker-net:
- ipv4_address: 172.20.10.13
- networks:
- docker-net:
- external:
- name: docker-net
复制代码 2. 执行脚本部署zookeeper至Docker:
- docker compose -f ./zk-docker-compose.yml up -d
复制代码
五、Kafka集群安装
1. 新建文件kafka-docker-compose.yml
- version: ‘2’
- services:
- kafka1:
- image: docker.io/wurstmeister/kafka
- restart: always
- hostname: kafka1
- container_name: kafka1
- ports:
- – 9093:9093
- – 9193:9193
- environment:
- KAFKA_BROKER_ID: 1
- KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193
- #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
- SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
- KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
- ALLOW_PLAINTEXT_LISTENER : ‘yes’
- JMX_PORT: 9999 #开放JMX监控端口,来监测集群数据
- volumes:
- – /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
- – /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
- external_links:
- – zook1
- – zook2
- – zook3
- networks:
- docker-net:
- ipv4_address: 172.20.10.14
- kafka2:
- image: docker.io/wurstmeister/kafka
- restart: always
- hostname: kafka2
- container_name: kafka2
- ports:
- – 9094:9094
- – 9194:9194
- environment:
- KAFKA_BROKER_ID: 2
- KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194
- #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
- KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
- KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
- ALLOW_PLAINTEXT_LISTENER : ‘yes’
- JMX_PORT: 9999 #开放JMX监控端口,来监测集群数据
- volumes:
- – /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka
- – /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka
- external_links:
- – zook1
- – zook2
- – zook3
- networks:
- docker-net:
- ipv4_address: 172.20.10.15
- kafka3:
- image: docker.io/wurstmeister/kafka
- restart: always
- hostname: kafka3
- container_name: kafka3
- ports:
- – 9095:9095
- – 9195:9195
- environment:
- KAFKA_BROKER_ID: 3
- KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195
- #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
- KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
- KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
- ALLOW_PLAINTEXT_LISTENER : ‘yes’
- JMX_PORT: 9999 #开放JMX监控端口,来监测集群数据
- volumes:
- – /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka
- – /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka
- external_links:
- – zook1
- – zook2
- – zook3
- networks:
- docker-net:
- ipv4_address: 172.20.10.16
- networks:
- docker-net:
- external:
- name: docker-net
复制代码 2. 执行脚本部署kafka至Docker:
- docker compose -f ./kafka-docker-compose.yml up -d
复制代码 3. listeners 和 advertised.listeners
-
listeners: 学名叫监听器,其实就是告诉外部连接者要通过什么协议访问指定主机名和端口开放的 Kafka 服务。advertised.listeners:和 listeners 相比多了个 advertised。Advertised 的含义表示宣称的、公布的,就是说这组监听器是 Broker 用于对外发布的。
比如说:
- listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094
- advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公网 ip>:端口
- kafka_listener_security_protocol_map: “INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT”
- kafka_inter_broker_listener_name: “INSIDE”
复制代码 advertised_listeners 监听器会注册在 zookeeper 中;
当我们对 172.17.0.10:9092 请求建立连接,kafka 服务器会通过 zookeeper 中注册的监听器,找到 INSIDE 监听器,然后通过 listeners 中找到对应的 通讯 ip 和 端口;
同理,当我们对 <公网 ip>:端口 请求建立连接,kafka 服务器会通过 zookeeper 中注册的监听器,找到 OUTSIDE 监听器,然后通过 listeners 中找到对应的 通讯 ip 和 端口 172.17.0.10:9094;
总结:advertised_listeners 是对外暴露的服务端口,真正建立连接用的是 listeners。
4.安装kafka-manager
4.1 新建文件kafka-manager-docker-compose.yml
- version: ‘2’
- services:
- kafka-manager:
- image: scjtqs/kafka-manager:latest
- restart: always
- hostname: kafka-manager
- container_name: kafka-manager
- ports:
- – 9000:9000
- external_links: # 连接本compose文件以外的container
- – zook1
- – zook2
- – zook3
- – kafka1
- – kafka2
- – kafka3
- environment:
- ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181
- KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095
- APPLICATION_SECRET: letmein
- KM_ARGS: -Djava.net.preferIPv4Stack=true
- networks:
- docker-net:
- ipv4_address: 172.20.10.10
- networks:
- docker-net:
- external:
- name: docker-net
复制代码 4.2 执行脚本部署kafka-manager至Docker:
- docker compose -f ./kafka-manager-docker-compose.yml up -d
复制代码 4.3 配置Cluster
5. 测试kafka
5.1连接容器
进入kafka容器的命令行:
- docker exec -ti kafka /bin/bash
复制代码 进入kafka所在目录:
- cd opt/kafka_2.13-2.8.1/
复制代码 5.2 创建Replication为2,Partition为2的topic
- bin/kafka-topics.sh –create –zookeeper zook1:2181 –replication-factor 2 –partitions 2 –topic partopic
复制代码 5.3 查看topic的状态
在kafka容器中的 opt/kafka_2.12-1.1.0/ 目录下输入
- bin/kafka-topics.sh –describe –zookeeper zook1:2181 –topic partopic
复制代码 到此这篇关于docker搭建kafka集群的方法实现的文章就介绍到这了,更多相关docker搭建kafka集群内容请搜索软件技术网以前的文章或继续浏览下面的相关文章希望大家以后多多支持软件技术网!
原创文章,作者:starterknow,如若转载,请注明出处:https://www.starterknow.com/108948.html