做網(wǎng)站用小圖標(biāo)在什么網(wǎng)下載,深圳聯(lián)合辦公空間,濟(jì)南網(wǎng)站建設(shè)山東聚搜網(wǎng)見(jiàn)效快,網(wǎng)站建設(shè)的物流在單臺(tái)服務(wù)器上使用 Docker Compose 部署 Kafka 3.7 KRaft 集群#xff08;3 節(jié)點(diǎn)#xff09; 。所有 3 個(gè) Kafka 節(jié)點(diǎn)#xff08;combined mode#xff1a;broker controller#xff09;運(yùn)行在同一臺(tái)物理機(jī)上#xff0c;通過(guò)不同端口區(qū)分#xff0c;適用于開(kāi)發(fā)、測(cè)試或…在單臺(tái)服務(wù)器上使用 Docker Compose 部署 Kafka 3.7 KRaft 集群3 節(jié)點(diǎn)。所有 3 個(gè) Kafka 節(jié)點(diǎn)combined modebroker controller運(yùn)行在同一臺(tái)物理機(jī)上通過(guò)不同端口區(qū)分適用于開(kāi)發(fā)、測(cè)試或輕量級(jí)生產(chǎn)場(chǎng)景。特點(diǎn)單機(jī)部署無(wú)需多臺(tái)云服務(wù)器使用官方apache/kafka:3.7.0鏡像內(nèi)置 JDK 17啟用SASL/PLAIN 認(rèn)證支持從本機(jī)外部如本地電腦訪問(wèn) Kafka環(huán)境要求操作系統(tǒng)CentOS 7/8/9、Ubuntu 等本例以 CentOS 為例Docker ≥ 20.10Docker Compose ≥ v2.0或docker-compose可用內(nèi)存 ≥ 4GB建議 6GB目錄結(jié)構(gòu)kafka-cluster/ ├── docker-compose.yml ├── kafka/ │ └── kafka_server_jaas.conf └── client/ ├── kafka_admin_client_jaas.conf └── client.properties第一步創(chuàng)建配置文件1. 創(chuàng)建 JAAS 文件服務(wù)端mkdir-p /opt/kafka-cluster/{kafka,client}cd/opt/kafka-clustercatkafka/kafka_server_jaas.confEOF KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required usernameadmin passwordStrongPassword123! user_adminStrongPassword123! user_producerStrongPassword456! user_consumerStrongPassword789!; }; EOF密碼請(qǐng)按需修改但所有節(jié)點(diǎn)必須一致。2. 客戶端認(rèn)證文件用于測(cè)試catclient/kafka_admin_client_jaas.confEOF KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required usernameadmin passwordStrongPassword123!; }; EOFcatclient/client.propertiesEOF security.protocolSASL_PLAINTEXT sasl.mechanismPLAIN sasl.jaas.configorg.apache.kafka.common.security.plain.PlainLoginModule required usernameadmin passwordStrongPassword123!; EOF第二步編寫docker-compose.yml單機(jī) 3 節(jié)點(diǎn)vi docker-compose.yml內(nèi)容如下version:3.8services:kafka1:image:apache/kafka:3.7.0container_name:kafka1ports:-9092:9092environment:KAFKA_NODE_ID:1KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093KAFKA_ADVERTISED_LISTENERS:SASL_PLAINTEXT://host.docker.internal:9092,PLAINTEXT://kafka1:9094KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXTKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka1:9093,2kafka2:9093,3kafka3:9093KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME:PLAINTEXTKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR:3KAFKA_CFG_AUTHORIZER_CLASS_NAME:org.apache.kafka.metadata.authorizer.StandardAuthorizerKAFKA_SASL_ENABLED_MECHANISMS:PLAINKAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL:PLAINKAFKA_OPTS:-Djava.security.auth.login.config/opt/kafka/config/kafka_server_jaas.confvolumes:-./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.confnetworks:-kafka-netextra_hosts:-host.docker.internal:host-gatewaykafka2:image:apache/kafka:3.7.0container_name:kafka2ports:-9093:9092environment:KAFKA_NODE_ID:2KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093KAFKA_ADVERTISED_LISTENERS:SASL_PLAINTEXT://host.docker.internal:9093,PLAINTEXT://kafka2:9094KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXTKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka1:9093,2kafka2:9093,3kafka3:9093KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME:PLAINTEXTKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR:3KAFKA_CFG_AUTHORIZER_CLASS_NAME:org.apache.kafka.metadata.authorizer.StandardAuthorizerKAFKA_SASL_ENABLED_MECHANISMS:PLAINKAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL:PLAINKAFKA_OPTS:-Djava.security.auth.login.config/opt/kafka/config/kafka_server_jaas.confvolumes:-./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.confnetworks:-kafka-netextra_hosts:-host.docker.internal:host-gatewaykafka3:image:apache/kafka:3.7.0container_name:kafka3ports:-9094:9092environment:KAFKA_NODE_ID:3KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:SASL_PLAINTEXT://:9092,PLAINTEXT://:9094,CONTROLLER://:9093KAFKA_ADVERTISED_LISTENERS:SASL_PLAINTEXT://host.docker.internal:9094,PLAINTEXT://kafka3:9094KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXTKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka1:9093,2kafka2:9093,3kafka3:9093KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME:PLAINTEXTKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR:3KAFKA_CFG_AUTHORIZER_CLASS_NAME:org.apache.kafka.metadata.authorizer.StandardAuthorizerKAFKA_SASL_ENABLED_MECHANISMS:PLAINKAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL:PLAINKAFKA_OPTS:-Djava.security.auth.login.config/opt/kafka/config/kafka_server_jaas.confvolumes:-./kafka/kafka_server_jaas.conf:/opt/kafka/config/kafka_server_jaas.confnetworks:-kafka-netextra_hosts:-host.docker.internal:host-gatewaynetworks:kafka-net:driver:bridge關(guān)鍵說(shuō)明配置說(shuō)明host.docker.internal線下想使用且沒(méi)有域名的話可以替換成公網(wǎng)ip(在實(shí)驗(yàn)時(shí)博主是配置的公網(wǎng)IP)端口映射9092→9092,9093→9092,9094→9092→ 外部通過(guò)宿主機(jī)IP:9092/9093/9094訪問(wèn)KAFKA_ADVERTISED_LISTENERS必須設(shè)為host.docker.internal否則客戶端連錯(cuò)地址所有節(jié)點(diǎn)共用同一 JAAS 文件用戶密碼一致第三步啟動(dòng)集群cd/opt/kafka-cluster docker compose up -d如果提示docker compose不存在請(qǐng)安裝sudocurl-Lhttps://github.com/docker/compose/releases/latest/download/docker-compose-$(uname-s)-$(uname-m)-o /usr/local/bin/docker-composesudochmodx /usr/local/bin/docker-compose查看日志docker compose logs -f kafka1成功標(biāo)志[main] INFO ... Kafka is now ready to serve requests首次啟動(dòng)可能需要 30~60 秒完成 controller 選舉。第四步驗(yàn)證集群從宿主機(jī)外部訪問(wèn)假設(shè)你的服務(wù)器公網(wǎng) IP 是142.12.153.25則外部可通過(guò)142.12.153.25:9092142.12.153.25:9093142.12.153.25:9094連接 Kafka。1. 創(chuàng)建 Topic在服務(wù)器本地執(zhí)行使用 PLAINTEXT 端口 9094 創(chuàng)建 topic無(wú)需認(rèn)證內(nèi)部專用docker run --rm -it--network kafka-cluster_kafka-netapache/kafka:3.7.0/opt/kafka/bin/kafka-topics.sh--create--topic test-single-host--partitions3--replication-factor3--bootstrap-server kafka1:9094,kafka2:9094,kafka3:9094#列出topicdocker run --rm -it--network kafka-cluster_kafka-net-v$(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf-v$(pwd)/client/client.properties:/tmp/client.propertiesapache/kafka:3.7.0bash-c export KAFKA_OPTS-Djava.security.auth.login.config/tmp/jaas.conf /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --command-config /tmp/client.properties #生產(chǎn)者生產(chǎn)消息echoHello Kafka|docker run --rm -i--network kafka-cluster_kafka-net-v$(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf-v$(pwd)/client/client.properties:/tmp/client.propertiesapache/kafka:3.7.0bash-c export KAFKA_OPTS-Djava.security.auth.login.config/tmp/jaas.conf /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka1:9092 --producer.config /tmp/client.properties --topic test-single-host #消費(fèi)者消費(fèi)消息docker run --rm -it--network kafka-cluster_kafka-net-v$(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf-v$(pwd)/client/client.properties:/tmp/client.propertiesapache/kafka:3.7.0bash-c export KAFKA_OPTS-Djava.security.auth.login.config/tmp/jaas.conf /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --consumer.config /tmp/client.properties --topic test-single-host --from-beginning 2. 從本地電腦線下測(cè)試生產(chǎn)消息# 替換 YOUR_SERVER_PUBLIC_IPSERVER_IP142.12.153.25echoHello from local!|docker run --rm -i-v$(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf-v$(pwd)/client/client.properties:/tmp/client.propertiesapache/kafka:3.7.0bash-c export KAFKA_OPTS-Djava.security.auth.login.config/tmp/jaas.conf /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 115.159.155.193:9092 --producer.config /tmp/client.properties --topic test-single-host 3. 消費(fèi)消息本地docker run --rm -it-v$(pwd)/client/kafka_admin_client_jaas.conf:/tmp/jaas.conf-v$(pwd)/client/client.properties:/tmp/client.propertiesapache/kafka:3.7.0bash-c export KAFKA_OPTS-Djava.security.auth.login.config/tmp/jaas.conf /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server${SERVER_IP}:9092 --consumer.config /tmp/client.properties --topic test-single-host --from-beginning --timeout-ms 10000 如果看到消息說(shuō)明單機(jī) Kafka 集群工作正常上面23均未檢驗(yàn)僅在本地客戶端通過(guò)過(guò)公網(wǎng)ip的方式連上kafka并進(jìn)行生產(chǎn)和消費(fèi)消息安全與網(wǎng)絡(luò)建議防火墻只開(kāi)放9092-9094給可信 IPsudofirewall-cmd --permanent --add-port9092-9094/tcpsudofirewall-cmd --reload不要用于公網(wǎng)生產(chǎn)環(huán)境建議僅限內(nèi)網(wǎng)或測(cè)試升級(jí)為 SASL_SSL如需加密傳輸停止容器docker compose down