环境搭建与快速入门

Kafka 搭建

在开始前，首先准备好自己的 docker 环境，可以安装Docker Desktop.
为了更方便的搭建学习环境，推荐使用 confluent 公司社区版 docker 镜像。

对于 confluent 版本和 apache 版本对照表如下：

Confluent Platform and Apache Kafka Compatibility

Confluent Platform	Apache Kafka®	Release Date	Standard End of Support	Platinum End of Support
7.0.x	3.0.x	October 27, 2021	October 27, 2023	October 27, 2024
6.2.x	2.8.x	June 8, 2021	June 8, 2023	June 8, 2024
6.1.x	2.7.x	February 9, 2021	February 9, 2023	February 9, 2024
6.0.x	2.6.x	September 24, 2020	September 24, 2022	September 24, 2023
5.5.x	2.5.x	April 24, 2020	April 24, 2022	April 24, 2023
5.4.x	2.4.x	January 10, 2020	January 10, 2022	January 10, 2023
5.3.x	2.3.x	July 19, 2019	July 19, 2021	July 19, 2022
5.2.x	2.2.x	March 28, 2019	March 28, 2021	March 28, 2022
5.1.x	2.1.x	December 14, 2018	December 14, 2020	December 14, 2021
5.0.x	2.0.x	July 31, 2018	July 31, 2020	July 31, 2021
4.1.x	1.1.x	April 16, 2018	April 16, 2020	April 16, 2021
4.0.x	1.0.x	November 28, 2017	November 28, 2019	November 28, 2020
3.3.x	0.11.0.x	August 1, 2017	August 1, 2019	August 1, 2020
3.2.x	0.10.2.x	March 2, 2017	March 2, 2019	March 2, 2020
3.1.x	0.10.1.x	November 15, 2016	November 15, 2018	November 15, 2019
3.0.x	0.10.0.x	May 24, 2016	May 24, 2018	May 24, 2019
2.0.x	0.9.0.x	December 7, 2015	December 7, 2017	December 7, 2018
1.0.0	–	February 25, 2015	February 25, 2017	February 25, 2018

首先新建文件docker-compose.yml，并写入如下内容：

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

然后在该文件目录下执行docker-compose up -d，既可启动 zookeeper 容器和 kafka broker 容器,docker-compose down可以关闭创建好的集群。

PS F:\docker-workspace\kafka> docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS          PORTS                          NAMES
aac9044963e5   confluentinc/cp-kafka:7.0.1       "/etc/confluent/dock…"   11 seconds ago   Up 10 seconds   0.0.0.0:9092->9092/tcp         broker
debcc3a0235d   confluentinc/cp-zookeeper:7.0.1   "/etc/confluent/dock…"   13 seconds ago   Up 12 seconds   2181/tcp, 2888/tcp, 3888/tcp   zookeeper

如上情况，即为创建成功。

Quick Start

创建 Topic

默认自动创建 Topic 配置是启用的，也就是说第一步也可以省略，只要有 Producer 或者 Consumer,就会自动创建 Topic 的。

docker exec broker kafka-topics --bootstrap-server broker:9092 --create --topic trumandu

查看 Topic 信息

PS F:\docker-workspace\kafka> docker exec broker kafka-topics --describe --bootstrap-server broker:9092  --topic trumandu
Topic: trumandu TopicId: 8peszWGBTgulbSZPYHiMzQ PartitionCount: 1       ReplicationFactor: 1    Configs:
        Topic: trumandu Partition: 0    Leader: 1       Replicas: 1     Isr: 1

生产消息

执行

docker exec --interactive --tty broker kafka-console-producer --bootstrap-server broker:9092 --topic trumandu

然后输入测试信息，完成后执行Ctrl-D可以退出命令行。

消费消息

PS F:\docker-workspace\kafka> docker exec --interactive --tty broker kafka-console-consumer --bootstrap-server broker:9092  --topic trumandu --from-beginning

hello
truman

执行Ctrl-C可以退出命令行

以上例子中kafka-topics kafka-console-producer 和kafka-console-consumer具体参数和用法详见文档。

参考

Apache Kafka® Quick Start