Kafka

Contents

Kafka#

Run the Kafka container using the following command:

docker run -p 9092:9092 -itd --name experimenting_kafka -v ./:/knowledge apache/kafka:4.0.0

In order to run notebooks, you are supposed to intsall python with some additional packages:

docker exec -u root experimenting_kafka apk add gcc python3-dev py3-pip musl-dev linux-headers
docker exec experimenting_kafka python3 -m venv /home/appuser/venv
docker exec experimenting_kafka /home/appuser/venv/bin/pip install ipykernel bash_kernel
docker exec experimenting_kafka /home/appuser/venv/bin/python -m bash_kernel.install

So, finally, once you have successfully set up all the preparation in the folder /opt/kafka/bin, you should be able to see scripts to manipulate kafka. They are represented in the following cell:

ls /opt/kafka/bin

connect-distributed.sh		 kafka-jmx.sh
connect-mirror-maker.sh		 kafka-leader-election.sh
connect-plugin-path.sh		 kafka-log-dirs.sh
connect-standalone.sh		 kafka-metadata-quorum.sh
kafka-acls.sh			 kafka-metadata-shell.sh
kafka-broker-api-versions.sh	 kafka-producer-perf-test.sh
kafka-client-metrics.sh		 kafka-reassign-partitions.sh
kafka-cluster.sh		 kafka-replica-verification.sh
kafka-configs.sh		 kafka-run-class.sh
kafka-console-consumer.sh	 kafka-server-start.sh
kafka-console-producer.sh	 kafka-server-stop.sh
kafka-console-share-consumer.sh  kafka-share-groups.sh
kafka-consumer-groups.sh	 kafka-storage.sh
kafka-consumer-perf-test.sh	 kafka-streams-application-reset.sh
kafka-delegation-tokens.sh	 kafka-topics.sh
kafka-delete-records.sh		 kafka-transactions.sh
kafka-dump-log.sh		 kafka-verifiable-consumer.sh
kafka-e2e-latency.sh		 kafka-verifiable-producer.sh
kafka-features.sh		 trogdor.sh
kafka-get-offsets.sh		 windows
kafka-groups.sh

Topics#

These are folders that contain events. To manipulate with topics, use the ./kafka-topic.sh script. The most important options for the script are:

  • --create to create a new topic.

  • --list to show created topics.

  • --delete to delete the topic.

The following cell illustrates the use of the --create option.

./kafka-topics.sh --create --topic myFirstTopic --bootstrap-server localhost:9092

Created topic myFirstTopic.

After creating, we can list all the topics that have been created:

./kafka-topics.sh --bootstrap-server localhost:9092 --list

myFirstTopic

And the --delete for the myFirstTopic:

./kafka-topics.sh --delete --topic myFirstTopic --bootstrap-server localhost:9092
Contents