Kafka basics with Docker

Docker is my favorite technology of all time. Well not just docker but containers in general are very cool. And something that I use for almost every new tech that I want to learn. Very quick very easy very efficient. Of course there is a learning curve, but once you do get over the curve its pretty fucking awesome.

So for this blog entry I am just going to go through the steps that I took to set up a simple containerized pipeline that makes use of Apache Kafka and Apache Spark.

I decided to use bitnami images for kafka and spark, as they come preconfigured with many of the settings. A year ago the image I remember using was wurstmeister for both kafka and zookeeper. But recently I learned that there’s something called KRaft (Kafka Raft Metadata mode) which was introduced to remove Apache Kafka’s dependency on ZooKeeper for metadata management.

First service in the docker-compose file. listing the kafka service

This runs in KRaft mode meaning needs no zookeeper. Also this is a very basic kafka setup. One broker. we can have multiple brokers for replication. Since this is for learning purpose it’s not secured. 9092 is the port on which we can interact even with kafka using our host machine. Container name is kafka.

I prefer using docker-compose as it is more cleaner and concise than just using shell terminal. Also in a multi container setup it deals with networking aspects by itself, of course we can create custom networks, but its just more efficient this way.