COS80023 - Task 6: Messaging with Kafka

Overview

Learn how to set up Kafka and what it does as a message broker.

Purpose

Experience how streaming data can be processed.

Task

Set up Kafka locally and stream messages from producer to consumer.

Time

This task is a Credit task and only mandatory if you would like to aim higher than a Pass grade. It should be submitted to Canvas for feedback. It should be discussed and signed off in tutorial 6 or 7.

This task should take no more than 2 hours to complete.

Resources

• Kafka from It can be downloaded directly, but there may be problems with unzipping it.
• Use this tutorial to set up a application using Kafka: https://kafka.apache.org/23/documentation/streams/quickstart
• It is helpful and instructive if you go through this tutorial first (although you don’t have to): https://kafka.apache.org/quickstart
• Any other online material

Feedback

Demonstrate your solution to the tutorial instructor.

Next

Get started on module 7.

Credit Task 6 — Submission Details and Assessment Criteria

For this task you only have to upload the screenshots of the producer and consumer consoles (terminals) when you have tested your setup. Your tutor will give online feedback and discuss the tasks with you in the lab when they are complete

Task 6

Set up a Kafka cluster on your local PC/Mac and configure it to use it with a wordcount application:

Overview of steps.

1. Launch a zookeeper instance
2. Start a kafka
3. Create topics streams-plaintext-input and streams-wordcount-output.
4. Start the Wordcount java
5. Start a producer that connects to streams-plaintext-input.
6. Start a consumer that connects to streams-wordcount-output. (You can also create a consumer that connects to streams-plaintext-input to listen to the raw data)
7. Type some text in the producer window and press Enter. Type some more and press Take a screenshot of your input and output windows and put into your document that you upload to Canvas.

Things to watch out for:

If you are using a Mac, the instructions in the tutorial apply. If you are using windows, you have to call \bin\windows\<somescript>.bat instead of

\bin\<somescript>.sh

I had to change the kafka-run-class.sh and kafka-run-class.bat files, because the directory to the kafka-streams-examples-2.3.0.jar file was wrong. If you are downloading directly, you have to adjust this.

The .properties files in the config directory have the Linux file separator / . If you have downloaded the .zip file from Canvas, I have changed the log

directory’s separators to \ in server.properties. Change back if you are using a Mac.

The exercise makes you open many command windows (or terminals). I suggest use one terminal to run all commands, because it preserves a history of commands and you can go back and change commands (using the ‘up’ button) without having to type or paste them again.

On a windows machine, you simply start the line with ‘start’, e.g. start bin\windows\kafka-server-start.bat config\server.properties

On a Mac, you add & to the end to start running the program in the background: bin/kafka-server.sh config/server.properties