(Last Updated On: April 22, 2022)

Apache Kafka is an open-source Stream Processing and Management Platform that accepts, stores, organizes, and distributes data to various end-users and applications. Data Overloading and Data Duplication can occur when users send hundreds of thousands of messages or data onto Kafka servers. Data on Kafka Servers is frequently unstructured and confused as a result of these issues.

As a result, Kafka console consumers and end-users are unable to obtain the data they require from Kafka Servers. Users can create separate Kafka Topics in a Kafka Server to avoid the issues of having untidy and unstructured data in the server. 

Users may simply produce and consume messages to and from the Kafka Servers using Kafka Topic, which allows them to store and arrange data according to different categories and use cases. 

What are kafka topics?

Topics are a dedicated and basic unit in Apache Kafka for event or message grouping. In other words, Kafka Topics are logically organized Virtual Groups or Logs that allow users to easily send and receive data between Kafka Servers.

When a Producer transmits messages or events to a Kafka Topic, the topics append the messages one after the other, forming a Log File. Producers can also Push Messages into the tails of these newly formed logs, while consumers Pull Messages from a single Kafka Topic.

  Users can do logical segregation between Messages and Events by defining Kafka Topics, which works in the same way that various tables in a database might have different sorts of data.

Based on your use cases, Apache Kafka allows you to build an unlimited number of topics. However, each topic in a Kafka Cluster should be given a distinct and recognisable name in order to distinguish it from other topics.

A producer is necessary in order to submit data to a Kafka topic. The producer’s job is to send and receive data and messages from Kafka topics.

We’ll see how a producer distributes messages to Kafka topics in this section.

1. To start a production company, take these steps:

Start both the zookeeper and the kafka servers in the first step.

To start all services in the proper order, type the following commands:

• Make the ZooKeeper service available.
• Important: Apache Kafka will soon stop requiring ZooKeeper.
• bin/zookeeper-server-start.sh config/zookeeper.properties

Run the following commands in a new window:

• Begin using the Kafka Broker service.
• bin/kafka-server-start.sh config/server.properties

2. Make a topic for your events to be stored on

Kafka is a distributed event streaming technology that enables you to read, write, store, and analyze events (also known as records or messages) across several machines.

Payment transactions, mobile phone geolocation updates, shipment orders, sensor measurements from IoT devices or medical equipment, and a variety of other events are examples of events. Topics are used to organize and store events. A subject is analogous to a folder on a filesystem, and events are the files in that folder, to put it simply.

You must first establish a subject before you can write your first events. Run the following commands in a fresh window:

• bin/kafka-topics.sh –create –topic quickstart-events –bootstrap-server localhost:9092

The kafka-topics.sh command can be used without any inputs to provide usage information for all of Kafka’s command line tools. It can, for example, display you information like the new topic’s partition count:

bin/kafka-topics.sh –describe –topic quickstart-events –bootstrap-server localhost:9092 Topic:quickstart-events PartitionCount:1 ReplicationFactor:1 Configs: Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0

3. Include some events in the subject matter

For writing (or reading) events, a Kafka client talks with Kafka brokers across the network. When the events are received, the brokers will store them in a reliable and fault-tolerant manner for as long as you require—even indefinitely.

Write a few events into your topic using the Kafka console producer client. By default, each line you input will be written to the topic as a separate event.

bin/kafka-console-producer.sh –topic quickstart-events –bootstrap-server localhost:9092 

4. Use Kafka connect to import and export your data as event streams

You undoubtedly have a lot of data in relational databases or traditional messaging systems, as well as a lot of apps that leverage these systems. Kafka Connect enables you to feed real-time data from other systems into Kafka and vice versa. As a result, integrating Kafka with other systems is quite simple. Hundreds of similar connectors are easily available, making the operation even simpler.

5. Use Kafka streaming to process your events 

You can use the Kafka Streams client library for Java/Scala to process data once it has been saved in Kafka as events. It enables you to build mission-critical real-time applications and microservices with data stored in Kafka topics as input and/or output. Kafka Streams combines the ease of creating and deploying ordinary Java and Scala client-side apps with the benefits of Kafka’s server-side cluster technology to create highly scalable, elastic, fault-tolerant, and distributed systems. Exact-once processing, stateful operations and aggregations, windowing, joins, event-based processing, and much more are all supported by the library.

Here’s an example of how the popular WordCount algorithm might be implemented:

KStream<String, String> textLines = builder.stream(“quickstart-events”); KTable<String, Long> wordCounts = textLines .flatMapValues(line -> Arrays.asList(line.toLowerCase().split(” “))) .groupBy((keyIgnored, word) -> word) .count(); wordCounts.toStream().to(“output-topic”, Produced.with(Serdes.String(), Serdes.Long()));

6. End the kafka environment 

Feel free to deconstruct the Kafka environment now that you’ve completed the quickstart—or keep experimenting around.

If you haven’t already, press Ctrl-C to terminate the producer and consumer clients.

Ctrl-C will terminate the Kafka broker.

Finally, use the Ctrl-C shortcut to terminate the ZooKeeper server.

Run the command: kafka delete to erase all data from your local Kafka environment, including any events you’ve made along the way.