In this blog we will explore the basics of Kafka. An open-source distributed streaming platform, developed by LinkedIn & donated to Apache Software Foundation.
Kafka generally used in building real-time streaming data application. So that you can stream the live data from source systems without any delay. Following are some capabilities of kafka:
- Publishing stream of records.
- Store streams of record in fault tolerant way.
- Subscribe/Consuming stream of records.
Download Kafka:
Download latest Kafka from : kafka.apache.org and un-tar it.
Start Zookeeper Server:
First you have to start the zookeeper server, as it was used by the kafka.
bin/zookeeper-server-start.sh config/zookeeper.properties
Once your zookeeper server started leave that terminal & do not close. Just not down the port number.
Start kafka-server:
Now once the zookeeper server is up and running, you can start the kafka-server.
bin/kafka-server-start.sh config/server.properties
Once your kafka-server started leave that terminal & do not close.
Create Topic:
Where producer application push the stream of record is called topic. Producer can push records in multiple topics. Lest create a sample topic for demo.
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic firsttopic --partitions 1 --replication-factor 1
List all topics:
bin/kafka-topics.sh --list --zookeeper localhost:2181You can delete topics using :
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic <<topic-name>>Produce Data:
Sending the streams of records to a kafka topic using producer.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic firsttopic
Consuming Data:
Receive the streams of records from a kafka topic.
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic firsttopic --from-beginning
In real-time applications you have integrate the kafka APIs for producing & consuming data. Kafka has five core APIs
- Producer API – To publish a stream of records to one or more Kafka topics
- Consumer API – To subscribe to one or more topics
- Streams API – To convert input stream into output stream
- Connector API – Used for reusable connectors, external data source setup
- Admin API – For managing Kafka objects
Refer this blog for more details kafka.apache.org/intro
Thanks!
Happy Learning! Your feedback would be appreciated!
Follow @shobhitsinghIN
