In this article, we will introduce 3 different techniques on how to list all topics in a Kafka cluster.
2. Using Kafka CLI Commands
The Apache Kafka binaries also a set of useful command line tools that allow us to interact with Kafka and Zookeeper via the command line.
If you don't have them, you can download them from the official Apache Kafka Downloads repository.
Now you can list all the available topics by running the following command:
kafka-topics \ --bootstrap-server localhost:9092 \ --list
Alternatively, you can also use your Apache Zookeeper endpoint. This can be considered legacy as Apache Kafka is deprecating the use of Zookeeper as new versions are being released.
kafka-topics \ --zookeeper localhost:2181 \ --list
3. Using Java, Scala or Kotlin
If you are using any language that runs in the Java Virtual Machine, you can use the official Kafka clients libraries maintained by the Apache Kafka team.
First let's add the following dependency to the
<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>2.6.0</version> </dependency>
Always use the latest versions from the Maven central repository for kafka-clients
3.2 Using AdminClient API
Once imported, you can use the AdminClient API to query those
Properties properties = new Properties(); properties.put("bootstrap.servers", "127.0.0.1:9092"); Admin admin = Admin.create(properties); Set<String> topicNames = admin.listTopics().names().get(); admin.close();
3.3 Using KafkaConsumer API
Alternatively, you can also list these topics by using any KafkaConsumer connected to the cluster. This may preferred if you already have a consumer connected.
Map<String, List<PartitionInfo>> topics = consumer.listTopics(); Set<String> topicNames = topics.keySet();
If you need to create a new consumer, you can do so by:
Properties properties = new Properties(); properties.put("bootstrap.servers", "127.0.0.1:9092"); properties.put("key.deserializer", StringDeserializer.class.getName()); properties.put("value.deserializer", StringDeserializer.class.getName()); KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties); // ... consumer logic consumer.close();
In this article, we have seen the most important techniques when listing Apache Kafka topics.