How to List All Kafka Topics

1. Overview

In this article, we will introduce 3 different techniques on how to list all topics in a Kafka cluster.

2. Using Kafka CLI Commands

The Apache Kafka binaries are also a set of useful command-line tools that allow us to interact with Kafka and Zookeeper via the command line.

If you don't have them, you can download them from the official Apache Kafka Downloads repository.

Now you can list all the available topics by running the following command:

kafka-topics \
  --bootstrap-server localhost:9092 \
  --list

Alternatively, you can also use your Apache Zookeeper endpoint. This can be considered legacy as Apache Kafka is deprecating the use of Zookeeper as new versions are being released.

kafka-topics \
  --zookeeper localhost:2181 \
  --list

3. Using Java, Scala or Kotlin

If you are using any language that runs in the Java Virtual Machine, you can use the official Kafka clients libraries maintained by the Apache Kafka team.

3.1 Dependencies

First let's add the following dependency to the pom.xml:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.6.0</version>
</dependency>

Always use the latest versions from the Maven central repository for kafka-clients.

3.2 Using AdminClient API

Once imported, you can use the AdminClient API to query those

Properties properties = new Properties();
properties.put("bootstrap.servers", "127.0.0.1:9092");
Admin admin = Admin.create(properties);
Set<String> topicNames = admin.listTopics().names().get();
admin.close();

3.3 Using KafkaConsumer API

Alternatively, you can also list these topics by using any KafkaConsumer connected to the cluster. This may be preferred if you already have a consumer connected.

Map<String, List<PartitionInfo>> topics = consumer.listTopics();
Set<String> topicNames = topics.keySet();

If you need to create a new consumer, you can do so by:

Properties properties = new Properties();
properties.put("bootstrap.servers", "127.0.0.1:9092");
properties.put("key.deserializer", StringDeserializer.class.getName());
properties.put("value.deserializer", StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
// ... consumer logic
consumer.close();

In this article, we have seen the most important techniques when listing Apache Kafka topics.