How to connect to Apache Kafka using Kerberos authentication

Overview

In this short article, we will show how we can connect to an Apache Kafka that is currently secured using Kerberos authentication. The steps explained in this article are generic to any Apache Kafka client and battle-tested using Kafkaide.

What is Kerberos?

Kerberos is a computer network authentication protocol that works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. It is typically used in large organizations where users need to access kerberized applications hosted on servers spread across different geographical locations.

The most notable feature of Kerberos is that it eliminates the need for users to remember and enter multiple passwords when authenticating to kerberized services. Users only need to remember their Kerberos password, which is used to obtain tickets from the Kerberos server. These tickets can then be used to authenticate with kerberized services without the need to enter a password. Kerberos authentication is often used in conjunction with Apache Kafka, as it provides an additional layer of security for data in transit.

Required configuration parameters

When working with Apache Kafka, there are a few important configuration parameters that you need to set. These are:

security.protocol - set value to SASL_SSL or SASL_PLAINTEXT, depending on whether you also use SSL/TLS or not.

ssl.truststore.location - if your security protocol is SASL_SSL, you will need to pass the path to the truststore file. Also set ssl.truststore.password if your truststore was created using a password.

sasl.mechanism - always set to GSSAPI which is the name for the Kerberos protocol.

sasl.kerberos.service.name - set value to the Kerberos service name, this will be custom to your setup but if you don’t know yours you should ask your Kerberos administrator.

sasl.jaas.config or java.security.auth.login.config - read next section.

Setting up the SASL config

As we said, we will have to provide the Kerberos ticket along with the rest of the consumer configuration. There are different ways in which you can achieve this.

Option 1 - Using system ticket cache

The first is by using your local Kerberos ticket cache.

You can inspect which tickets are available by running klist from your command line.

sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;

Option 2 - Using keytab file

Sometimes you have the keytab file containing your ticket. If that’s your case, you can pass the keytab location instead of using the system cache.

sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/alice.keytab" principal="alice@example.com";

Option 3 - Previous options as file

An alternative to setting up sasl.jaas.config is having that same configuration stored in a file and passing that file location as java.security.auth.login.config property.

As an example, you can create a file at /path/to/kerberos.sasl.config with any of the following content, equivalent to the examples of the last two sections.

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=true;
};

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/path/to/alice.keytab"
  principal="alice@example.com";
};

Finally, when configuring your Kafka client, set the following config:

java.security.auth.login.config=/path/to/kerberos-sasl.jass.config

Conclusion

In this article, we’ve seen what are the required configuration when connecting to an Apache Kafka cluster secured using Kerberos authentication.

This configuration is standard for any type of Kafka client, whether you use the Apache Kafka Consumer API, Producer API or an Apache Kafka editor such as Kafkaide.