Skip to main content
LinkedIn
Copied!

Table of Contents

Configuring External Kafka as a Stream service

Version:

Only available versions of this content are shown in the dropdown

Configure external Kafka as a stream service provider to use your own managed Kafka infrastructure.

You can configure external Kafka either by setting relevant properties on every Pega Platform node, or by using Dynamic System Settings, which are applied globally. For more information, see How to compose the key of a PRCONFIG dynamic system setting.

This procedure applies only to on-premises deployments.

Enable external Kafka

To use external Kafka as stream service, configure the following prconfig settings in your environment:

<!-- Set stream provider as external Kafka -->
<env name="services/stream/provider" value="ExternalKafka"/>
<!-- Provide bootstrap servers -->
<env name="services/stream/provider/url" value="broker1:9092, broker2:9092"
/>
<!-- Optional: provide desired replication factor. By default, we will set
replication factor equal to the number of brokers in the previous setting
-->
<!-- Available from 8.4.1 onwards -->
<env name="services/stream/external/replication/factor" value="2"/>

Connection properties

To establish a secure connection between Pega Platform nodes and your external Kafka cluster, the following connection properties can be set by using prconfigs:

<!-- Security protocol -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<!-- SSL certificate settings -->
<env name="services/stream/encryption/truststore/path" value="/path/to
/truststore.jsk"/>
<env name="services/stream/encryption/truststore/password" value="
my_passowrd"/>
<env name="services/stream/encryption/keystore/path" value="/path/to
/keystore.jks"/>
<env name="services/stream/encryption/keystore/password" value="
my_password"/>
<env name="services/stream/encryption/key/password" value="my_password"/>
<!-- SASL configuration. See supported mechanisms here: https://docs.
confluent.io/current/kafka/authentication_sasl/index.html -->
<env name="services/stream/encryption/sasl/mechanism" value="PLAIN"/>
<!-- It is important to escape any nested quotes with the &quot; This
doesn't apply to Dynamic System Settings -->
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.scram.ScramLoginModule required
username=&quot;my_user&quot; password=&quot;my_password&quot;;"/>

Topic name pattern

By default, topics originating from Pega Platform have the pega- prefix, so that it is easy to distinguish them from topics created by other applications. You can configure this pattern, to customize topic names per environment.

<env name="services/stream/name/pattern" value="pega-dev-{stream.name}"/>
Make sure that every Pega Platform node is configured with the same pattern. You can set the value for a pattern by using Dynamic System Settings. For more information, see How to compose the key of a PRCONFIG dynamic system setting.

Example: Confluent cloud

In this example, you configure Pega Platform stream service to point to https://confluent.cloud.

Follow the Quick Start for Apache Kafka using Confluent Cloud guide on the Confluent Documentation website to get your first Kafka cluster running. Once you have the cluster ready, copy the Kafka connection details. Here is an example of the connection details we are going to use:

bootstrap.servers=pkc-4nya7.us-east-1.aws.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="my_first_user" password="this_is_password";
sasl.mechanism=PLAIN

Add the following prconfig settings on every Pega Platform node.

For Pega Platform version 8.4.0 configurations, use:

<env name="services/stream/provider" value="ExternalKafka"/>
<!--
8.4.0 doesn't allow to configure replication factor. We use replication
factor equal to number of bootstrap servers as a default.
Therefore, you can copy the same bootstrap server many times to achieve
desired replication factor.
Starting from 8.4.1 we allow to configure desired replication factor
-->
<env name="services/stream/provider/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092, pkc-4nya7.us-east-1.aws.confluent.cloud:9092, pkc4nya7.us-east-1.aws.confluent.cloud:9092"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username=&quot;
my_first_user&quot; password=&quot;this_is_password&quot;;"/>

For Pega Platform version 8.4.1+ configurations, use:

<env name="services/stream/provider" value="ExternalKafka"/>
<env name="services/stream/provider/url" value="pkc-4nya7.us-east-1.aws.
confluent.cloud:9092"/>
<!-- Confluent cloud requires replication factor to be set to 3 -->
<env name="services/stream/external/replication/factor" value="3"/>
<!-- security settings -->
<env name="services/stream/encryption/security/protocol" value="
PLAINTEXT|SSL|SASL_PLAINTEXT|SASL_SSL"/>
<env name="services/stream/encryption/sasl/mechanism" value="SASL_SSL"/>
<env name="services/stream/encryption/sasl/jaas/config" value="org.apache.
kafka.common.security.plain.PlainLoginModule required username=&quot;
my_first_user&quot; password=&quot;this_is_password&quot;;"/>

All Pega Platform nodes then need to be restarted.

Did you find this content helpful?

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

Ready to crush complexity?

Experience the benefits of Pega Community when you log in.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us