Close popover

Table of Contents

Creating a Kafka data set

Version:

You can create an instance of a Kafka data set in the Pega Platform to connect to a topic in the Kafka cluster. Topics are categories where the Kafka cluster stores streams of records. Each record in a topic consists of a key, value, and a time stamp. You can also create a new topic in the Kafka cluster from the Pega Platform and then connect to that topic.

Use a Kafka data set as a source of events (for example, customer calls or messages) that are used as input for Event Strategy rules that process data in real time.

You can connect to an Apache Kafka cluster version 0.10.0.1 or later.
  1. In Dev Studio, click Create Data Model Data Set .

  2. Provide the data set label and identifier.

  3. From the Type list, select Kafka.

  4. Provide the ruleset, Applies to class, and ruleset version of the data set.

  5. Click Create and open.

  6. In the Connection section, in the Kafka configuration instance field, select an existing Kafka cluster record ( Data-Admin-Kafka class) or Creating a Kafka configuration instance (for example, when no records are present) by clicking the Open icon.

  7. Check whether the Pega Platform is connected to the Kafka cluster by clicking Text connectivity.

  8. In the Topic section, perform one of the following actions:

    • Select the Create new check box and enter the topic name to define a new topic in the Kafka cluster.
    • Select the Select from list check box to connect to an existing topic in the Kafka cluster.
    By default, the name of the topic is the same as the name of the data set. If you enter a new topic name, that topic is created in the Kafka cluster only if the ability to automatically create topics is enabled on that Kafka cluster.
  9. Optional:

    In the Partition Key(s) section, define the data set partitioning by performing the following actions:

    1. Click Add key.

    2. In the Key field, press the Down Arrow key to select a property to be used by the Kafka data set as a partitioning key.

      By default, the available properties to be used as keys correspond to the properties of the Applies To class of the Kafka data set.
    By configuring partitioning you can ensure that related records are sent to the same partition. If no partition keys are set, the Kafka data set randomly assigns records to partitions.
  10. Optional:

    If you want to use a different format for records than JSON, in the Record format section, select Custom and configure the record settings:

    1. In the Serialization implementation field, enter a fully qualified Java class name for your PegaSerde implementation.

      com.pega.dsm.kafka.CsvPegaSerde
    2. Optional:

      Expand the Additional configuration section and define additional configuration options for the implementation class by clicking Add key value pair and entering properties in the Key and Value fields.

  11. Click Save.

  • Creating a Kafka configuration instance

    A Kafka configuration instance represents an external Apache Kafka server or cluster of servers that is the source of stream data that is processed in real time by Event Strategy rules in your application. You must create a Kafka configuration instance before you can create Kafka data sets for connecting to specific topics that are part of the cluster. You can create an instance of a Kafka cluster in the Data-Admin-Kafka class of Pega Platform.

  • Types of Data Set rules

    Learn about the types of data set rules that you can create in Pega Platform.

  • About Data Set rules

    Data sets define collections of records, allowing you to set up instances that make use of data abstraction to represent data stored in different sources and formats. Depending on the type selected when creating a new instance, data sets represent Visual Business Director (VBD) data sources, data in database tables or data in decision data stores. Through the data management operations for each data set type, you can read, insert and remove records. Data sets are used on their own through data management operations, as part of combined data

  • Data Set rules - Completing the Create, Save As, or Specialization form

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.