Table of Contents

Article

Kafka with Charlatan

Kafka depends on the Zookeeper client library and the Zookeeper driver library; it uses the API of both libraries. The Zookeeper client library depends on the Zookeeper driver.

Thumbnail
Kafka - Zookeeper dependency

You can substitute the Zookeper library with the Charlatan library - a server that accepts and handles Zookeeper requests using the Zookeeper messaging protocol.

Charlatan consists of the following sub-projects:

  • Charlatan-common: contains watch management, session management, common data access object (DAO) interfaces
  • Charlatan-server: contains Zookeeper protocol implementation, Netty-based implementation of the server that accepts and handles zookeeper requests

Charlatan provides the following functionalities:

Node tree support

In order to support create, read, update, and delete (CRUD) operations on nodes, Charlatan DAO interface implementations use the Pega Platform database. Charlatan stores the node tree in the following table:

nodeIdnode ID
parentIdnode parent ID
datanode data
modepersistent/ephemeral
sessionsession which created the node
...fields with other node properties

Charlatan node tree

Broker watches support

In order to support watches, all updates need to be stored in the database. The existing interface functionality can also be implemented by using Apache Ignite/Hazelcast, or other databases.

Once a broker sets watch on a specific node change, it should receive a one-time update, regardless of whether the change was performed by this or any other broker. Every broker pulls updates created by different brokers from the table and notifies its watches.

For every node CRUD operation, an entry in the following node updates table is created and stored:

idupdate ID
typeevent type, for example: Created, Deleted, ChildrenChanged, DataChanged
pathfull node path
brokerbroker identifier that generated the change
timestampevent time

The node updates table

Ephemeral nodes support

If a node was created as ephemeral, it should be removed once the broker is off. An ephemeral broker from a previous broker session prevents the alive broker from starting. Removal of ephemeral nodes triggers alive brokers to re-select the leader, if needed, and to start replication operations.

Remove ephemeral nodes when broker is off and notify the rest of the brokers about the updates.

The database is used to monitor broker session states. With every broker’s keepalive message, the last_seen timestamp of the session is updated. See the following pr_data_streamservice_sessions table:

idbroker ID
sessionbroker session identifier
last_seentimestamp of the last ping

The broker sessions table

Stale sessions

Every Charlatan server periodically monitors its stale sessions and other Charlatan servers' stale sessions. If a stale session (last_seen timestamp timed out) is found, the server invalidates the stale session; it removes ephemeral nodes created by that session and notifies all connected brokers that are interested in this event.

A Charlatan server can invalidate sessions created by itself and sessions created by a different Charlatan sever.

When a Charlatan server/Pega Platform node crashes, broker sessions are invalidated by running the Charlatan server. If there are no more running Charlatan servers, the sessions are invalidated during the Charlatan server startup.

When a Kafka broker crashes, the Charlatan server invalidates its session.

 

To view the main outline for this article, see Kafka as a streaming service.

Published February 21, 2019 — Updated March 6, 2019

Related Content

Have a question? Get answers now.

Visit the Pega Support Community to ask questions, engage in discussions, and help others.