LinkedIn
Copied!

Table of Contents

Kafka with Charlatan

Kafka depends on the Zookeeper client library and the Zookeeper driver library; it uses the API of both libraries. The Zookeeper client library depends on the Zookeeper driver.

Thumbnail
Kafka - Zookeeper dependency

You can substitute the Zookeper library with the Charlatan library - a server that accepts and handles Zookeeper requests using the Zookeeper messaging protocol.

Charlatan consists of the following sub-projects:

  • Charlatan-common: contains watch management, session management, common data access object (DAO) interfaces
  • Charlatan-server: contains Zookeeper protocol implementation, Netty-based implementation of the server that accepts and handles zookeeper requests

Charlatan provides the following functionalities:

Node tree support

In order to support create, read, update, and delete (CRUD) operations on nodes, Charlatan DAO interface implementations use the Pega Platform database. Charlatan stores the node tree in the following table:

nodeId node ID
parentId node parent ID
data node data
mode persistent/ephemeral
session session which created the node
... fields with other node properties

Charlatan node tree

Broker watches support

In order to support watches, all updates need to be stored in the database. The existing interface functionality can also be implemented by using Apache Ignite/Hazelcast, or other databases.

Once a broker sets watch on a specific node change, it should receive a one-time update, regardless of whether the change was performed by this or any other broker. Every broker pulls updates created by different brokers from the table and notifies its watches.

For every node CRUD operation, an entry in the following node updates table is created and stored:

id update ID
type event type, for example: Created, Deleted, ChildrenChanged, DataChanged
path full node path
broker broker identifier that generated the change
timestamp event time

The node updates table

Ephemeral nodes support

If a node was created as ephemeral, it should be removed once the broker is off. An ephemeral broker from a previous broker session prevents the alive broker from starting. Removal of ephemeral nodes triggers alive brokers to re-select the leader, if needed, and to start replication operations.

Remove ephemeral nodes when broker is off and notify the rest of the brokers about the updates.

The database is used to monitor broker session states. With every broker’s keepalive message, the last_seen timestamp of the session is updated. See the following pr_data_streamservice_sessions table:

id broker ID
session broker session identifier
last_seen timestamp of the last ping

The broker sessions table

Stale sessions

Every Charlatan server periodically monitors its stale sessions and other Charlatan servers' stale sessions. If a stale session (last_seen timestamp timed out) is found, the server invalidates the stale session; it removes ephemeral nodes created by that session and notifies all connected brokers that are interested in this event.

A Charlatan server can invalidate sessions created by itself and sessions created by a different Charlatan sever.

When a Charlatan server/Pega Platform node crashes, broker sessions are invalidated by running the Charlatan server. If there are no more running Charlatan servers, the sessions are invalidated during the Charlatan server startup.

When a Kafka broker crashes, the Charlatan server invalidates its session.

 

To view the main outline for this article, see Kafka as a streaming service.

Suggest Edit
Did you find this content helpful?

100% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.