Support Article
KafkaException occurs when sending messages to Kafka bus
SA-66833
Summary
The requirement is to use custom Kafka modules in Pega 7.3. Third-party Kafka library JAR files are referenced in prconfig.xml.
KafkaException occurs when using the Pega application to send messages to a Kafka bus.
Error Messages
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[catalina.jar:?]
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:319) ~[catalina.jar:?]
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:88) ~[catalina.jar:?]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[catalina.jar:?]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) ~[tomcat-coyote.jar:8.5.15-dev]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-coyote.jar:8.5.15-dev]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) ~[tomcat-coyote.jar:8.5.15-dev]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) ~[tomcat-coyote.jar:8.5.15-dev]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-coyote.jar:8.5.15-dev]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.5.15-dev]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) ~[kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188) ~[kafka-clients-0.10.0.1.jar:?]
at xx.yyy.pega.distribution.publisher.kafka.KafkaPublisher.initialize(KafkaPublisher.java:50) ~[pega-kakfa-client.jar:?]
at com.pegarules.generated.SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.SendKafkaMessage01_01_03(SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.java:109) ~[?:?]
... 74 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.clients.producer.internals.DefaultPartitioner is not an instance of org.apache.kafka.clients.producer.Partitioner
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:203) ~[kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:225) ~[kafka-clients-0.10.0.1.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188) ~[kafka-clients-0.10.0.1.jar:?]
at xx.yyy.pega.distribution.publisher.kafka.KafkaPublisher.initialize(KafkaPublisher.java:50) ~[pega-kakfa-client.jar:?]
at com.pegarules.generated.SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.SendKafkaMessage01_01_03(SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.java:109) ~[?:?]
... 74 more
2017-10-02 15:11:35,479 [abcdefg] [STANDARD] [ ] [Fraud:01.02] (internal.mgmt.Executable) ERROR a_host|aa.bb.ccc.dd.ee an_operator - Exception
java.lang.RuntimeException: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at com.pegarules.generated.SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.SendKafkaMessage01_01_03(SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.java:115) ~[?:?]
at com.pegarules.generated.SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.invoke(SendKafkaMessage_010103_BLkrxruaR6FmX4XnkjcT0Q.java:73) ~[?:?]
at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:222) ~[prprivate.jar:?]
at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:119) ~[prprivate.jar:?]
at com.pega.pegarules.generation.internal.library.LibraryFunctionUtilityImpl.resolveMethodCall(LibraryFunctionUtilityImpl.java:2923) ~[prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.Executable.resolveMethodCall(Executable.java:11215) ~[prprivate.jar:?]
at com.pegarules.generated.activity.ra_action_sendkafkamessage_bc6005b35aafc2b77e21cb27f9391736.step2_circum0(ra_action_sendkafkamessage_bc6005b35aafc2b77e21cb27f9391736.java:220) ~[?:?]
at com.pegarules.generated.activity.ra_action_sendkafkamessage_bc6005b35aafc2b77e21cb27f9391736.perform(ra_action_sendkafkamessage_bc6005b35aafc2b77e21cb27f9391736.java:87) ~[?:?]
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3586) ~[prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10770) ~[prprivate.jar:?]
at com.pegarules.generated.activity.sh_action_sendinfotokafkaeventbus_6f374bd572afb974baf3bb96b6086c03.step5_circum0(sh_action_send
Steps to Reproduce
- Update prconfig.xml to include the third-party Kafka library JAR files and restart the server.
<env name="compiler/defaultPaths" value="/data/share/sharedlayer/lib/apitrust-api-00.00.08.001.jar:/data/share/sharedlayer/lib/bcpkix-jdk15on-1.55.jar:/data/share/sharedlayer/lib/bcprov-jdk15on-1.55.jar:/data/share/sharedlayer/lib/hamcrest-core-1.3.jar:/data/share/sharedlayer/lib/commons-configuration-1.10.jar:/data/share/sharedlayer/lib/pega-kakfa-client.jar:/data/share/sharedlayer/lib/toolkit-kafka-serialization-1.11.0-SNAPSHOT.jar:/data/share/sharedlayer/lib/toolkit-encryption-1.11.0-SNAPSHOT.jar:/data/share/sharedlayer/lib/protobuf-java-2.6.1.jar:/data/share/sharedlayer/lib/commons-lang-2.6.jar:/data/share/sharedlayer/lib/lz4-1.3.0.jar:/data/share/sharedlayer/lib/snappy-java-1.1.2.6.jar:/data/share/sharedlayer/lib/slf4j-api-1.7.25.jar:/data/share/sharedlayer/lib/commons-lang3-3.6.jar:/data/share/sharedlayer/lib/jcl-over-slf4j-1.7.25.jar:/data/share/sharedlayer/lib/commons-codec-1.10.jar:/data/share/sharedlayer/lib/gson-2.8.2.jar:/data/share/sharedlayer/lib/kafka-clients-0.10.0.1.jar"/>
- Implement a custom connector to post a message to a Kafka queue.
Root Cause
Pega 7.3 has an inbuilt kafka-clients-0.10.0.1.jar library. The custom library causes a conflict with the inbuilt library.
Resolution
Perform the following local-change:
Disable the out-of-the-box kafka-clients-0.10.0.1.jar in the application as below:
- Shut down the servers.
- Backup the PegaRULES database.
- Run the below SQL Update:
update pr_engineclasses set pzcodeset='pega-enginecodenouse' where pzjar = 'kafka-clients-0.10.0.1.jar';
commit;
- Restart the servers.
Published December 13, 2018 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.