Replay
Overview
Did you know that the Kafka, Elasticsearch, Hazelcast and Cassandra services in your Pega deployment will no longer be available in embedded mode as of Pega Infinity '24.2?
This webinar discussed more about transitioning the dependent, third-party services of your Pega implementations to an externalized deployment model while there is still time left for you or your clients to take action.
Per the client advisory notice issued on June 5, 2024, you do not have to externalize Hazelcast until the Pega Infinity '25 release. Embedded Hazelcast will remain available in Infinity '24.2, although retains its deprecated status. See Externalization of services - Support Roadmap for the latest support position.
Presenters
- Gabe Edwards: Senior Fellow, Software Architect: Pega Platform Engineering
- Amar Lingala: Consulting Manager, Pega Consulting
- Felipe Kohn: Technical Solutions Director, Pega Consulting
Agenda
- Why is Pega requiring externalized deployment of these third-party services?
- Can migrating to Pega Cloud solve this problem for me?
- What is involved in transitioning to the externalized deployment model?
- What offerings does Pega have that can help me execute the transition?
Slides and related links
- Slides from this webinar
- Pega Support Center: Externalization FAQs, including links to other relevant articles.
- Brochure: Architecture and Infrastructure Modernization (AIM) - Pega Consulting
Questions and Answers
Here is the Q & A from this session. Thank you to everyone who submitted questions during the live session!
Planning
| Question | Answer |
|---|---|
| Can upgrade of independent services be done in isolation to the Pega Platform upgrade? | Yes! Assuming you are already on an Infinity version that supports externalization. There's a great table at https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalization-of-services.html which shows the Pega versions that are ready for externalization. |
| We are on Pega Cloud. Do we need to take care of this too? | No! On Pega Cloud this is all done for you behind the scenes. It may have already happened for you. |
| Is there cost impact of externalizing these services? | It's hard to estimate cost impacts due to the large variety of enterprise architectures and capabilities. And you may also see cost savings from increased efficiency. If you have questions around Pega licensing, please reach out to your account team. |
| Would we be able to continue with VM Based deployment for Hazelcast even after Infinity 25? | No, you'll want to use the Hazelcast Docker images provided by Pega, deployed into the Kubernetes cluster. |
| For updating to the latest images of these third party providers we still need to follow Pega's guidance on what has been certified compatible and which versions aren't, correct? | That's right. You can see those certified combinations for each third-party service on our docs site. Here's the Elasticsearch one, for example: https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalize-search-in-your-deployment.html |
| Do we get any environments for POC and validating all the changes? | There might be options available on Pega Cloud. Please reach out to your Pega Account Executive and they can help. |
| Does Pega Platform (going forward) support the latest & greatest of those external product versions? Or will there be limitations of product versions? | There are compatibility requirements laid out on the service pages here: https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalization-of-services.html |
| As I understand all of these services can be externalized without having to run Pega in containers? It is not ideal but technically feasible to externalize these services from Pega running on VMs. Is it correct? | Yes, you are correct. All of these microservices can be externalized running on VMs. It does not need to be tied to a move to Kubernetes. |
| Is there is a migration runbook (or something like a guideline) for on-prem apps to move from embedded to externalized Kafka and search? | Instructions for migration of each service can be found here: https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalization-of-services.html. |
| If we need to use Pega Cloud, can our Pega nodes be on-prem, but the externalized services could be on cloud? | No. All infrastructure must be co-located. All on Pega Cloud or all on Client Managed Cloud / On-Premise. |
| Are we exploring the possibility of leveraging Java Virtual Threads for concurrency as an option or alternative for Queue Processors in upcoming releases? | We're always looking for ways to provide more efficiency and scalability! Virtual Threads are an interesting possibility to investigate. |
| Where can I find some instructions or guidelines for deploying Pega application in Kubernetes approach? | Check here: https://docs.pega.com/bundle/platform/page/platform/deployment/client-managed-cloud/pega-kubernetes-architecture.html |
| Which of these four services can be shared (common) across multiple Pega installations? |
Elasticsearch, Kafka and Cassandra can be shared across Pega Instances. Hazelcast is internal to the container (or dedicated to a Pega Instance when running on VMs). Also, DO NOT share these services between Prod and Non-Prod Pega Instances. |
| Is Amazon Aurora on AWS supported? | Yes. See the "Amazon RDS cloud databases" table in https://docs.pega.com/bundle/platform/page/platform/deployment/platform-support-guide/platform-support-guide.html#ariaid-title15. |
| Container images are only "Docker" or are any alternatives to Docker available? | Docker is required. |
| Container images: Pega Platform is it only web nodes or all nodes types? | There are a variety of images provided by Pega. See https://docs.pega.com/bundle/platform/page/platform/deployment/client-managed-cloud/pega-docker-images-manage.html. To answer the question specifically, there's only one Pega image shared for both web vs. util nodes. |
| Do we need to use specific docker images for externalized services from Pega GitHub repository? Or we use any image available on Docker Hub? | Pega provides the SRS image and the Clustering (Hazelcast) image. See https://docs.pega.com/bundle/platform/page/platform/deployment/client-managed-cloud/pega-docker-images-manage.html. Other third party images would come from your vendors. |
Hazelcast
| Question | Answer |
|---|---|
| How do we externalize Hazelcast in VM? | You can find a doc on that topic (and many others) at: https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/deploy-hz-vm-based-cluster-overview.html |
| This talks about deploying Hazelcast using docker command on VM using Pega provided docker image. We don't have docker installed and we want to deployed prcluster_service.war on the VM instead as explained this presentation. |
For a VM Deployment with Client/Server Hazelcast: - The Hazelcast Server nodes will run prcluster_service.war on three JVMs (preferably on three separate VMs) |
| We are working on Infinity 24.1 upgrade from 8.8 and trying to use Client/Server Hazelcast model. Can you please provide instructions on how deploy same using VM Deployment? | You can find a doc on that topic (and many others) at: https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/deploy-hz-vm-based-cluster-overview.html |
| So with the VM option for HZ, this will be no longer valid for the Infinity 25 since those HZ pods will need to be running in same K8S cluster? | That's right. The VM deployment options will be de-supported once Docker+Kubernetes is required. |
| For externalized images on containerized deployment, are they open-sourced or vendor supported. Does Pega support to consume client owned service? | Yes, in fact Hazelcast is the only service provided by Pega. All others must be provided by the client, either services they operate themselves or via a vendor. |
| In the diagram, Hazelcast is still in Pega cluster. Is it embedded one or externalized? | Hazelcast is internal to the cluster but external to the Pega Pods. Embedded Hazelcast would have Hazelcast running on every Pod. |
| Hazelcast was displayed a bit differently than for the other externalized services. As Hazelcast needs to service every single Pega node (e.g. for Cache distribution), where is it deployed? Outside the Pega node cluster? | It's shown differently because it's provided via a Pega image and is tied to a single Pega environment (not shared across Pega environments or other applications). |
Elasticsearch & SRS
| Question | Answer |
|---|---|
| When we externalize elastic search where is the index file stored? Is it still in Pega nodes or they are stored in external pods? | For Elasticsearch, Kafka and Cassandra, the repositories (i..e., index directories, Kafka binaries / data and Cassandra file stores) are local to the external service - NOT within the JVMs local file systems. |
| Is Pega Cloud SRS a single, shared platform? | SRS is deployed as a multi-tenant service on Pega Cloud but there are many instances. It's not a single platform. |
| Pega documentation provides general guidance for Elastic Search externalization. But do you have the specific cloud deployment versions like Azure along with the external storage and non-functional requirements to ensure that the ECK meets the Pega requirements? | General prerequisites and sizing guidelines can be found at https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalize-search-in-your-deployment.html. We do not provide vendor-specific configuration, although the shared community will build up this expertise. |
| Does Pega SRS support VM deployment or does the SRS image require Kubernetes? | Yes, you can find VM-based instructions at https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/connect-new-vm-to-srs.html. |
| When using SRS + externalized Elastic Search, will we still be able to configure Custom Search Properties instances to specify which properties to be searchable / return on search results / list properties? | Yes, Custom Search Properties work in both modes. |
| What is the advantage of using SRS compared to using elastic search externalized directly? Is SRS free? | The Search & Reporting Service (SRS) is Pega software that supports those functions. It's included with Pega. |
| Is SRS a Pega Infinity node? | No, it is its own service. |
Kafka
| Question | Answer |
|---|---|
| I understand external Kafka is what is being recommended for 24.2. Does that mean, internal Kafka will not be supported at all even as a deprecated state? | External Kafka will be required in 24.2. Internal/embedded Kafka will not be supported at all in 24.2. |
| Is Pega is moving away from Kafka for queue processors as well or will this remain with Kafka architecture? | Queue Processors will continue to use Kafka. |
| Regarding data flows and queue processors: the partitions are distributed across available nodes. Whenever a new node comes in the rotation we see that data flows/queue processors shut down for few minutes. Is this down time improved by the externalization? | The re-partitioning pause occurs whether Kafka is externalized or not. This issue has been noted and we are investigating ways to mitigate the pause. |
| How do we differentiate between the internal Pega Kafka configuration and the customer Kafka configuration so we do not have clashes? | All Internal Pega Kafka Topics (QP/Stream Dataset) will have prefix attached. This is configurable via prconfig (Topic Name Pattern) https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalization-of-services.html |
| If I use external Kafka, will there not be any stream node which manages Queue Processors? |
With external Kafka, the Pega Instance should NOT and MUST NOT have STREAM nodes. Queue Processors would run on Background Processing nodes which would interact with the external Kafka. |
Cassandra
| Question | Answer |
|---|---|
| What is needed to access external Cassandra from VM deployment? | Connecting to an external Cassandra from a VM Deployment is managed with JVM arguments to define the connection string to the Cassandra instance. |
Support
| Question | Answer |
|---|---|
|
If there is a security vulnerabilities in externalized service software version, Is Pega going to identify and notify those issues? And what if the security of any image is not meeting the requirement of the organization? |
For the Hazelcast images, yes, Pega will provide support and updated versions to address vulnerabilities. For the others, primary responsibility will be in the client's (or their vendors') hands. Pega does not own or support the third-party services, so you will want to work directly with your vendors and/or the open-source community. Note that Hazelcast is an exception to this since it is licensed and distributed by Pega. We will notify of any vulnerabilities in Pega images, including Hazelcast, and turn around fixes in new images. We take the security of these images very seriously. Our images are scanned at every step of the lifecycle and we release new images with the latest functional and security fixes regularly. Please reach out to Pega Support if you find any vulnerabilities or other problems with any Pega image. |
| 1) Externalizing can impact the performance as this external services can be on a different data center 2) security concerns as sensitive data is moving to external service and overhead of managing the security. Are these true? | Both true statements and things to watch for. We provide instructions for each service on how to encrypt and protect your data. |
| If a Pega app deployed in a private cloud is going to use these external services, what protocols or auth mechanisms should be followed? | You can find best practices and instructions on how to set up security protocols and mechanisms under each service listed at https://docs.pega.com/bundle/platform/page/platform/deployment/externalization-of-services/externalization-of-services.html. |
Operations
| Question | Answer |
|---|---|
| How and when is the data written and made available to the external services? Will there be any issues with the increased requests made at a time such as no data returned/timeout? | This varies from service to service and application to application. The promise of service-based systems includes the ability to autoscale to react to increased traffic and data volumes in order to maintain response times. We've had great success with this approach on Pega Cloud. |
| Does it mean that sequence to shut down and startup of different node type will be changed? As of today, stream nodes needs to be shut down first and first node to start. | VM Startup sequence w/ External Services: Hazelcast, SRS, Background Processing & WebUser. Shutdown sequence is reversed. Kafka, Elasticsearch and Cassandra would be up all the time (similar to the Database). |
| Is prclusterservice.war available for download? | prclusterservice.war is available in the Pega Media under the archives folder in all versions and patches of Pega 'Infinity up through 24.1. |
| We were able to retain Pega logs by using an external directory however we could not get to do the same for Tomcat logs (ex: Catalina), Is the template now evolved to retain Catalina logs? | This is managed by updating the Tomcat logging. Properties within the conf folder of Tomcat. |
Roadmap
| Question | Answer |
|---|---|
| Will AWS ECS using Fargate be officially supported in 24.2 or 25? | Fargate is not on the roadmap at the moment. |
| With externalization, are there plans to open the tooling that can be used with Pega (e.g., instead of using Hazelcast, opting for REDIS)? | We're constantly evaluating different technologies and keeping options open when possible. There are no current plans to support Redis as a Hazelcast substitute. |
| For Azure deployments is there a plan to support Azure event hub in place of Apache Kafka? | We don't currently support Azure Event Hub but the question has been raised a few times. It's something we're investigating but not currently on the roadmap. |
AIM
| Question | Answer |
|---|---|
| Does the AIM workshop help client do a POC in a sandbox environment? | Yes. AIM can help set-up on-prem and/or client cloud environments with any Pega supported platforms. |
| How do I get started with the AIM workshop? | Please contact your Pega Consulting Solutions Executive (CSE) or send an email to [email protected] or [email protected] |