Version -  Available in Pega Platform version 24.2 and later.

Ingestion Strategy - 

After configuring the application, the next step involves strategizing the Data Ingestion process for the respective data classes. Rather than developing all components from the ground up, the Pega Common Application offers predefined rules to facilitate the same.

Data ingestion in applications typically consists of two phases: initial migration and batch updates, with update frequency varying by project.

1. Initial Migration- One time Activity

Pega Common Application supports initial migration of data with the help below pre-configured rules-

1. Excel Templates for each entity

We could edit an existing template to include properties that were added during implementation or to map a column from the file to a different property than in the original template.

The sole manual task required in this process is to ensure that your data conforms to the entity structure, which will facilitate the migration of existing customer information into the system.

2. Once the files are ready, you have 2 options

    a. Manually uploading the data using Data Import Wizard – For Small Data Sets.

You can manually import the data one by one using the wizard

AppStudio --> Tools --> Data Import Wizard 

Select the file, entity name and Purpose (Add\Update) to start the process.

The wizard uses the same excel template to map the source to entity field types. In case of any changes\ corrections you can edit the mapping.

You could either skip validation or continue with validation. Post which the data is loaded into the system.

b. Using OOTB file listeners – For Large Data sets

Like excel templates, each entity also has a file listener associated with it. You just need to enable the listener and add your file source location.

You can find the detailed steps to enable them in this documentation -  https://docs.pega.com/bundle/common-data-model/page/common-data-model/implementation/configure-file-listener.html

Steps for Migration

• Import a small batch of records first to identify and resolve any issues before bulk uploading.
• Load related entities in the correct order: upload each entity, then establish their relationships. You can find the correct order here - https://docs.pega.com/bundle/common-data-model/page/common-data-model/implementation/load-data.html

Applying business logic when importing

Although the wizard\listener will only run the basic validations while importing, you could still extend and add your business logic before the data is moved to your system.

You can find these extension points here - https://docs.pega.com/bundle/common-data-model/page/common-data-model/implementation/apply-business-logic.html

In summary, by utilizing either option, you can efficiently import your existing customer and application data into your application with minimal configuration adjustments.

2. Batch Updates- Daily\Weekly

Once the application is operational, it may be necessary to maintain up-to-date data. This can be achieved by conducting batch updates to insert or modify information within the entities.

The OOTB file listeners can be extended to maintain this data using data flow. It may also be necessary to transform the data prior to processing the file listeners.

Based on the integration contract between the data source and the application, multiple approaches are available:

1. Align the data to match the file template and send it to the listener (no customization required; however, aligning the data is an external task).
2. If data alignment is not preferred, process the existing data by using a staging table and implementing logic as a template before sending it to the listener (some customization is needed; the effort occurs within the application).
3. Modify the out-of-the-box template and implement custom logic to ingest the data (full customization).

Note: For file listener usage recommendations, see the details at: https://docs.pega.com/bundle/common-data-model/page/common-data-model/implementation/file-listener-import-recommendations.html