Using the data import File Listener
The data import File Listener uses multi-threading to increase the throughput. This is faster than the data import wizard, which uses a single thread. Always use the data import File Listener when importing large data volumes (millions of rows).
Pega Sales Automation comes with a file listener for the following entities:
- Lead (individual and business)
- Opportunity (individual and business)
- Customer activity
- Account (in 8.2 version only)
The data import File Listener data import process is divided into the following steps:
- Import recommendations
- Preparing the data
- Configuring the data import File Listener
- Running the data import File Listener
For best performance, while using the data import File Listener, keep in mind the following recommendations:
- Before starting the import for all your records, import a few records to start with and fix any issues.
- The size of the File Listener base upload should not exceed 1 million records in a single file.
- Batch size value recommended for upload is 1000.
- Use Add Only mode for the initial data migration.
- To ensure a maximum parallel processing, there must be as many input files for the file listener as there are threads, because each thread is processing one file at the time.
- High-volume import process should not include generating unique IDs. Include pyID for work object records in the import file. Update the unique ID stored in the pc_data_uniqueID table after importing contacts. Set the value in the table to the highest pyID in the contact table.
- Database indexes improve query performance, however, when updating a database table, indices can have a negative impact if there are too many indices or a database table is very large. A table with 100,000,000 rows can cause lower performance when modifying the table. Remove non-essential indices during the import phase.
Preparing the data
Data import File Listener uses the same underlying APIs as the data import wizard to process files placed in predetermined folders on the server. Importing data by using the data import File Listener requires templates. It is recommended to use the data import wizard to make any template changes prior to using the file listener. For more information, see Preparing data and Pega Sales Automation sample data templates.
Configuring the data import File Listener
- For on-premises configuration, perform the following steps:
- In the navigation panel of App Studio, click Settings > Application Settings.
- In the File Listener Configurations section, enter the base folder in the File Listener source location and email address to which you want to receive notifications.
You can use the following configuration as an example.File Listener example configuration
- Optional: To improve performance and to disable creating audit history, in the Initial Data Migration check box.
section, select the
After the migration is completed, clear the check box to generate audit records.
- Click .
- Optional: If you want to modify the default template and purpose configuration, in the navigation panel of Dev Studio, click App,and then search for and open the ResourceSettings data transform.
You can use the following configuration as an example.Resource settings example configuration
By default, the data import File Listener is configured with SA_<name of objects> as a template and Add or update as a purpose.
- For Pega Cloud configuration, perform the following steps:
It is recommended to use SFTP server implementation.
- In the header of Dev Studio, search for and select the storage/class/defaultstore:/type dynamic system setting (DSS).
- In the Value field, enter
- Click .
- In the header of Dev Studio, search for and select the FileListenerSourceLocation and the base folder in the File Listener source location.
- Click .
Running the data import File Listener
- In the navigation panel of Dev Studio, click Records > Integrations-Resources > File Listener, and then open a listener that you want to run.
- Under Block startup check box. section, clear the
- Under section, enter the source file format.
.csv and .txt formats are supported.
- Under the section, set the number of threads per node to the number of CPUs on that node.
The number of threads per node should be the same as the number of CPUs on that node. By doing this, you improve the performance of the initial load.
- Click .
- In the header of Dev Studio, click the Admin Studio. menu and then click
- In the navigation panel of Admin Studio, click Resources > Listeners, and then open a listener that you configured.
- Under section, enter user name and password.
What to do next
After an entire file is processed, output files are created in the source file location that you specified in App Studio. The output file lists failed records and information about each error. The data import results summary is emailed to the notification email addresses that are listed as part of the File Listener configuration process.
Published August 31, 2018 — Updated March 22, 2019