Article

Using the data import File Listener

The data import File Listener uses multi-threading to increase the throughput. This is faster than the data import wizard, which uses a single thread. Always use the data import File Listener when importing large data volumes (millions of rows).

Pega Sales Automation comes with a file listener for the following entities:

  • Operator
  • Contact
  • Household
  • Lead (individual and business)
  • Opportunity (individual and business)
  • Organization
  • Task
  • Customer activity
  • Territory
  • Account (in 8.2 version only)

The data import File Listener data import process is divided into the following steps:

  1. Import recommendations
  2. Preparing the data
  3. Configuring the data import File Listener
  4. Running the data import File Listener

Import recommendations

For best performance, while using the data import File Listener, keep in mind the following recommendations:

  • Before starting the import for all your records, import a few records to start with and fix any issues.
  • The size of the File Listener base upload should not exceed 1 million records in a single file.
  • Batch size value recommended for upload is 1000.
  • Use Add Only mode for the initial data migration.
  • To ensure a maximum parallel processing, there must be as many input files for the file listener as there are threads, because each thread is processing one file at the time.
  • High-volume import process should not include generating unique IDs. Include pyID for work object records in the import file. Update the unique ID stored in the pc_data_uniqueID table after importing contacts. Set the value in the table to the highest pyID in the contact table.
  • Database indexes improve query performance, however, when updating a database table, indices can have a negative impact if there are too many indices or a database table is very large. A table with 100,000,000 rows can cause lower performance when modifying the table. Remove non-essential indices during the import phase.

Preparing the data

Data import File Listener uses the same underlying APIs as the data import wizard to process files placed in predetermined folders on the server. Importing data by using the data import File Listener requires templates. It is recommended to use the data import wizard to make any template changes prior to using the file listener. For more information, see Preparing data and Pega Sales Automation sample data templates.

Configuring the data import File Listener

  1. For on-premises configuration, perform the following steps:
    1. In the navigation panel of App Studio, click Settings > Application Settings.
    2. In the File Listener Configurations section, enter the base folder in the File Listener source location and email address to which you want to receive notifications.

      You can use the following configuration as an example.

      Thumbnail
      File Listener example configuration
    3. Optional: To improve performance and to disable creating audit history, in the File Listener Configurations section, select the Initial Data Migration check box.

      After the migration is completed, clear the Initial Data Migration check box to generate audit records.

    4. Click Save.
    5. Optional: If you want to modify the default template and purpose configuration, in the navigation panel of Dev Studio, click App,and then search for and open the ResourceSettings data transform.

      You can use the following configuration as an example.

      Thumbnail
      Resource settings example configuration

      By default, the data import File Listener is configured with SA_<name of objects> as a template and Add or update as a purpose.

  2. For Pega Cloud configuration, perform the following steps:
    It is recommended to use SFTP server implementation.
    1. In the header of Dev Studio, search for and select the storage/class/defaultstore:/type dynamic system setting (DSS).
    2. In the Value field, enter filesystem.
    3. Click Save.
    4. In the header of Dev Studio, search for and select the FileListenerSourceLocation and the base folder in the File Listener source location.
    5. Click Save.

Running the data import File Listener

  1. In the navigation panel of Dev Studio, click Records > Integrations-Resources > File Listener, and then open a listener that you want to run.
  2. Under Listener nodes section, clear the Block startup check box.
  3. Under Source properties section, enter the source file format.

    .csv and .txt formats are supported.

  4. Under the Listener properties section, set the number of threads per node to the number of CPUs on that node.

    The number of threads per node should be the same as the number of CPUs on that node. By doing this, you improve the performance of the initial load.

  5. Click Save.
  6. In the header of Dev Studio, click the Switch Studio menu and then click Admin Studio.
  7. In the navigation panel of Admin Studio, click Resources > Listeners, and then open a listener that you configured.
  8. Under Requestor login section, enter user name and password.

What to do next

After an entire file is processed, output files are created in the source file location that you specified in App Studio. The output file lists failed records and information about each error. The data import results summary is emailed to the notification email addresses that are listed as part of the File Listener configuration process.

    Published August 31, 2018 — Updated March 22, 2019

    Related Content

    Have a question? Get answers now.

    Visit the Pega Support Community to ask questions, engage in discussions, and help others.