Configuring a file service and file listener to process data in files
Use file services to read and process the data in files that were exported from another system or copied to a specific folder in a file repository. The file listener monitors your file directory and calls the file service so that you can easily import the contents of character-based files into your Pega Platform system.
- Determine the source location, that is, the location of the files to be processed.
- Determine how the listener filters the files to process, for example, whether it processes every file that it finds, or only those files with a specific file extension or some other distinguishing characteristics.
- Determine how the listener identifies the records to process, for example, whether there is one record in each file or multiple records in each file, and the data structure of the records.
- Determine how the file service processes files and choose the rules that implement that process; for example, the file service might create work objects or add incoming data to existing work objects.
- Ensure that the Pega Platform server has read and write access to the file system or repository where the input files arrive so that the listener can move and rename files.
The file listener monitors the file directory and calls the file service when files arrive. The file service might use a parse rule for XML, structured, or delimited data, or a JSON data transform to read the file and map the data into fields on the clipboard. A service activity processes the data, typically by creating a case from the file data and launching a flow.
Both the file listener and the file service have a role in processing the files:
- When files that match the listened for pattern arrive, the file listener calls the file service to process the file data.
- The file service processes the file by using the rules that you configured for it.
- Based on the file listener configuration, the file listener either deletes
the file or moves it to the
work_<name of listener>/completedsubdirectory.
- The listener also creates a
reportssubdirectory to hold report files that contain the processing results.
To configure the file service and file listener, perform the following steps:
Create the file service rule to process your files. The most common use case is to use a Parse Delimited rule to map properties to the fields in a CSV file. For more information, see Creating a Parse Delimited rule.Additionally, you can use the following rules to configure a file service to process your files:
- Parse XML – Use a Parse XML rule to import XML files. For more information, see XML Stream rules.
- Parse Structured – Use a Parse Structured rule to process fixed-format flat files. For more information, see Creating a Parse Structured rule.
- JSON data transform – Use a JSON data transform to process data that is in JSON format. For more information, see Data transforms.
Create a service activity to configure the file service to perform an action on each record of the file, one record at a time.In addition to parsing the CSV file, the service activity typically creates a work object for each record. When you create the service activity, make sure that you enter the following values:
- In the Method field, enter Call
The svcAddWorkObject function creates a work object by using the flow rule specified in the Flow Type field. For example, you could call the BatchNewPolicy flow type to add the created work objects to a workbasket for processing new policies.
- In the workPage field, enter pyWorkPage.
For more information about creating a service activity, see Creating an activity.
- In the Method field, enter Call svcAddWorkObject.
Create the service package that is required to group related Service File rules.For more information, see Service Packages.
If you are using multithread processing, create a service request processor. A request processor enables asynchronous processing by defining a queue in Pega Platform that can have items placed in it for processing.For more information, see Service Request Processor data instances.
To define how the system processes the files that the file listener pulls into your application, create a Service File rule. For more information, see Service File rules.
Create the file listener that monitors your file directory and calls the file service to process the data in the file.
- For Pega Cloud or on-premises users on systems running 7.2.2 or 7.3.1, see Configure a file listener associated with a storage destination in the archive on Pega Community.
- For Pega Cloud or on-premises users on systems running 7.4 and higher, see Configuring a file listener associated with a storage repository.