Creating and sourcing large data pages to store large reference data in offline-enabled applications

This tutorial describes how to set up a large data page for offline-enabled applications. Using large data pages to store reference data in your custom mobile app improves the performance of the custom mobile app. Only individual records of the large data page that have actually changed are synchronized, not the entire content. Furthermore, for efficient memory management, only records that are needed to render a specific screen on the mobile device are loaded to memory, which leaves the remaining parts of the reference data in the device's permanent store.

Large data pages can be automatically sourced from report definitions. You define them in Pega Platform™ just like you would define any other data page. After the data page is created, you mark it as a large data page in Pega Platform. Large data pages can also be sourced from a connector, activity, or data transform, but you must create a custom JavaScript populator function as described in the following steps. If you are setting up a large data page that uses any of the unsupported functionality listed in Additional considerations, you must also create a custom JavaScript populator function.

To create and source large data pages, complete the following tasks:

This tutorial takes approximately 1 hour to complete.


Before you start this tutorial, review the following prerequisites and perform the following tasks:

  • Make sure that you have experience creating and using list-structured data pages on Pega Platform. For more information, see Data pages.
  • Create a simple Pega Platform offline-enabled custom mobile app that uses a list-structured data page that links to a user interface control such as a drop-down list.
  • Make sure that the datapages/addReportDefinitionToDatapage Dynamic System Setting that is owned by the Pega-Engine ruleset is set to true or not set. Otherwise, you must create a custom JavaScript function.

If you plan to use the custom populator functionality, review the following prerequisites:

  • Make sure that you are proficient in writing JavaScript code.
  • Read Custom populator JavaScript functions for large data pages, which describes the populator JavaScript functions to use for large data pages.
  • Make sure that you are proficient in SQL and familiar with how to use the SQLite syntax to make queries.

Creating a data page automatically

First, create a data page to hold a large amount of data. After you create the data page, configure the large data page for use in an offline-enabled custom mobile app. In the following example, a large data page, named D_Movie, contains columns that provide the following movie information: title, release date, whether it is a sequel, description, modification date and time, and short description. The data page is sourced from a report definition.

  1. In your report definition, add the pyModificationDateTime column to the results. The column is already predefined in the base class (@baseclass), which is the topmost class in the class hierarchy. All other classes are derived from the base class.
    The pyModificationDateTime column returns the time when the record was last modified in the data source. The data source must update the value of the pyModificationDateTime column at each add, update, or delete operation. The values in this column are expressed in a DateTime format (yyyyMMdd'T'hhmmss.SSS z) starting from midnight, January 1, 1970 UTC (19700101T000000.000 GMT).
  2. Add the pyIsRecordDeleted column to the result (for example, a report definition). This column is already predefined in the base class (@baseclass).
    The pyIsRecordDeleted column returns true only for records that were removed. The data source should always maintain removed records tracking in order to return appropriate values. You can remove rows manually; however, you must force synchronization of all the access groups that use the data source in offline-enabled apps.
  3. Create a data type called Movie that includes the Title, ReleaseDate, IsSequel, Description, and MovieShortDescription fields.
  4. Optional: If the source of the data type is local data storage, in the Records tab click Configure source to create a dedicated table for the class, and add the pyModificationDateTime and pyIsRecordDeleted properties to the list. When you do this, the pyModificationDateTime and pyIsRecordDeleted properties are optimized.​​
  5. Create a data page called D_Movie.
  6. In the Data page definition section, select List from the Structure list.
  7. In the Object type field, select or enter the name of the new data type that you created in step 1, for example, Cars-Dinoco-Data-Movie.
  8. In the Scope field, select Node.
  9. In the Data sources section, select Report Definition from the Source list.
  10. In the Name field, enter a name for the report definition for the data page, for example: MovieRD.
  11. Click the Open icon to the right of the Name field to configure the report definition:
    1. In the Edit columns section, make sure that the following columns are present. You must add the .pyIsRecordDeleted column and the .pyModificationDateTime column manually.​​
    2. Click the Parameters tab and add a new Date Time type parameter called pyModificationDateTime.​​
    3. Return to the Query tab and define a filter condition that uses the pyModificationDateTime column. Define at least one other condition and additional filters, if necessary.
      During the initial synchronization, the clients receive all the data after it is filtered by the data source on the server side, regardless of the value in the pyModificationDateTime column. Subsequent synchronizations return only rows that are newer than the value of the pyModificationDateTime parameter. You must not modify the pyModificationDateTime > param.pyModificationDateTime filter condition that is required by a large data page; therefore the use of Use Null if Empty or default values for the pyModificationDateTime parameter is not supported. For more information, see pyModificationDateTime parameter optimization.
      All parameter-based filter conditions are ignored on the server. On the client, the pyModificationDateTime > param.pyModificationDateTime filter condition is ignored, while all other filter conditions are applied (either parameter-based or hardcoded by means of providing a string value).
    4. Optional: To bind your data with data from another class, click the Data Access tab and enter a data page name in the Prefix field to declare a class join.
      You can add only one join for a specific data page. Other options on the Data Access tab are ignored on the client.
    5. Optional: Click Edit conditions to configure a filter to apply to the join.
  12. Click Save.

Declaring a data page as large

After you create a data page that holds large data records in your custom mobile app, declare it as a large data page.

In the pyDataPageWhiteListForOffline rule, declare both the target large data page, as well as any source large data pages, if applicable. Create a separate entry in this rule for each large data page. Large data pages that are sourced from data sources other than report definitions must always contain at least one record.
  1. In Dev Studio, click Mobile > Offline.
  2. Click the first Modify whitelist link to edit the pyDataPageWhiteListForOffline rule.
  3. In the HTML source field, declare the D_Movie data page as large.
  4. Optional: To improve system performance, for each large data page that is used in Pega Platform, add a list of column indexes after the declaration made in the HTML source field, as shown in the following example:
    D_Movie;large;[["index1_column1","index1_column2","index1_column3"],["index2_column1","index1_column2","index2_column3 COLLATE NOCASE"]]

  5. Individual column index names must be specified within double quotation marks and must match the SQL queries that are performed on a specific large data page. Add COLLATE NOCASE after an indexed column name to make queries with LIKE statements faster and case insensitive. For more information, see Index optimization.
  6. Click Save.

Optional: Creating a data page based on a custom JavaScript populator function

If your implementation requires functionality that is listed as unsupported in the Additional considerations section or if a large data page is sourced from a connector, activity, or data transform, create a data page, declare it as large, and define a custom populator for the JavaScript function that performs an operation on the large data page by using query statements. For details about populator JavaScript functions and examples for the above use cases, see Custom populator JavaScript functions for large data pages.

A custom populator takes precedence over the automated process.

Initially, you need to create a large data page and then create a JavaScript custom populator function called moviesCustomFunction(). The target data page for this example is D_Movie. The query operation selects the Title, Release date, Description, and Is sequel columns from this existing large data page. No query parameters are used because there is no need to pass any parameters to the SQL WHERE clause. To use the new custom populator function called moviesCustomFunction() in your Pega Platform application, save the JavaScript code as a text file. Then include the text file called movies_datapage_api in the pypega_ui_userscripts_offline Static Content Bundle rule. This action ensures that the JavaScript code that you created is accessible within the Pega Platform offline-enabled mobile app.

  1. Follow the Creating a data page automatically procedure. Skip steps 8 through 10.
  2. Follow the Declaring a data page as large procedure.
  3. Add a JavaScript code as a text file:
    1. In the navigation panel, right-click the name of your case and click Create > Technical > Text File.
    2. In the Label field, enter movies_datapage_api as the name for the new text file.
    3. In the App Name (Directory) field, enter webwb.
    4. In the File Type (extension) field, enter js.
    5. Create and open the new text file.
    6. In the File source field, paste your JavaScript code, similar to the following snippet:
       var moviesCustomFunction = function(parametersMap, clientStore, onSuccess, onFailure) { var targetDatapageName = "D_Movie"; var queryParameters = []; var query = "SELECT Title, ReleaseDate, Description, IsSequel FROM D_Movie"; clientStore.runQuery(query, queryParameters, targetDatapageName, onSuccess, onFailure); } pega.ui.ClientCache.registerLargeDatapage("D_Movie", moviesCustomFuction);
    7. Click Save.
  4. Attach the text file to the Pega Platform application:
    1. In Dev Studio, search for pypega_ui_userscripts_offline.
    2. Click pypega_ui_userscripts_offline to edit the UI-Kit rule.
    3. In the App name field, enter webwb.
    4. In the File extension field, enter js.
    5. In the File name field, enter movies_datapage_api.
    6. Click the arrow next to the Private edit button and click Check out to branch to make sure that the pypega_ui_userscripts_offline rule is checked out to the branch for the current application.
  5. Click Save.

Additional considerations

To use large data records in your custom mobile app, keep in mind the following information:

  • Large data pages that are sourced from a data source other than a report definition must include at least one record. All key columns that exist in a record must be returned.
  • For performance reasons, do not reuse the same data pages if they are expected to hold different data. This does not mean that you must always define a separate data page for each user interface element. It is a good practice to define separate data pages more often than you would in the case of Pega Platform desktop applications.
  • On the Android platform, do not define your property as softflagINTERNAL. Under the iOS platform, do not define your property as softflag.INTERNAL@.
  • Do not use the Do Not Reload When refresh strategy when it is used with the pyModificationDateTime time stamp. However, the other refresh strategies on the Load Management tab of the Data Page rule form are supported.
  • Do not use target data pages as source data pages for other target data pages.
  • You can use any of the following functionality in your report definition, but these items are ignored on the client:
    • Select values button when defining your filter logic
    • Aliases (columns are accessible by a name defined in the report definition's Column source field)
    • Calculations
    • Default parameter values
    • Use Null if Empty strategy
  • If you create on one screen multiple elements of the user interface that interact with the same data source, but use different query parameters, make sure that they all have separate target data pages defined. An example is two lists of elements filtered by two separate categories are those that can be fixed or selected individually.
  • If you source a data page by using the same parameters for multiple user interface elements, make sure that they reuse the target data page. An example is for each element list to have a couple of columns displaying different properties of the elements.
  • If you use SQL queries, keep in mind that the function that generates SQL queries is cached on the client; the generated SQL queries themselves are not cached. Also, the WHERE clause of an SQL query might differ depending on the number of provided parameters. If a parameter is not provided, a part of the filtering logic that uses this parameter is ignored.
  • If your existing data page uses pzInsKey key, and you are creating a new report definition, define the pzInsKey field in the report definition manually.
  • If you are creating a new data page, define its keys in the data type class.

Index optimization

When declaring a data page as large, you can define indexes for the TEXT column of a large data page in the pyDataPageWhiteListForOffline rule and add one of the collating sequences, for example, COLLATE NOCASE, COLLATE BINARY, or COLLATE RTRIM. Collating sequences optimizes the indexes by making the LIKE statements in a query faster. The following example shows how to define a collating sequence for an index column of the data page in this rule:

D_myDatapage;large;[["firstName"], ["firstName COLLATE NOCASE"]]

When you add COLLATE NOCASE after an indexed column, the query with the LIKE statement is faster and case insensitive. To use SQL queries with the equal operator, you need to add another index. The following example shows two ways of creating indexes:
CREATE INDEX myIndex ON D_myDatapage (firstName); // This makes equal operator queries fast and case sensitive CREATE INDEX myIndex ON D_myDatapage (firstName COLLATE NOCASE); // This makes LIKE queries fast and case insensitive

Based on your requirements, you can create one or both indexes to speed up queries. The add operation takes more time because of the cost of creating the second index. For more information, see LIKE optimization.

Consider the following information when you use the source properties in a report definition in filters:

  • You must create indexes manually in the list of offline-supported data pages.
  • The filtering relationship and the filter order determine whether a specific index works or not.
  • If you do not provide a report definition parameter that is used in a filter and no default or null value is set, the filter is ignored.

pyModificationDateTime parameter optimization

The main use of the pyModificationDateTime parameter by the synchronization process is to limit the time that is needed to update the data stored on a user's device. The time spent fetching the source data should depend on the amount of data that is returned rather than the size of the entire source database.

Consider the following information about the pyModificationDateTime parameter:

  • If your source is a database table, create an index for the pyModificationDateTime column.
  • The Filter conditions to apply field of filters that you edit in the Query tab must contain the pyModificationDateTime filter and another condition that contains any supported filter logic.
  • When editing the conditions of a join, you cannot use the pyModificationDateTime parameter as a filter condition or source column.
Suggest Edit

100% found this useful

Related Content

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.