Close popover

Table of Contents

Providing feedback to text analytics models

Increase model accuracy by correcting the sentiment value, intent, category, or entity classification of the analyzed text. After collecting the corrected records, retrain the corresponding model to classify similar records more accurately in future analyses.

Perform the following tasks to learn about providing feedback to text analytics models:

Providing feedback through an API

Create a feedback loop for your models by using the pxCaptureTAFeedback activity. Through this activity, you can manually correct unexpected or inaccurate text analysis outcomes, for example, by updating sentiment value from positive to negative or changing the assigned topic. You can then update the model that performed the analysis by feeding it the corrected outcomes.

  • For categorization models, provide your feedback in the ActualResult field. The actual result can be a sentiment value (for example, Negative), an intent type (for example, Complaint), or topic assignment (for example, Customer Support > Phone).
    See the following example for reference:
    "Feedback to text categorization models"
    Providing feedback to categorization models through testing
    • To submit the record for a review before the model includes it as feedback, select Mark For Review.
    • To exclude the record from being included as feedback to the model, select Is Test.
  • For text extraction, provide your feedback in the form of annotated text in the Text field, for example, <START:PERSON> John <END> works at <START:ORGANIZATION> Pegasystems <END>. He lives in <START:LOCATION> Boston <END>.
    See the following example for reference:
    "Feedback to text extraction models"
    Providing feedback to text extraction models through testing

    While entering your feedback in the Text field, you must preserve all correct annotations. Otherwise, the model treats all removed annotations as negative feedback.

Feeding the feedback data to text analytics models

You can update a text analytics model by extending the training data with the recorded feedback. Updating the model with carefully recorded and reviewed feedback ensures that the model gradually improves its prediction accuracy. You can rebuild the model by including the feedback in the training data or you can download the feedback data and move it to another environment.

Downloading feedback data

As a result of the pxCaptureTAFeedback activity, the feedback records are stored in the applicable model and available for download. You can export these records and move them between environments.

  1. In the navigation panel of Prediction Studio, click Predictions.
  2. Open a text analytics model.
  3. In the model record, select Download > Feedback data (zip).
    Downloading the recorded feedback

Updating models through the API

You can use the captured feedback data to update the corresponding model. The update process involves rebuilding the model so that the model learns from the feedback you provided.

  1. In the navigation panel of Prediction Studio, click Predictions.
  2. In the tile that represents the model that you want to update, click More and then select Update > <language>
  3. Proceed with on-screen instructions until the Source selection step.
  4. In the Source selection step, ensure that the Include recorded feedback option is selected.
  5. Optional: To examine the feedback data, click Preview.
    Updating a model with feedback recorded through the API
  6. Proceed with on-screen instructions to update the model.

Updating models by uploading the feedback data

You can update a model by importing feedback data from an .xls file. Use this option when you want to move the feedback data between environments, for example, from the production environment to the development environment, for testing purposes.

  1. In the navigation panel of Prediction Studio, click Predictions.
  2. In the tile that represents the model that you want to update, click the More icon and then select Update > <language>
  3. Proceed with on-screen instructions until the Source selection step.
  4. In the Source selection step, ensure that the Upload data source option is selected.
  5. Upload the feedback data:
    1. In the Upload data source window, select the file with feedback data from your directory.
    2. Select Append to the existing data.
    3. Click Upload.
  6. Proceed with on-screen instructions to update the model.

Tips on providing feedback for text categorization models

When working with text categorization models (sentiment, topic, and intent detection), you must select a portion of the text and provide your feedback.

Consider the following sample classification use case:

  • Text to analyze – I bought this new phone from UplusTelco. It is an amazing phone with a great camera. The battery life is excellent. However, I had some issues with the warranty period time limit. I decided to call customer care. They put me on hold for 3 hours and were very unhelpful. Finally, they told me the additional warranty offer has expired last year. 
  • Taxonomy topics – Customer Service, Warranty, Phone Camera, Screen Brightness
  • Topic expected – Customer Service
  • Topic found – Warranty

Analysis

The talking points in the analyzed document include the phone camera, warranty, and customer service. Therefore, providing feedback on the entire document is inefficient and can confuse the model, which is detrimental to the classification accuracy.

Solution

Be as granular as possible when providing feedback for text categorization. Divide the analyzed document and provide excerpt-specific feedback. For example:

  • Document to analyze – I decided to call customer care. They put me on hold for 3 hours and were very unhelpful.
  • ActualResult – Customer service

Tips on providing feedback for text extraction models

For text extraction models, you can provide feedback on named entity recognition for regular text extraction models. You can also provide feedback for the email parser rule that distinguishes between various email components.

Named entity recognition

Consider the following classification use case:

  • Text to analyze – John works for uPlusTelco in New York. He works in UX design building user-friendly UIs. He was born in New Jersey.
  • Taxonomy topics – Customer Service, Warranty, Phone Camera, Screen Brightness
  • Entities detected after the classification model was run for the first time – uPlusTelco (Person), UX (Organization), New York (Location), New Jersey (Location)

Analysis

By analyzing the detected entities, the following conclusions can be made:

  • The model did not detect John as a Person (false negative).
  • The model falsely detected UX as an Organization (false positive).
  • The model falsely detected uPlusTelco and a Person, instead of as an Organization (false positive).
  • The model correctly detected New York and New Jersey as a Location (true positive)
  • The model correctly did not assign an entity type to UIs (true negative).

Based on these observations, you can provide the following feedback to the text extraction model:

  • False positives - State that UX is not an Organization and uPlusTelco is not a Person but an Organization.
  • False negatives - State that John is a Person.

For example:

<START:PERSON> John <END> works for <START:ORGANIZATION> uPlusTelco <END> in <START:LOCATION> New York <END>. He works in UX design building user-friendly UIs. He was born in <START:LOCATION> New Jersey <END> where:

  • John and uPlusTelco are annotated with corrected entities.
  • UX is marked as not being an Organization because it is not annotated.
  • All correctly detected entities (New York and New Jersey) were also annotated in the feedback. Otherwise, the model considers removing annotations as negative feedback.

Consider the following points when providing feedback to entity models:

  • Always provide the entire document as feedback.
  • You must annotate the changes as well as the correct entities. Models recognize as feedback only the sentences whose annotations differ from those that were provided by the model. Therefore, the model trains only on relevant sentences, which decreases the training time.
  • For irrelevant entities, remove the <START:EntityType><END> tags.

Email parser

You can also provide feedback on the Email Parser entity model. This model divides emails into such meaningful parts as the body, disclaimer, greeting, and signature. By distinguishing between various email parts, you can select only the ones that are relevant to your analysis.

Consider the following document:

Hi Team,

Great job with the project!

Best regards,
The Management

Disclaimer: This is proprietary corporate communication.

After the analysis, the email parser produced the following output:

<START:Greetings>

Hi Team,

<END>

<START:Body>

Great job with the project!

<END>

<START:Signature>

Best regards,
The Management

Disclaimer: This is proprietary corporate communication.

<END>

Analysis

In this output, the signature also involves the disclaimer. You must provide feedback to the models to split the signature from the disclaimer, for example:

<START:Greetings>

Hi Team,

<END>

<START:Body>

Great job with the project!

<END>

<START:Signature>

Best regards, The Management

<END>

<START:DISCLAIMER>

Disclaimer: This is proprietary corporate communication.

<END>

You must provide the entire document as feedback, including the updated and the correct annotations, in a single API call. This means that the model considers as feedback only the annotations that are different than those originally applied in the output. 

Related Content

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.