LinkedIn
Copied!

Table of Contents

Monitoring predictions

Analyze how successful your predictions are in predicting the outcomes that bring value to your business. Gain insights by reviewing performance charts for predictions and the models that drive them.

This procedure applies to predictions that are used in customer engagement and case management. For information about analyzing text predictions, see Monitoring text predictions.

If a prediction uses more than one model, you can select the outcome for which you want to see performance statistics, for example, clicks, conversion, or clicks and conversion combined.

  1. In the navigation pane of Prediction Studio, click Predictions.

  2. From the list of predictions, open a prediction that you want to analyze.

  3. On the Analysis tab, click Prediction.

    1. In the Outcomes field, select the outcome to analyze.

      If a prediction predicts a single outcome, the outcome is already selected. If a prediction predicts outcomes that occur one after another, you can select each individual outcome or the two outcomes combined.
      For a prediction that predicts whether a customer who is likely to click a web banner is also likely to accept the corresponding offer and convert, you can select from the following outcomes:
      • Clicks
      • Conversion
      • Clicks + Conversion
    2. In the Time frame field, select the period that you want to analyze.

      The charts show the performance measures that are relevant to the selected outcome. For example, a performance analysis with regard to churn covers the churn rate, lift, AUC, and total cases, as in the following figure:

      Prediction analysis
  4. Click the Models tab.

  5. If more that one model drives the prediction, in the Models field, select the model the performance of which you want to analyze.

    The performance chart shows how effective the model is over time. If a shadow model runs alongside an active model, the chart displays the performance curves for each model in the active-shadow pair. The comparison can help you determine which model is more effective and decide whether you want to promote the shadow model to the active model position. The analysis is available for both outcome-based and supporting models.

    The following figure shows the performance chart for the active model, Predict Churn DRF. This model is paired with a shadow model, Churn GBM. The chart shows two distinct lines that illustrate the performance of each model over time. Hover over the lines with your cursor to see the exact scores for each model at different times. In this example, the two lines converge around January 27, suggesting that the models performed equally well. The detailed information on mouse hover shows that the active model had a slightly higher score.

    A model performance chart in a prediction
    active-shadow-analysis icon

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.