Deliver quality with continuous testing: Turbocharge your delivery using automation

Linoy Alexander,

Implementing DevOps for an organization is all about setting up a standardized, repeatable deployment processes in the form of continuous integration and continuous delivery (CI/CD) deployment pipelines. However, in order to realize the true value of a deployment pipeline, investment needs to be made in automating the pipeline. If you really want to turbo charge these deployment pipelines without compromising on quality, then continuous testing is critical.

Continuous Testing

The real purpose of any DevOps process is to speed up the feedback cycle: ensuring that the application is properly packaged, correctly deployed, performant, and that the end user experience is as expected. Every part of the pipeline is a form of testing that validates expected functionality. In a good DevOps process, developers are notified as soon as possible in the event of a test failure in order to fix and issue a new version. This process should be continuous, fast, reliable and deterministic.  The best way to achieve such a process would be through embracing continuous testing through automation which is best represented as a testing cycle.

The Testing Cycle
The Testing Cycle
The Testing Cycle

The best approach to this testing cycle is to have a diverse collection of automated tests following industry best practices that run autonomously and continuously in the CI/CD pipeline. Strategic selection of the types of tests that are run in the pipeline will provide maximum coverage, and enables a shift left approach to application development. As part of this cycle, developers should be continuously adding new tests as the application functionality evolves, while pipeline administrators and others monitoring application quality ensure that these tests are being run through every pipeline deployment while enforcing application quality metrics as deployment gates.

Testing strategy

Let’s first discuss which types of test should be considered as part of continuous testing strategy. Most people end up with an overinvestment in two types of tests, manual regression testing and UI end to end test automation, which is often just where the manual test cases were automated. There are a number of significant challenges with those approaches

  • Manual regression test execution is often most expensive form of testing as it involves human time and effort in the worst way possible, in repetitive rote execution of tasks and steps, which human beings are generally quite bad at.
  • UI testing is better in that computer can repetitively execute the end user experience and thus a greater number of tests can be run. However, you will quickly hit diminishing returns here if you only rely on UI end to end scenario tests as they are often brittle, slow running, and become increasingly difficult to maintain and update. You may eventually spend more time maintaining these tests than actually developing new feature

While it may seem counterintuitive, the way to address these problems is to invest in further test automation, but to be much more strategic about the types of tests that are leveraged.  The most effective strategy is to invest in a diverse collection of automated tests as suggested by the test pyramid which is widely used in the industry.

The Test Pyramid
The Test Pyramid
The Test Pyramid


Tests at the bottom of the pyramid, unit tests, are often the fastest, the least expensive to run, and the easiest to maintain. While individually these tests might only cover a small unit of functionality, you get excellent coverage by having a large number of these tests. Tests at the top of the pyramid provide greater coverage per individual tests, however are the most expensive to run, hardest to maintain, take the most time to run, and ultimately should represent the least number of tests in the test suite. The higher up the pyramid you go, the higher the overall cost and slower and more complicated the test. Your goal is to ensure that the investment in test automation is geared to providing fast and meaningful feedback to the application developer. Pega provides the tools and integrations to allow your organization to invest into test automation and continuous testing that easily matches this test pyramid.

Unit Testing

Starting for the bottom of the pyramid, a majority of tests should be unit tests, which verify the smallest meaningful unit of functionality in the application. In Pega this is often a rule, like a decision rule such as a Decision Table. For example, ensuring that a credit approval decision table returns the correct status for a given set of conditions or perhaps a data mapping rule, such as Data Transform or Data Pages such as ensuring that a given client ID returns the expected client details. The purpose here is to test the logic represented in the rule in isolation of the rest of the application functionality.

Functional Testing

The next group of tests should focus on testing related sets of functionality, one step higher than individual rules covered by unit tests. For example, you may want to ensure that a credit application process (or case) can run end to end for different types of client profiles and resulting in the correct outcome each time.  In a Pega application, you can perform these end-to-end functional tests using the Pega API that allow you to perform case management and data management operations which you may complement with any additional application APIs. Using these APIs, you can leverage test frameworks such as SoapUI, RestAssured or others like it.

UI Testing

UI-based functional tests are best used to for verifying important end-to-end user scenarios and demonstrating that your user acceptance criteria are met. Pega provides an intuitive low code UI testing framework called Pega Scenario Testing that allows you to create these end-to-end tests quickly and easily without any expertise in these testing technologies. Of course, if you have expertise writing UI tests in third party tools, we also provide a complete starter kit for Java Selenium on the Pega Marketplace.

Application Quality Dashboard

The last aspect here is to measure and validate the quality of your application over time. Pega provides the Application Quality Dashboard, which makes it easy to understand the health of your application and take action.  This dashboard quickly identifies areas within your application that need improvement by displaying metrics related to your application's health, such as Guardrail compliance scores, test coverage and pass percentages. It even breaks these scores down by the case types and the datatypes in application which is a great way to focus your investment into improving the quality of the application, one case type and datatype at a time.

Bringing it together

Pega's Deployment Manager is a low code deployment automation tool that provides these CI/CD best practice pipelines out of the box. It has built is support to execute all the tests mentioned above in the all the right stages in the pipeline, such as running unit tests when developer merge changes to validating quality gates such as test coverage, Guardrail scores and security configuration. It has built in notification to alert developer, release managers and other stakeholders when tests or quality gates fail and customizability to allow you to modify the deployment process to suit your needs and help you implement true continuous testing.



Don't Forget

About the Author

Linoy Alexander is director of developer productivity at Pegasystems.