Delivering efficient agile testing at scale

By Rich Alexander, Vice President of Requirements and Testing, Nationwide and Nate Buchanan, Senior Manager, Intelligent Testing Services, Accenture

Your IT organization boasts a suite of hundreds of applications that serve your businesses well. You’ve achieved a high bar in quality, and successfully made the transition from waterfall-style development to agile.

However, you recognize that testing is holding you back. It’s slowing time-to-market, and, relative to industry benchmarks, it’s expensive.

In short, your challenge is to optimize the application testing function without losing any of the quality your businesses and customers depend on.

That was Nationwide’s goal. Specifically, Nationwide wanted to reduce the cost of testing across its enterprise without impacting application quality. Nationwide had been pivoting to an agile delivery methodology since 2005, a move that had resulted in improved application quality and maturity, with many teams practicing Behavior Driven Development (BDD).

However, there was a problem: overspending on a testing workforce that had to contend with overgrown test suites and a lack of easily maintainable automation.

Nationwide teamed with Accenture to assess the current state of their testing capability and create a business case for transforming it to a Quality Engineering model. Five value levers were identified to drive the savings:

• Test Optimization
• “Codeless” Automation
• Test Environment Management
• Test Data Management
• Workforce Globalization

Of these, the team focused on the first two for year one of the effort, concluding that addressing these would result in the biggest near-term returns. Using cognitive computing test tools (in this case Hexawise, an intelligent, scenario-based test development tool) the team optimized test suites for those scenarios customers use most—and not waste time with poorly documented legacy regression tests that may not have been testing key features. Additionally, by taking advantage of next-generation test automation tools (selecting Tricentis’ Tosca), testers without coding backgrounds could build and maintain test suites.

Transforming the IT workforce was an important prerequisite to propagating these improved capabilities throughout an organization of Nationwide’s scale. In the world of IT staffing, there is always pressure to “do more with less,” coupled with the ever-increasing difficulty and expense of hiring new programmers straight out of a university.

For Nationwide, the solution to this conundrum was to shift cost-saving practices such as test automation to new roles not requiring deep technical skillsets. Recognizing that it can be problematic to enable non-technical resources to drive a transformation to agile or DevOps, Nationwide instead defined a new “Quality Engineer” role blending test optimization, codeless automation and performance testing skills. Importantly, the HR team was enlisted to build process providing training in the new tools and methodologies for the entire IT organization.

“For us, the demand for developers has always exceeded the supply. This, coupled with expense pressures, has meant that we needed to find a different way to meet our workforce needs," says Rich Alexander, Vice President of Requirements and Testing at Nationwide.

“As the graphic above illustrates, the Healthcare industry has had the same challenge. They are solving it by staffing more Physician Assistants to do simpler tasks, which allows Physicians to address the difficult medical issues. We created the Quality Engineer role at Nationwide to solve the same challenge. Thanks to tool advancements, they no longer need to have development skills to create automated tests, which means that our Developers can focus on the more complex task of building the software.”

Nationwide created a core team which rolled out their new testing process for each application suite in four steps:

• First, the team analyzed the existing application and test suite;
• Next, they designed an optimal target state test suite using Hexawise;
• Then, using Tosca, they created automated modules for each functionality outlined in the optimized test suite in an iterative fashion and outlined a series of tests to be created for each one
• Lastly, they created a Quality Engineering role within the applications team that took ownership of the ongoing build-out of the tests once the core modules were completed

In this first wave of this four-step process, Nationwide began with just those applications with the highest potential return, with other application suites targeted in subsequent waves. To date three such waves have been completed, with 8 of 41 application suites transformed.

The result? Where Nationwide had originally targeted at 20 percent reduction in cost, they’ve so far recorded 29 percent, significantly more than what they originally expected. Ultimately, they anticipate savings approaching 20 million dollars!

Perhaps the most critical lesson learned from the engagement was the importance of executive sponsorship in encouraging different stakeholders to support the transformation. The Business Unit CIOs “bought in” to the business case which ensured that IT leadership for each line of business was aligned with the savings targets and the process for achieving them.

In short, Nationwide created a process for making Agile testing more cost-effective in a large IT organization, something that will continue to power a transition to DevOps for years to come.