Quick Transition
Using an innovative workcell transitioning approach, Accenture ramped up more than 500 people in just over six weeks. Critical to the transition was successfully gathering knowledge from multiple organizations internal and external to CMS.
Accenture conducted more than 450 knowledge transfer sessions to capture over 700 knowledge artifacts. The team also hosted 16 days of intensive, requirements “lock-in” sessions that included Accenture developers, testers, business architects, technical architects and CMS business executives. The team captured 3,300 testable and traceable requirements.
This successful transition—unprecedented in its scale and sense of urgency—reduced risk and positioned the team to start hands-on delivery and rescue activities as soon as possible. The entire transition of the program from incumbent to Accenture took eight weeks…four weeks faster than originally proposed.
Successful Operations
To stabilize the site, Accenture completely took over maintenance and operations during the peak period of the 2014 open enrollment. The team tackled defects and delivered urgently needed fixes ahead of schedule, thereby maintaining operations throughout the peak enrollment period.
The effort included:
- Stabilizing the notices infrastructure and software to generate 2.9 million notices with a daily peak of more than 340,000 notices.
- Significantly improving Account Transfer performance, reducing the response time for inbound transfer by 99.8 percent.
- Stabilizing the FFM application in preparation for an expected enrollment surge. On March 31, 2014—the last day of the first open enrollment—the FFM supported a peak of more than 185,000 concurrent users with fast response times. For the second and third open enrollments, there was no unplanned downtime for any components for which Accenture was responsible. The Accenture team also accelerated online response times to under a quarter of a second.
To keep the system fully functioning, especially during peak enrollment, Accenture provided continual monitoring and reporting for HealthCare.gov. Each week, the team delivered multiple software releases, with significant enhancements.
Shop Delivery
Accenture quickly mobilized a team to take over Small Business Health Options Program (SHOP). In just eight months, the team started with requirements analysis and successfully delivered a commercial off-the-shelf (COTS) package.
This included customization, performance testing, functional testing, security auditing and integration with other CMS systems. Accenture collaborated closely with the COTS manufacturer to launch a new line of business for the marketplace in this very short time period, with a smooth go-live and very low defect count.
Innovative, Cloud-Based Solutions
The new federal and state health insurance exchanges created new marketplaces for insurance companies to sell new insurance products to new consumers. To stabilize the pricing of risk, the ACA included new reinsurance and risk adjustment programs.
The former helped offset larger-than-expected claims, and the latter helped transfer payments from issuers that took on lower-than-expected risk to those that took on higher-than-expected risk.
These programs required gathering confidential claims information from 800 different insurance issuers and then performing complex, risk stabilization calculations and analytics. CMS needed to provide a solution where issuers-maintained control of their confidential claims information, as input to the risk calculations, but CMS controlled the risk algorithms, software and reference data.
Accenture developed a cutting-edge solution that provides issuers a complete data processing environment, which each issuer owns and operates. The “EDGE” system uses Amazon Web Services (AWS) to connect with more than 800 issuers, to share and process claims information in the cloud according to the CMS analytical algorithms. Issuers maintain complete control of their proprietary claims and pricing data; CMS has visibility to the outputs, but not the inputs, of the algorithms. Additionally, 135 issuers elected to participate as AWS-deployed servers, using a fully automated environment provisioning process that has successfully and securely processed the issuers’ data without requiring internal infrastructure investment. The other issuers used an on-premise deployment model, which still took advantage of the same software images and upgrade processes.
The EDGE system enables CMS to create a level playing field for all issuers. It provides consistent software and data version management across the universe of independent installations. EDGE simplifies and expedites deployment for issuers, reducing time from several days in a standard software distribution and configuration model to as little as 15 minutes, while enabling hands-free software upgrades and execution of remote commands.
Program Integrity
Millions of Americans count on the integrity of their enrollments and tax subsidy calculations. And the 800 issuers count on the integrity of how their plans are displayed on HealthCare.gov, how their enrollments are processed, and how they get paid for reinsurance and risk adjustment.
Accenture worked with CMS to build in program integrity. We built an entirely new system for policy-based payments, to make sure the tax subsidy payments to issuers resolved to the penny, for each and every enrollment in the FFM. In addition, CMS hired an independent contractor and auditor to vet the EDGE system for reinsurance and risk adjustment payments. That contractor determined that a full 100 percent of the billions of dollars in payments was accurately calculated.
At a detailed level, we built in logging and auditing capabilities for all key transactions. And we worked with CMS to build a robust security program, which proactively detects potential vulnerabilities. Our joint security team achieved Authority to Operate (ATO) on schedule, for all of the systems that Accenture built and maintained.
Technology. Delivery. Discipline.
Rescuing the website required immediately instilling technology discipline, along with significant investment in automation and tooling. To improve the fully manual test and release process, the Accenture team created fully automated, regression test suites at all levels:
- 15,500-unit test cases in Java
- 1,400 functional tests, end-to-end, for the user interfaces
- 15,400 functional data validation and batch validation tests
- 990 transactions tested in a comprehensive, end-to-end performance suite, reflecting a real-world transaction mix.
This comprehensive regression suite runs on a regular basis, and provides automated emails and executive dashboards for monitoring. The automated functional testing consists of more than 300,000 test steps and is typically executed five times per week. The automation of this functional testing saves over 50,000 hours per year in manual testing. This high level of automation has allowed the team to dramatically improve the quality, reliability and speed of new software builds.
Accenture also created automated build and deployment scripts, not only for the software, but also for the software environments. Development and testing teams can now “click a button” to automatically provision new environments in AWS. They can also click a button to deploy software releases to those environments. This includes checking out the versioned code from the repository, building the software, checking code quality, running it through a battery of automated tests, validating performance and moving the software items into the environment. We have moved from fully manual processes, which are time-intensive and error-prone, to a fully automated Development and Operations (DevOps) capability.
Moreover, Accenture created a robust operations monitoring stack. The stack includes a suite of monitoring tools and dashboards that integrate with:
- Automated paging software for the operations team, with built-in, automated escalations if a team member does not respond quickly
- Detailed transaction and log information
- Interactive chat tools used by the operations teams, where the monitoring tools also posts automated updates
In addition to monitoring live traffic on the site, down to the level of individual transactions, the stack also monitors the performance of synthetic transactions. The synthetic transactions run every few minutes, around the clock, to proactively uncover production issues before consumers do.
Multi-Speed IT
Beyond just the technology, Accenture also brought methodology discipline to the program. This included introducing delivery techniques to make sure CMS realized business benefits incrementally, over time, without having to wait for a full-blown or big bang solution.
Additionally, Accenture and CMS together created an enterprise-ready agile approach, for new development. Accenture combined its proven, enterprise agile methods with CMS’ Expedited Life Cycle (XLC) system development process. This approach blends the benefit of agile with government requirements for documentation, auditing, traceability and independent verification. In sum, this approach enabled efficient delivery with sound risk management and proven software delivery practices.
CMS embraced agile, with actively engaged product owners, who represented multiple stakeholders and made decisions quickly. After each three- to four-week sprint, Accenture demonstrated the updated software for CMS acceptance, giving CMS real-time visibility into progress of the software build. This built confidence and allowed agile teams to incorporate feedback and re-prioritize along the way. The program’s use of agile has also helped improve collaboration between IT and the business on strategic system initiatives.
Architecture Simplification
HealthCare.gov, as originally built, had technical debt built up from the technical complexity of the software. The Accenture team innovated and simplified the technical plumbing of HealthCare.gov and other related systems.
This had multiple business benefits:
- Accenture saved CMS more than $4M per year in software licenses, data center costs and other labor; Accenture significantly reduced the number of servers required
- The website is now faster; response times went from roughly half a second to a quarter of a second
- The website is now easier to maintain and extend
This approach was so successful that another contractor is considering doing architecture simplification on its software, for similar benefits.
Cloud First Approach
Accenture delivered the new EDGE software in the cloud, as mentioned. That is not the only use of the cloud—Accenture has taken a “cloud first” approach, wherever using the cloud makes sense and meets CMS’ security needs.
For example, the suite of automated functional tests is run from servers in the cloud, which can scale up or scale down, as necessary. As another example, the new environments for new software capabilities are deployed in AWS. Accenture made it as simple as literally pushing a button: provisioning a new environment plus the COTS middleware plus the application software is fully automated.
Cloud-Based CRM
Accenture recommended using salesforce.com to quickly build a Customer Relationship Management (CRM) solution that would help improve collaboration and outreach with issuers. The initial prototype was deployed within weeks; the full solution was complete and operational in four months.
The unified CRM solution replaced dozens of disassociated email templates, contact lists and Excel spreadsheets. Email generation improved from 30 minutes to 10 minutes. Issuer lookup and update time improved from 20 minutes to 1 minute. Reporting time improved from 30 minutes to 2 minutes. These results all occurred on a platform that is now more accurate and easier to use.
Cloud-Based DevOps
Accenture implemented full DevOps processes and technologies to accelerate development and deployment. Automated tasks across environments include code check-out, build and packaging; code quality scanning; security scanning; unit testing, functional testing and initial performance testing; software deployment; and release smoke testing.
The team used tools such as Junit, github, Amazon Web Services, Splunk, JIRA Software (including Jira, Confluence and HipChat), Fortify, Python and Bash to support continuous delivery.
The DevOps approach has significantly reduced manual errors, improved software release quality and allowed the team to deliver faster.
Culture
Transforming HealthCare.gov required a shift in culture. Team members adopted a “one team, one goal” mindset and created a blame-free environment, where everyone would succeed or fail together.
It was not a client/vendor relationship that had clear lines of separation. Rather, it was one cohesive team that worked collaboratively and transparently. Everyone was open about schedules, risks and defects, and all worked together to share knowledge and solve problems proactively. Instead of finger pointing, people were rewarded during daily meetings for candidly acknowledging, “This isn’t working…and here’s what we need to do to fix it.”
