Vermont’s contractor was CGI, who was also responsible for the U.S. project. The QA contractor was Gartner Consulting. Nearly all of the released findings stem from Gartner’s QA reports.
Similar to Oregon’s health exchange project, at no point in time did the QA contractor assess risk as anything but RED – high risk. For Vermont, it was 12 straight high risk bi-weekly status reports, the last one being after the October 1 go-live when the probability of most risks occurring hit 100%. The most critical risks throughout the project were:
- Planning. Gartner notes a lack of planning around integration – probably the hardest part of the project and testing – the most important part of the project.
- Project Management. A laundry list of project management issues: lack of project controls, appears to be very little control over changes in schedule, deliverables or scope. Impact to other project areas are not analyzed or alternatives presented, plan continues to be reworked and updated, there is no way to report on or assess progress. It is not clear what components will be delivered when. The dates seem to be unrealistic and unachievable. Some dates violate CMS schedule requirements.
- Scheduling. Numerous schedule issues (“missed delivery dates and milestones” is noted in 8 of 14 released QA reports) including one stunning problem probably caused by technical issues or poor management: Development (and, later) test, (and, later) prod, (and, later) pre-prod, training and DR environments are not ready as planned. Essentially no environment was ready as planned – NONE. Also: “Plan continues to be reworked and updated, there is no way to report on or assess progress. It is not clear what components will be delivered when. The dates seem to be unrealistic and unachievable. Some dates violate CMS schedule requirements.”
- Staffing. Within five months of go-live, Vermont had four unfilled positions, including Test and Training Leads and CGI had 20 (!) unfilled positions. Worse is that there was no staffing plan to address this gap.
In some cases, the QA reports include Vermont’s risk management strategy. One strategy is especially damning.
- Risk Rating: HIGH
- Risk: It is likely that functionality will not be tested until it is too late.
- Vermont’s Risk Response: Accept
My response: D’oh! This shows that Vermont was fully prepared to implement features that it had not tested. Unacceptable.
This audit is not as comprehensive as Oregon’s audit as it is almost entirely based on Gartner’s QA reports. And some of them have redactions that, of course, makes you wonder what is hidden – could it be worse? Nonetheless, more lessons learned when undertaking a large IT project.