DOD Business Systems Modernization
Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas
GAO-08-896, Sep 8, 2008
Additional Materials:
Contact:
(202) 512-6256
contact@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
The Department of Defense (DOD) has long been challenged in implementing key information technology (IT) management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk. One of the larger business system investments is the Department of the Navy's Enterprise Resource Planning (Navy ERP) program. Initiated in 2003, the program is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its useful life and be fully deployed in fiscal year 2013. GAO was asked to determine whether key IT management controls are being implemented on the program. To do this, GAO analyzed, for example, requirements management, economic justification, earned value management, and risk management.
DOD has implemented key IT management controls on its Navy ERP program to varying degrees of effectiveness. To its credit, the control associated with managing system requirements is being effectively implemented. In addition, important aspects of other controls have at least been partially implemented, including those associated with economically justifying investment in the program and proactively managing program risks. Nevertheless, other aspects of these controls, as well as the bulk of what is needed to effectively implement earned value management, which is a recognized means for measuring program progress, have not been effectively implemented. Among other things, these control weaknesses have contributed to the more than 2-year schedule delay and the almost $600 million cost overruns already experienced on the program since it began, and they will likely contribute to future delays and overruns if they are not corrected. Examples of the weaknesses are: (1) Investment in the program has been economically justified on the basis of expected benefits that far exceed estimated costs ($8.6 billion versus $2.4 billion over a 20-year life cycle). However, important estimating practices, such as using historical cost data from comparable programs and basing the cost estimate on a reliable schedule baseline were not employed. While these weaknesses are important because they limit the reliability of the estimates, GAO's analysis shows that they would not have altered the estimates to the point of not producing a positive return on investment. (2) Earned value management has not been effectively implemented. To its credit, the program office has elected to implement program-level earned value management. In doing so, however, basic prerequisites for effectively managing earned value have not been executed. In particular, the integrated master schedule was not derived in accordance with key estimating practices, and an integrated baseline review has not been performed on any of the first increment's releases. (3) A defined process for proactively avoiding problems, referred to as risk management, has been established, but risk mitigation strategies have not been effectively implemented for all significant risks, such as those associated with data conversion and organizational change management, as well the risks associated with the above-cited weaknesses. The reasons that program management and oversight officials cited for these practices not being executed range from the complexity and challenges of managing and implementing a program of this size to limitations in the program office's scope and authority. Notwithstanding the effectiveness with which important aspects of several controls have been implemented, the above-cited weaknesses put DOD at risk of investing in a system solution that does not optimally support corporate mission needs and mission performance, and meet schedule and cost commitments.
Status Legend:
Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
- In Process
- Open
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to improve the reliability of Navy ERP benefit estimates and cost estimates, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that future Navy ERP estimates include uncertainty analyses of estimated benefits, reflect the risks associated with not having cost data for comparable ERP programs, and are otherwise derived in full accordance with the other key estimating practices, and economic analysis practices discussed in this report.
Agency Affected: Department of Defense
Status: Open
Comments: The department has made progress toward implementing our recommendation. Specifically, the July 2011 economic analysis applied uncertainty analysis of estimated benefits, including those associated with legacy system retirements. Further, the economic analysis used a risk-adjusted life cycle cost estimate. However, as of September 2011, we have not received documentation to verify the risk analysis approach used to develop the cost estimate.
Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to enhance Navy ERP's use of earned value management (EVM), the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) an integrated baseline review on the last two releases of the first increment is conducted, (2) compliance against the 32 accepted industry EVM practices is verified, and (3) a plan to have an independent organization perform surveillance of the program's EVM system is developed and implemented.
Agency Affected: Department of Defense
Status: Open
Comments: The department has taken some actions to address our recommendation, but additional steps are still needed. Specifically, to its credit, in August 2008, the program office conducted an integrated baseline review of its second release, Release 1.1, which resulted in recommendations to mature and implement EVM processes. (Note: the third release, Release 1.2, is no longer part of the program.) However, it has yet to verify that the program is in compliance with the 32 accepted industry EVM practices and have an independent organization perform surveillance of the program's EVM system. DOD stated that because the program completed the development phase in March 2010 and the remaining work has been focused on deployment and sustainment, no additional actions are necessary.
Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to increase the quality of the program's integrated master schedule, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the schedule (1) includes the logical sequencing of all activities, (2) reflects whether all required resources will be available when needed, (3) defines a critical path that integrates all three releases, (4) allocates reserve for the high-risk activities on the entire program's critical path, and (5) incorporates the results of a schedule risk analysis for all three releases and recalculates program cost and schedule variances to more accurately determine a most likely cost and schedule overrun.
Agency Affected: Department of Defense
Status: Open
Comments: As of September 2011, DOD has made some progress, but has not fully implemented our recommendation. To its credit, the program is now using metrics to track and logically link activities and account for resources and their availability. However, a March 2010 metrics report shows that not all activities were logically sequenced, which can affect the calculation of the critical path and finish date. Moreover, in July 2010, DOD reported that it was not feasible to define a critical path integrating all three releases because key Release 1.0 functionality deliverables were completed prior to Release 1.1 development, and Release 1.2 was removed from the program. Finally, though it planned to conduct a schedule risk analysis in September 2010 so that reserves could be established for high-risk activities, it reported in August 2011 that it does not plan to conduct one because development tasks were complete as of March 2010, and the remaining tasks are focused on deployment and sustainment. Nonetheless, because the schedules are not integrated and personnel are assigned to activities across two releases, if deployment activities in one schedule were to be delayed, the other schedule that requires the same resources would likely also be delayed.
Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to improve Navy ERP's management of program risks, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) the plans for mitigating the risks associated with converting data from legacy systems to Navy ERP and positioning the commands for adopting the new business processes embedded in the Navy ERP are re-evaluated in light of the recent experience with Naval Air Systems Command (NAVAIR) and adjusted accordingly, (2) the status and results of these and other mitigation plans' implementation are periodically reported to program oversight and approval authorities, (3) these authorities ensure that those entities responsible for implementing these strategies are held accountable for doing so, and (4) each of the risks discussed in this report are included in the program's inventory of active risks and managed accordingly.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: The department has taken actions to address the intent of this recommendation. First, the program office reevaluated its plans for mitigating risks associated with data conversion and adopting new business processes. Second, the program manager and System Command officials report monthly to the Navy Enterprise Resource Planning (ERP) Systems Integration Board (NESIB) on performance, and periodically brief oversight and approval authorities on the implementation of risk mitigation plans. Third, the NESIB requires actionable reporting on performance by the program manager and System Command officials, and the program manager is to report to the Milestone Decision Authority on implementation of risk mitigation strategies. Fourth, the program's risk inventory has been updated to include risks related to adopting new business processes and data conversion.