Air Traffic Control
FAA Reports Progress in System Acquisitions, but Changes in Performance Measurement Could Improve Usefulness of Information
GAO-08-42, Dec 18, 2007
Additional Materials:
Contact:
(202) 512-4803
contact@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
Acquiring new systems on budget and on schedule is critically important in transitioning to the Next Generation Air Transportation System (NextGen). However, air traffic control modernization has been on GAO's high-risk list since 1995, in part due to acquisitions exceeding budget and schedule targets. The Federal Aviation Administration's (FAA) Air Traffic Organization (ATO) has responsibility for managing air traffic control acquisitions. GAO was asked to examine (1) ATO's goals, performance measures, and reporting for systems acquisitions; (2) the validity of ATO's performance measures; and (3) the implications of using ATO's performance measures to assess progress in transitioning to NextGen. To address these issues, GAO compared ATO's measures with attributes of successful performance measures, interviewed agency officials, and sought perspectives of aviation experts.
To be consistent with federal guidance and with targets set in the Department of Transportation's strategic plan, ATO established annual acquisition goals and performance measures that call for a high percentage of its major acquisitions to be within 10 percent of budget and on schedule. ATO identifies major acquisitions and reports performance against its goals using its most recently approved budget and schedule estimates. To measure on-budget performance, ATO calculates budget increases over an 8-month period--between January and August of each year. To measure on-schedule performance, ATO selects a minimum of two annual milestones from its major acquisitions and calculates the percentage of milestones that are on schedule. Because ATO's acquisition performance measures lack objectivity, reliability, coverage of core activities, and clarity, and focus only on the preceding year, they may not provide a valid assessment of performance over time. On the positive side, the measures are aligned with FAA's strategic objectives, are measurable, have no overlap, and address governmentwide priorities. However, the performance measures lack objectivity because ATO has no objective criteria for designating which programs are "major" and should be selected for performance reporting. This makes it possible for subjective considerations to dominate the outcome and leaves the performance measures vulnerable to bias in the selection of programs for reporting. The lack of objective criteria for designating major programs also impairs the reliability of the measures (the ability of the measures to produce the same results each time they are applied under similar conditions) and undermines assurance that ATO managers include all core program activities in performance reporting each year. The performance measures also lack clarity in that they do not indicate that ATO measures the performance of many acquisitions against the most recently approved budget and schedule estimates rather than the original estimates. ATO's acquisition performance measurement and reporting could mask budget increases and schedule delays that could have a negative effect on the transition to NextGen. Although ATO reported performance that exceeded its goals for fiscal years 2004 through 2006 and showed nearly steady improvement, when measured against original baselines, acquisition performance improved but was lower than reported. Going forward, the absence of original budget and schedule information on ATO's acquisitions could give the impression that the transition to NextGen is progressing more smoothly than might actually be the case. It will be important for ATO and Congress to recognize budget increases and schedule delays so that the capacity, efficiency, and safety benefits of NextGen can be realized in a cost-efficient and timely fashion.
Status Legend:
Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
- In Process
- Open
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical air traffic control (ATC) acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the objectivity, reliability, and inclusion of core programs in ATO's acquisition performance measures by establishing written, objective criteria and guidance for managers to use in determining which programs are major--and thus selected for performance reporting--and in selecting schedule milestones.
Agency Affected: Department of Transportation
Status: Open
Comments: During the past year, FAA developed 5 categories for its acquisitions based on cost and other factors. FAA is in the process of determining the relationship between the 5 categories and how it will define "major program."
Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the clarity of ATO's annual acquisition performance measurement process by disclosing in its Performance and Accountability Reports that the measurement for on-budget performance covers 8 months and is measured against the most recently approved budget baselines. Similarly, improve the wording of the target and reporting for on-schedule acquisitions to disclose that this measures 1 year of performance against selected program milestones.
Agency Affected: Department of Transportation
Status: Closed - Implemented
Comments: Comparing FAA's FY 2008 Performance and Accountability report with the 2010 report, FAA has implemented this recommendation. The performance measure in the 2010 report states "Ensure that major system investments are within 10% variance of current baseline total budget estimate at completion, and that 90% of major system investments selected annual milestones are achieved." AThe recommendation is closed implemented. An accomplishment report will be prepared.
Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to identify or establish a vehicle for regularly reporting to Congress and the public on ATO's overall, long-term performance in acquiring ATC systems by providing original budget and schedule baselines for each rebaselined program and the reasons for the rebaselining. If this information is not added to FAA's annual Performance and Accountability Report, then the Performance and Accountability Report should reference where this information can be found.
Agency Affected: Department of Transportation
Status: Closed - Implemented
Comments: In December, 2007, we reported that FAA?s acquisition performance measure focused on performance during the preceding year, rather than from the initiation of the acquisition--a period that often spans many years. FAA would report that it met its performance goal if a defined percentage of its major acquisitions remained within budget and on schedule during the fiscal year. However, the budget and schedule baselines, as stated at the start of a fiscal year, often reflect decisions in prior years to increase acquisitions? budgets or to delay schedules beyond the original budget and schedule estimates. Comparing an acquisition?s current budget and schedule baselines against its original baselines would, in many cases, show more acquisitions being over budget or behind schedule than FAA was reporting. We reported that FAA?s performance reporting could give the impression to Congress and the American people that FAA?s acquisitions performance was better than was actually the case. We recommended that FAA establish a vehicle for regularly reporting to Congress and the public on its overall, long-term systems acquisitions performance by providing original budget and schedule baselines for each rebaselined program, and the reasons for the rebaselining. In response, FAA established a new appendix to Capital Investment Plan, which is reported annually to Congress and made available to the public over the internet. FAA first published the new appendix, titled ?Response to GAO Report,? in its fiscal year 2010 - 2014 Capital Investment Plan, but it did not contain sufficient information to address our recommendation. FAA included additional information in its fiscal year 2011 - 2015 Capital Investment Plan, including each acquisition?s original and current budget and schedule as well as the reasons for changes, as we recommended,. For example, the appendix shows that FAA had extended the completion date for its Integrated Terminal Weather System (ITWS) from the original completion date of July 2003 to June 2010 and increased its budget from $276.1 million to $282.1 million. The appendix explained that the rebaselining was necessary due to, among other things, added requirements. By presenting the most accurate and complete assessment possible when reporting its performance in acquiring ATC systems, FAA will better facilitate congressional understanding and oversight of FAA?s progress in implementing NextGen.
Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the usefulness of ATO's acquisition performance reporting by including information (in the Performance and Accountability Report or elsewhere) on the potential effects that any budget or schedule slippages could have on the overall transition to NextGen. This also could include information concerning any mitigation plans ATO has developed to lessen the effects of program slippages on the implementation of NextGen systems.
Agency Affected: Department of Transportation
Status: Open
Comments: FAA officials stated that the Performance and Accountability Report (PAR) is not set up to accommodate this kind of information. In the PAR, FAA reports only on the status of acquisitions against the acquisition performance goals. There is no performance goal or criterion against which to measure the impact on NextGen. Officials said the NextGen implementation plan might be a better place to discuss the impact of acquisition slippages on the overall transition to NextGen.