|Session:||111th Congress (Second Session)|
|Witness(es):||Micah D. Lowenthal|
|Credentials: ||Director, Nuclear Security and Nuclear Facility Safety Program, Nuclear and Radiation Studies Board, Division on Earth and Life Studies, National Research Council, The National Academies and Study Director, Committee on Advanced Spectroscopic Portals, National Research Council, The National Academies|
|Committee:||Homeland Security and Governmental Affairs Committee, U.S. Senate|
|Subject:||Nuclear Terrorism: Strengthening Our Domestic Defenses|
Hearing on Nuclear Terrorism: Strengthening our Domestic Defenses
Micah D. Lowenthal, Ph.D.
Director, Nuclear Security and Nuclear Facility Safety Program
Nuclear and Radiation Studies Board
Division on Earth and Life Studies
National Research Council
The National Academies
Study Director, Committee on Advanced Spectroscopic Portals
National Research Council
The National Academies
Committee on Homeland Security and Governmental Affairs
June 30, 2010
Good morning Chairman Lieberman, Ranking Member Collins, and members of the committee. My name is Micah Lowenthal and I am the director of the program on nuclear security and nuclear facility safety in the National Research Council’s Nuclear and Radiation Studies Board.1 I am here to describe some of the findings and recommendations from the interim report of a congressionally mandated National Research Council study on procuring next-generation radiation detectors for screening cargo as part of the Global Nuclear Detection Architecture.2 I am the study director supporting the committee of experts that authored the report.3 I will begin by providing background on the request for this study and the remainder of my testimony will focus on the portion of the report that addresses costs and benefits of the new detectors in the context of the Global Nuclear Detection Architecture.
BACKGROUND ON THE REQUEST FOR THE STUDY
Containerized cargo entering the United States at sea ports and truck land-border crossings is currently screened for radiation using fixed detectors, called radiation portal monitors (RPMs), in conjunction with handheld radioisotope identifiers (RIIDs). The Department of Homeland Security (DHS) is seeking to deploy new radiation detectors, called advanced spectroscopic portals (ASPs), to replace the current RPM and RIID combination, which has known deficiencies. The ASPs consist of new detector equipment and new software, including improved algorithms for isotope identification.
The ASP procurement has had several setbacks from the beginning of the government’s effort to develop, buy, and deploy them.4, 5 In Title IV of Division E of the Consolidated Appropriations Act, 2008 (Public Law 110-161), Congress required that the Secretary of Homeland Security submit to Congress a report certifying that ASPs would provide a "significant increase in operational effectiveness" over continued use of existing screening devices. This certification is a precondition for proceeding with full-scale procurement and deployment of ASPs. Congress also directed DHS to request that the National Academies advise the Secretary on the certification decision by helping to validate testing completed to date, providing support for future testing, assessing the costs and benefits of this technology, and bringing “robustness and scientific rigor to the procurement process.”
Due to delays in the test and evaluation program, the Academies and DHS agreed that the study committee would issue an interim report that provides (1) the committee's evaluation of the testing program, (2) advice on how the Domestic Nuclear Detection Office (DNDO) can complete and make more rigorous its ASP evaluation for the Secretary and the nation, and (3) provide guidance on what a cost benefit analysis (CBA) should encompass. The interim report was issued in June 2009; the committee is continuing its evaluations and plans to issue its final report in the late summer.
THE INTERIM REPORT
In the interim report, the committee provided advice on the difficult task of analyzing costs and benefits of the ASPs. The committee stated that a CBA can provide a structure for evaluating whether a proposed program (such as the ASP program) is reasonable and justified. A CBA can provide insight about alternative choices--for example, whether the benefits of a given program exceed its costs, and which choices are most cost-effective.
The committee was mindful throughout the study that the Secretary is faced with the decision whether to procure ASPs, so the analysis should focus on what information she would need for that decision. One of the committee’s criticisms of DHS’ analytic approach and certification criteria as of June 2009 was that after completing the cost-benefit analyses planned at that time, even if ASPs met the criteria, DHS still would not know whether the benefits of the ASPs outweigh the additional costs associated with them, or whether the ASP procurement funds are more effectively spent on other parts of the Global Nuclear Detection Architecture (the opportunity costs).
To be effective, the CBA must include three key elements: (1) a clear statement of the objectives of the screening program; (2) an assessment of meaningful alternatives to deploying ASPs; and (3) a comprehensive, credible and transparent analysis of benefits and costs.
A clear statement of the objectives of the screening program
A cost-benefit analysis should clearly define the ASP program objectives, including describing the capabilities of ASPs in the context of their role in the Global Nuclear Detection Architecture. The committee noted that it was not clear whether ASPs were intended primarily to increase operational efficiencies or to increase the likelihood of detecting smuggled nuclear materials.
An assessment of meaningful alternatives to deploying ASPs
The committee’s view is that to make a decision about ASP deployment, DHS should consider tradeoffs and interactions among different elements of the Global Nuclear Detection Architecture.6 Alternative investments can be made to help prevent the smuggling of nuclear materials into the United States. DHS could invest its resources in: (1) procuring and maintaining ASPs; (2) using a different technology or approach to fill the same gap that ASPs are meant to address; or (3) addressing different gaps, for example, different threats and different modes of transport, like rail, aircraft, or small watercraft. Furthermore, it would be best if DNDO evaluated a number of alternatives against a broad set of scenarios that represent not only cargo screening today, but how it may change in the future. For example, the preferred modes and routes of shipping and transportation are not static. Nor are the threats. A more comprehensive evaluation of security benefits would factor in trends in shipping patterns and practices, as well as an adversary’s ability to adapt to defenses.
For example, the enhanced capabilities provided by the ASP-C7 system are relevant to cargo containers entering the United States by truck, but not by rail. In the future, it is probable that less cargo will be brought directly into major U.S. seaports. Some of the fastest-growing ports in North America are in Canada and Mexico,8 and it is expected that these ports will handle increasing amounts of cargo destined for the United States. Much of this cargo will be unloaded onto on-dock rail and will cross U.S. borders on rail.
A comprehensive, credible and transparent analysis of in-scope benefits and costs
DNDO’s analyses as of June 2009 focused on benefits in the form of operational efficiencies, not security benefits. The savings from operational efficiencies, however, appeared not to exceed the additional lifecycle costs of the new systems, so to justify the expenditure, the CBA needed to address security benefits, as well.
DNDO recognized the difficulty in assessing two of the cost-risk elements in the CBA with respect to equipment performance: assessing the probabilities of failure to detect threat material, and factoring in potential consequences of such a failure.
It is a complex task to evaluate the probability of an adversary attempting to smuggle threat material into the United States. In fact, that probability is impossible to determine definitively. Despite this difficulty, analysts need to understand what they can about these probabilities based on analytical tools and information from the intelligence community and other sources.
The consequences are likewise uncertain for other reasons: the variability in the possible consequences is very broad, so the benefits from avoiding a nuclear smuggling incident are both difficult to quantify and factor into a CBA. However, that difficulty does not make it less important to consider these benefits.
Security benefits can result from improvements in detection, identification, and interdiction capabilities, which increase the probability that a given nuclear smuggling event is prevented. Benefits can also result from deflection (e.g., to overseas targets) or deterrence (effectively reducing the probability that someone will attempt to smuggle nuclear material).
While it is difficult to assess security benefits, there are several analytic approaches that have been used to justify programs or regulations in the context of security against terrorism. The committee offered three of them in its interim report. Each approach suffers a common shortcoming – it does not answer the question of whether the benefits of implementing ASPs exceed the program’s costs. However, each in a different way can provide insights that could help the Secretary weigh the merits of acquiring and deploying ASPs or alternative nuclear detection technologies or deployments.
A capability-based planning approach is a structured assessment of the options for how a program, including the people and technology, can meet specific operational goals across a wide range of circumstances.9 In this context, it can be used to evaluate how alternative detection technologies or deployment strategies reduce the risk of nuclear smuggling in the United States. The set of options could include alternative deployments of ASPs and current RPMs, deployment of alternative RIIDs, or (depending on the scope of analysis) shifting emphasis between port-of-entry and non-port-of-entry detection. The alternatives would be compared based on what is important to the program objective or the larger mission. For example, a measure of the detection system’s performance might be its probability of detecting a specific threat object (e.g., a billet of highly enriched uranium). The strength of a capabilities-based planning approach is that it can provide a rich comparison of the security benefits emphasizing the circumstances under which each alternative might be preferred. The weakness of this approach is that exploring the circumstances that affect the system’s capabilities can quickly lead to a large and complex analysis. Analysts must balance this complexity, which may be needed for fidelity, against the need for simplicity to draw salient insights about a system’s capabilities.
Game theory could provide insight into the benefits from deterrence or deflection associated with the parts of the Global Architecture. Studies of other security applications have found that the simple presence of security can significantly deter criminals. For example, using game theory, Ayres and Leavitt10 predicted that there would be a difference in the security impact of observable security measures and known-but-not-observable security measures. Looking at vehicle theft statistics they found that increased use of hidden radio-transmitter devices for retrieving stolen vehicles in a given area resulted in substantial overall declines in auto thefts. In contrast, observable security measures against car theft just tended to shift or deflect the risk of theft to other vehicles, but not lower overall rates of theft. In summary, having an effective defense in some cars, and no way for an adversary to determine which cars have it, reduced theft rates.
Likewise, the existence of some radiation monitoring at seaports and land border crossings may deflect adversaries, causing them to focus on other gaps in the nation’s security that are identified as easier targets. For these reasons, increasing detection probabilities in screening of truck-borne cargo may have only a modest overall benefit as long as there are significant gaps in the Global Architecture. This has sometimes been described as fortifying the locks on the front door but leaving the windows open. Improved detection can be expected to become more beneficial as those gaps are filled.
The general weakness of game theory is that analysts have to make assumptions about adversaries’ goals, resources, and reasoning. For example, it is difficult to estimate the value to the adversary of different outcomes (what constitutes success and what are the costs of being caught). But still, useful insights can be gleaned. Consider the case of deterrence. Screening only a fraction of the containers entering the United States may provide for effective deterrence (or deflection), if detection probabilities are sufficiently high and if smugglers cannot predict which containers will not be screened. This benefit quickly evaporates if adversaries are able to stage several smuggling attempts simultaneously because the chance of at least one attempt succeeding grows rapidly with the number of attempts.11 Like other game theoretic analyses, this enters a psychological realm, ascribing logical thinking to the adversary, such as that the threat material is a scarce and valuable asset and that the risk of discovery at a port of entry is not desired.
Cost-effectiveness analysis and break-even analysis
Finally, cost-effectiveness analysis and break-even analysis are related approaches that have been used to assess costs and benefits when it is difficult or impossible to perform a complete cost-benefit analysis. Because the security goals of the ASP program may be difficult to value monetarily, comparing program alternatives using cost-effectiveness measures such as dollars per life saved or dollars per attack avoided could provide insights into their relative merits. Break-even analysis seeks the conditions that must be met for benefits to exceed costs. In security applications, these conditions could be a required reduction in overall risk12 or a baseline estimate of a threat of attack that exists.13 In cases where break-even analysis identifies meaningful bounds on decisions, that is, cases where the threshold conditions can easily be judged to exist, this approach can simplify decision making. The downfall of break-even analysis is that these conditions do not always exist.
I want to reiterate that these and other methods for evaluating security benefits can provide different insights, but none is likely to provide fully quantitative and definitive results. But most policy decisions are made without fully quantitative and definitive results, so DNDO should incorporate these benefits to provide the most informative CBA it can. Proceeding this way will undoubtedly result in greater insight than not including these considerations.
This concludes my testimony to the committee. Thank you for the opportunity to testify on this important topic. I would be happy to elaborate on any of my comments during the question and answer period.
1 The National Research Council is the operating arm of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine, chartered by Congress in 1863 to advise the government on matters of science and technology. The Nuclear and Radiation Studies Board is responsible for oversight of National Research Council studies on safety and security of nuclear materials and waste.
2 The report is titled Evaluating Testing, Costs, and Benefits of Advanced Spectroscopic Portals for Screening Cargo at Ports of Entry: Interim Report. The abbreviated version of the report is available online at http://www.nap.edu/catalog.php?record_id=12699
3 Dr. Robert Dynes, a physicist at the University of California, member of the National Academy of Sciences, and former president of the University of California, chaired this study.
4 Government Accountability Office. 2008. COMBATING NUCLEAR SMUGGLING: DHS’s Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results. GAO-08-979.
5 Shea, D.A., J.D. Moteff, D. Morgan. 2009. The Advanced Spectroscopic Portal Program: Background and Issues for Congress. Congressional Research Service, RL34750.
6 It would be more appropriate for an analysis of the tradeoffs among different spending options across the Global Architecture to be carried out at a higher level than the ASP cost-benefit analysis so that it can provide guidance and support for multiple programs in a coordinated fashion. The committee has, however, seen no evidence that the higher level tradeoff analysis has been done, and a 2009 report by the GAO confirmed this view.
7 The ASP-C is the variant of the ASP designed for screening truck-borne containerized cargo, and to date is the only ASP tested in support of certification.
8 Some of the fastest-growing ports in North America include Manzanillo, Lazaro Cardenas, Vancouver, and Prince Rupert.
9 Davis, P.K. 2002. Analytic Architecture for Capabilities-Based Planning, Mission-System Analysis, and Transformation. Report # MG-1513-OSD, RAND Corporation, Santa Monica, CA.
10 Ayres, I and S.D. Levitt. 1998. Measuring Positive Externalities from Unobservable Victim Precaution: An Empirical Analysis of Lojack. The Quarterly Journal of Economics, v. 113, no. 1: 43-77.
11 Bier, V.M. and N. Haphuriwat. 2009. Analytical method to identify the number of containers to inspect at U.S. ports to deter terrorist attacks. Annals of Operations Research. November.
12 Willis, H.H. and T. LaTourrette. 2008. Using Probabilistic Terrorism Risk Modeling for Regulatory Benefit-Cost Analysis: Application to the Western Hemisphere Travel Initiative. Risk Analysis, v. 28, no. 2: 325-339.
13 Martonosi, S.E., D.S. Ortiz, H.H. Willis. 2005. Evaluating the viability of 100 percent container inspections at America’s ports. In H.W. Richardson, P. Gordon, and J.E. Moore II, The Economic Impacts of Terrorist Attacks. Edward Elgar Publishing, Cheltenham, UK.