Quality Control and Criminal Justice

At first glance, airplane crashes, medical mistakes, and lab accidents may appear to be unrelated to the criminal justice system. However, a recent symposium held at the University of Pennsylvania Law School examined how error reduction and quality control systems in other fields might help improve the fairness and accuracy of the criminal justice system.

The symposium, sponsored by the recently established Quattrone Center for the Fair Administration of Justice at Penn Law, began with a panel of experts on quality control systems in health care, aviation, and laboratory science. The panelists discussed how the lessons learned in other sectors could apply to the criminal justice system.

Cary Coglianese, of the Penn Program on Regulation, moderated the opening panel. Just as it is widely recognized that “to err is human,” Coglianese noted that taking steps to reduce errors has been a longstanding and fundamental concern in a wide variety of important human endeavors. Efforts to reduce errors in the criminal justice system can benefit from an examination of how quality control has been addressed in other complex and interconnected systems, he said.

Lucian Leape notes how law might be able to learn from the medical field.

The panel’s first speaker, Lucian Leape of the Harvard University School of Public Health, began by noting the similarities between medicine and law: they are dominated by individuals who have power over their subjects, and from the outside, they both appear to be incredibly inefficient, bureaucratic systems. Because of these similarities, law may be able to learn from medicine’s attempt to reduce errors. Dr. Leape noted that the checklists and other management tools used in aviation had been particularly effective in reducing the error rate in medicine and that they may be helpful to law as well.

The second speaker, Lee Fleisher, a professor from the University of Pennsylvania School of Medicine, indicated that, in addition to the measures described by Dr. Leape, a focus on data and measurable outcomes has proven instrumental in improving quality control in medicine. The challenge for the criminal justice system will be to create a process that can generate agreement on a common set of data metrics for quality measurement over time and across different jurisdictions.

Lee Fleisher argues that being able to quantify and measure outcomes could improve the criminal justice system.

Christopher Hart, the Vice-Chairman of the National Transportation Safety Board, followed by describing the similarities between criminal justice “mishaps” and aviation mishaps. He suggested that a wrongful conviction in the criminal justice system bears affinities with an error in aviation, such as a plane crash. Both errors, he noted, could probably be traced to unwanted and often unforeseen interactions between interdependent subsystems in a larger, complex system.

Reducing errors in complex systems requires collaboration to identify and prioritize potential safety issues and evaluate whether the proposed solutions actually work, said Hart. In aviation, Hart said collaboration worked successfully to decrease fatal aviation accidents by 65% from 1997-2007.

Christopher Hart compares criminal justice mishaps to aviation mishaps, emphasizing a need for collaboration.

However, Hart cautioned that this collaborative approach to examining system wide safety concerns was not without its challenges. He indicated that conflicts can arise among competing interests and that regulators’ ability to collaborate is complicated by their enforcement role. To address these challenges, he suggested that individuals begin their analysis from a common starting point which recognizes that everyone involved in the process wants to improve the system. Furthermore, he also recommended that the way to start to improve a system is to focus first on a longstanding problem that resists remedies. If the problem is longstanding, he reasoned, that means it probably results from defective systems and processes rather than from the individuals who are involved.

Susan Silbey, a sociologist and anthropologist from the Massachusetts Institute of Technology, concluded the panel by assessing ways that have been used by scientific laboratories to minimize hazards. She indicated that although many researchers and professionals emphasize safety culture as a means of reducing risk, such an emphasis on a “safety culture” can sometimes obscure the very concrete steps needed to reduce errors. She noted that, in both university laboratories and the criminal justice system, quality improvements depend on a consistent and collaborative feedback loop between observed outcomes and corrective actions.

Susan Silbey describes how methods to improve university laboratories could be used to improve the criminal justice system.

Silbey also noted that sometimes when systems for collecting safety-related data are designed, managers are sometimes too afraid of how data will be used, and as a result the type and nature of the data they decide to collect tend to be limited, both in detail and usefulness.  She suggested that rather than depending completely on support by top-level management, successful organizational change often depends on the support of mid-level participants who embrace new practices and procedures.

In the criminal justice system, change is needed if errors in convicting innocent individuals – as well as failing to identify and punish guilty individuals – are to be effectively reduced. In the question-and-answer period that followed the panelists’ presentations, Dr. Leape reiterated the value of devising a checklist for the appropriate steps that should be followed in a prosecution (similar to what is done in surgeries or aviation), allowing for an analysis on which steps had the highest rates of failure. The establishment of a system that generates such data, he indicated, would constitute the first step towards reducing the frequency of systemic errors in the criminal justice system.