This report presents our findings from NIOSH Research Contract No. 2008-N-10989: "Designing a Pilot Program for Strategic Mine Safety and Health Improvements through the Use of Surveillance Data to Guide Targeted Inspection Activities." The primary goal of the project was to use Bayesian time-series forecasting methods to develop a statistical algorithm for targeting high-risk mines that MSHA personnel might ultimately use as an enforcement tool. Although primarily reliant on MSHA's extensive historical data on each U.S. mine's regulatory violations and reported injuries, the algorithm also incorporates some information collected by the Department of Energy's Energy Information Administration on large coal mines.
In the context of MSHA regulation, targeting can be defined as a systematic, data-driven effort to single out the most dangerous mines for inspection. The flexibility of Bayesian time-series forecasting makes this approach particularly well-suited to dynamic regulatory environments like mining, in which political, statutory, and technological factors may alter the mix of safety hazards and operators' incentives over time. The algorithm is constructed in a way that enables it to incorporate data on a mine's past behavior, update its predictions in real time based on new inspection data, and adjust its parameters to reflect changes in underlying conditions (such as a legislative reform or shift in technology).
First and foremost, the study highlights the overriding importance of specifying an appropriate "target" for regulatory targeting. It is not obvious, a priori, precisely which goal(s) the agency should be trying to further, and therefore which mines it should be seeking to target. We considered the following three goals and corresponding selection criteria:
1. Mitigate risks faced by miners at the nation's most dangerous mines by targeting mines with the highest predicted injury rates;
2. Save as many lives (and avert as many injuries) as possible nationwide by targeting mines with the highest expected number of injuries; and
3. Ensure that miners' remuneration reflects the level of hazards they face on the job by targeting mines in which compensation levels do not reflect miners' true levels of occupational risk (a situation suggestive of market failure).
In our view, all three of these criteria have strong theoretical and practical arguments to recommend them. The third criterion, however, cannot be implemented at this time because mine-level data on wages and fringe benefits do not exist. Therefore, we confined our analysis to the first two approaches: targeting mines with the highest predicted injury rates (a rate-targeting algorithm), and targeting mines with the highest predicted injury counts (a count-targeting algorithm).
We used three different metrics to evaluate each algorithm's performance: one that focuses on the algorithm's ability to select mines that report non-zero injuries; one that estimates the maximum number of injuries might be prevented; and one that compares average injury rates across targeted and non-targeted mines. At each stage of the analysis, we compared the performance of each algorithm to a "naive" predictive model based on each mine's average injury rate or count during the previous four quarters. Overall, our results are encouraging. Regardless of the metric examined, both algorithms mostly - although not universally - outperform their naive baselines. The critical factors that determine each algorithm's performance are the number of mines targeted and, to a lesser extent, the injury type examined (i.e., total, traumatic, or fatal injuries). A second important finding is that the rate-targeting and count-targeting algorithms have very different properties. The count-targeting algorithm almost universally outperforms the rate-targeting algorithm by every metric examined. However, the two algorithms outperform their respective naive baselines within different ranges and to varying degrees. Overall, we conclude that using statistical targeting techniques to prioritize mines for inspection would likely help MSHA make better use of its scarce enforcement resources, regardless of whether the agency chooses to target injury rates, injury counts, or some mixture of the two.
In addition to completing our primary task of designing and testing the statistical targeting algorithms, we investigated several other policy-relevant correlates of mine safety. These supplementary empirical analyses focused on four questions. First and most importantly, we probed the effect of unionization on coal mine safety and on the strength of regulatory enforcement. This line of inquiry was pursued extensively and yielded two academic publications. As described in Section 4.1 (page 25), unionization predicts a large and significant decline in traumatic injuries and fatalities, especially among large mines. Unionized mines also undergo longer, more frequent and more intense MSHA inspections, although this effect declines sharply with mine size. The remaining three questions - the relationship between mine size and safety, the impact of MSHA inspections on regulatory compliance, and the relationship between reported injuries and regulatory compliance - were explored in a much more cursory fashion. Nevertheless, as reported in Sections 4.2 through 4.4 these preliminary analyses did bring to light several puzzling patterns that we feel are sufficiently intriguingto warrant further investigation.
