Large-Scale “Expert” Mortality Surveys in Conflicts—Concerns and Recommendations

Abstract

SEVERAL LARGE-SCALE RETROSPECTIVE MORTALITY SURveys in conflict settings in Darfur, the Democratic Republic of Congo (DRC), Northern Uganda, and Iraq have had major political implications, and, thus, were scrutinized by policy makers, researchers, and the media. The controversies they generated led to serious criticism—some well-founded, some less so—that may have undermined the credibility of mortality surveys in conflict settings. For example, a 2006 survey estimated that more than 650 000 Iraqis died mostly from violence since the USled invasion in 2003; in contrast, another study found a substantially lower estimate of violence-related deaths at approximately 151 000. A 2007 study estimated that 5.4 million have died in DRC since 1998; another report questioned the methods of this study and claimed that the excess death estimate was at least 3 times too high. Rebuttals from various sources and conflicting studies focused on sampling and nonsampling biases. Given these concerns, higher standards and improved methods are needed for undertaking and reporting large-scale mortality surveys. To track mortality prospectively, a surveillance system should be established as early as possible in humanitarian emergencies. However, humanitarian organizations often rely on cross-sectional surveys to estimate death rates retrospectively. These surveys are time consuming, costly, logistically challenging, and require technical expertise. Methodologies to measure mortality have not been sufficiently validated. Mortality surveys are useful at the beginning of emergencies to document baseline mortality rates, monitor effectiveness of programs, prioritize interventions, and advocate for funding. These surveys generally are not published in peer-reviewed journals and may not attract much attention. These are the “operational” surveys that allow humanitarian organizations to make practical field decisions. They are neither designed to meet publication standards nor to establish precise benchmarks. Although their quality can be improved, these operational surveys are essential surveys to guide field programs. Large-scale mortality surveys in conflict settings—often seeking to measure mortality at a national level and over a relatively long period—have different objectives from smaller operational surveys. These “expert” surveys are rare and primarily conducted for advocacy purposes, to document severity of crises and possible human rights violations, and to call for additional international attention and funding. Their findings are broader in nature and often do not directly effect field interventions. These surveys are complex, expensive, and time consuming to implement and to disseminate results. Sampling design and methods of retrospective mortality surveys in conflict settings are areas of unsettled science. Lack of accurate population data before the crisis is complicated by population displacement, so accurate population listings necessary for random sampling are rarely available. Thus, cluster sampling is a default method in most humanitarian settings and has often meant reliance on 30 30 cluster surveys. Given the uneven distribution of violence, sample sizes, particularly number of clusters, should be significantly larger than have previously been used; especially because many surveys attempt to interpret findings by age, sex, or location. Therefore, 30 30 cluster surveys should rarely, if ever, be used in expert mortality surveys if the purpose is to estimate mortality at a national level. Larger sample sizes with more clusters are needed. Recall periods are another important issue. Shorter periods (eg, 1-3 months) may reduce recall bias and focus results on more proximate mortality events. However, they tend to increase imprecision in estimating rates for a given sample size because the number of deaths in shorter periods is likely to be fewer. Calendar error may also prove more significant in shorter recall periods because an error of only 1 week (eg, when a death occurred) would have a greater effect on estimates. Duration of recall periods in expert surveys should be appropriate to the time dimensions of the crisis-related mortality of interest while ensuring the sample power is sufficient for precision and valid comparisons. Nonsampling errors may be particularly important in expert surveys. Surveyors and survey participants may be under pressure to increase or decrease reported deaths. Who the surveyors are, how they are trained and super-