Population Health Metrics Research Consortium Research Articles

Revising the WHO verbal autopsy instrument to facilitate routine cause-of-death monitoring

ObjectiveVerbal autopsy (VA) is a systematic approach for determining causes of death (CoD) in populations without routine medical certification. It has mainly been used in research contexts and involved relatively lengthy interviews. Our objective here is to describe the process used to shorten, simplify, and standardise the VA process to make it feasible for application on a larger scale such as in routine civil registration and vital statistics (CRVS) systems.MethodsA literature review of existing VA instruments was undertaken. The World Health Organization (WHO) then facilitated an international consultation process to review experiences with existing VA instruments, including those from WHO, the Demographic Evaluation of Populations and their Health in Developing Countries (INDEPTH) Network, InterVA, and the Population Health Metrics Research Consortium (PHMRC). In an expert meeting, consideration was given to formulating a workable VA CoD list [with mapping to the International Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) CoD] and to the viability and utility of existing VA interview questions, with a view to undertaking systematic simplification.FindingsA revised VA CoD list was compiled enabling mapping of all ICD-10 CoD onto 62 VA cause categories, chosen on the grounds of public health significance as well as potential for ascertainment from VA. A set of 221 indicators for inclusion in the revised VA instrument was developed on the basis of accumulated experience, with appropriate skip patterns for various population sub-groups. The duration of a VA interview was reduced by about 40% with this new approach.ConclusionsThe revised VA instrument resulting from this consultation process is presented here as a means of making it available for widespread use and evaluation. It is envisaged that this will be used in conjunction with automated models for assigning CoD from VA data, rather than involving physicians.

Assessing verbal autopsy as a component of vital registration: a data-driven simulation study

BackgroundVerbal autopsy (VA) has been proposed as an alternative or supplement to mortality estimation in countries with low vital registration (VR) coverage. Low VR in countries can result in inaccurate estimations of disease burden and inefficient allocation of resources. We used VA and VR data from the states of Hidalgo and Morelos (Mexico) to explore the reliability of augmenting incomplete VR with VA data. MethodsAs part of the Population Health Metrics Research Consortium study, we identified death certificates from 2009 in the states of Hidalgo and Morelos, Mexico, and created a sample. We used data on location of death to simulate 100 scenarios of VR coverage ranging from 10% to 90% and augmented each with VA data to simulate full coverage, using different strategies to blend VR and VA information. We attributed underlying cause of death to the VAs using the Tariff method, an additive algorithm that predicts 34 causes of death for adults, 21 for children, and six for neonates. We measured the reliability of cause fraction predictions comparing VR only to VR and VAs. To quantify uncertainty, we replicated this 500 times for each scenario. FindingsAs the percent (simulated) VR coverage decreased and the missing deaths were assigned using VAs, the mean absolute difference in cause fractions for adults increased from 0·219% (95% CI 0·217–0·221) at the 90% VR coverage level to 1·84% (1·82–1.86) at the 10% VR coverage level. These differences were greater among child and neonatal deaths, and varied by cause of death. InterpretationA mean change of two percentage points in cause fractions from a 34-cause list is substantial when the true cause fractions can fall below 2%. This study shows that countries whose mortality estimates are based on incomplete VR data could see substantial changes in estimated cause composition if they supplemented their VR with VA data. FundingBill & Melinda Gates Foundation through grant 51488 “Population Health Metrics Research Consortium—Mexico”.

Can computers measure the chronic disease burden using survey questionnaires? The Symptomatic Diagnosis Study

BackgroundMeasuring the burden of chronic conditions in the developing world is a critical global health challenge. Scarce resources and the lack of biometry tests for chronic conditions such as depression and arthritis limit current measurement. Computer-based diagnosis based on self-reported signs and symptoms (symptomatic diagnosis) is a promising method for more accurately measuring the chronic disease burden. MethodsAs part of the Population Health Metrics Research Consortium (PHMRC) study, we collected 1379 questionnaires in Mexico from individuals who had a chronic condition that had been diagnosed with gold-standard diagnostic criteria or individuals who did not have any of the ten target conditions. Our primary analytical goal was to develop an algorithm to accurately diagnose chronic conditions. To this end, we tested methods previously developed for verbal autopsy. We analysed the performance of each method and compared performance to existing epidemiological measurement techniques. FindingsThe top-performing method is capable of achieving 68% concordance with gold-standard diagnosis. Concordance ranged from approximately 90% for depression, angina pectoris, and cirrhosis, to 40% for osteoarthritis and vision loss. The prevalence fraction of each condition could be measured with less than 3% absolute error. These findings roughly parallel validated verbal autopsy methods at an expectedly higher performance level. InterpretationSymptomatic diagnosis outperforms current techniques and is a viable method for measuring the burden of chronic diseases in areas with low health information infrastructure. It is a critical global health challenge to better characterise the epidemiology of chronic conditions in these areas, and is a powerful, unique solution capable of collecting an array of prevalence data in a single survey. This technology can provide myriad benefits to the field of epidemiology, including higher-resolution prevalence data, flexible data collection with rapid interpretability, and individual diagnosis for certain conditions. FundingThe Population Health Metrics Research Consortium funded the data collection as part of a Gates Grand Challenges in Global Health initiative (GC-13).

Ensemble modelling in verbal autopsy: the Popular Voting method

BackgroundVerbal autopsy (VA) is a highly valuable tool for assessing causes of death in resource-limited settings without medically certified death certificates. The Population Health Metrics Research Consortium (PHMRC) collected 12 535 VAs in four countries for which the true cause of death was reliably known. This project led to the development of three new computer algorithms to determine cause of death from these VAs, all of which predict underlying cause of death more accurately than the status quo: physician review. Concurrently, ensemble models, or blends of well-performing models, have been shown to have favourable predictive validity and have begun to be implemented in global health metrics settings. MethodsWe developed a simple ensemble model based on the three top performing PHMRC methods: the Simplified Symptom Pattern (SSP), the Tariff, and the Random Forest (RF). This ensemble method functions at the individual-record level, examining the predicted cause of death from the three component models and selecting cause of death by a simple majority (Popular Voting). Sensitivity analyses revealed that selecting the prediction made by RF in cases where all three models disagreed was preferable, and this ensemble method was adapted accordingly. FindingsThe Popular Voting method performed better in cause-specific mortality fraction accuracy than did any individual model alone for adults, children, and neonates, and performed better in chance-corrected concordance than did any individual model except SSP in adults. The three component models disagreed in 16% of all cases, and unanimously agreed in 47% of cases. InterpretationAs VA continues to be an effective source of data for estimating cause of death, accurate and inexpensive methods for analysing VA interview responses are increasingly important. The recent development of the three highly accurate PHMRC computational models allows for the option of a meta-model such as the ensemble introduced here. This ensemble model for VA achieves superior performance, and could be applied to other VA samples to accurately assess the relative mortality burden from a variety of diseases and injuries. FundingPopulation Health Metrics Research Consortium.

Development of a shortened verbal autopsy instrument for routine surveillance

BackgroundVerbal autopsies (VAs) have been widely used to determine cause of death for research in developing countries, but their use in vital registration systems has been limited due to the interview time of VA instruments (VAIs). This study aimed to create a shortened VAI to address this issue by using the Population Health Metrics Research Consortium (PHMRC) gold standard VA validation dataset and the Tariff method to systematically reduce the size of the PHMRC VAI. MethodsThe Tariff method calculates a score for each symptom–cause pair to measure strength of association between the two. The effect on performance after low-ranking symptoms were deleted from the VAI was measured using cause-specific mortality fraction (CSMF) accuracy and chance-corrected concordance (CCC) as test statistics at population and individual levels, respectively. FindingsWith negligible effect on performance, we were able to reduce the number of items from 183 to 105 in the adult VAI, from 127 to 72 in the child VAI, and from 149 to 76 in the neonatal VAI. CSMF accuracy in adults decreased from 0·770 (95% CI 0·766–0·775) in the full VAI to 0·770 (0·765–0·775) in the reduced VAI, in children from 0·785 (0·776–0·789) to 0·783 (0·776–0·787), and in neonates from 0·827 (0·818–0·835) to 0·818 (0·812–0·827). Mean CCC in adults decreased from 50·5% (95% CI 50·3–50·6) to 49·7% (49·5–49·8), in children from 52·3% (51·5–52·5) to 51·9% (51·6–52·2), and in neonates from 48·2% (47·9–48·5) to 47·5% (47·2–47·7). InterpretationWe have been able to reduce the number of items by over 40% in the PHMRC VAI while retaining its performance characteristics, thus removing a major impediment to its routine use in vital registration systems. This shortened VAI is currently being piloted to determine the effects of reduced item number on survey time. FundingThe Health Information Systems Knowledge Hub of the University of Queensland, a strategic partnership initiative of the Australian Agency for International Development.

Population Health Metrics Research Consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets.

BackgroundVerbal autopsy methods are critically important for evaluating the leading causes of death in populations without adequate vital registration systems. With a myriad of analytical and data collection approaches, it is essential to create a high quality validation dataset from different populations to evaluate comparative method performance and make recommendations for future verbal autopsy implementation. This study was undertaken to compile a set of strictly defined gold standard deaths for which verbal autopsies were collected to validate the accuracy of different methods of verbal autopsy cause of death assignment.MethodsData collection was implemented in six sites in four countries: Andhra Pradesh, India; Bohol, Philippines; Dar es Salaam, Tanzania; Mexico City, Mexico; Pemba Island, Tanzania; and Uttar Pradesh, India. The Population Health Metrics Research Consortium (PHMRC) developed stringent diagnostic criteria including laboratory, pathology, and medical imaging findings to identify gold standard deaths in health facilities as well as an enhanced verbal autopsy instrument based on World Health Organization (WHO) standards. A cause list was constructed based on the WHO Global Burden of Disease estimates of the leading causes of death, potential to identify unique signs and symptoms, and the likely existence of sufficient medical technology to ascertain gold standard cases. Blinded verbal autopsies were collected on all gold standard deaths.ResultsOver 12,000 verbal autopsies on deaths with gold standard diagnoses were collected (7,836 adults, 2,075 children, 1,629 neonates, and 1,002 stillbirths). Difficulties in finding sufficient cases to meet gold standard criteria as well as problems with misclassification for certain causes meant that the target list of causes for analysis was reduced to 34 for adults, 21 for children, and 10 for neonates, excluding stillbirths. To ensure strict independence for the validation of methods and assessment of comparative performance, 500 test-train datasets were created from the universe of cases, covering a range of cause-specific compositions.ConclusionsThis unique, robust validation dataset will allow scholars to evaluate the performance of different verbal autopsy analytic methods as well as instrument design. This dataset can be used to inform the implementation of verbal autopsies to more reliably ascertain cause of death in national health information systems.

Assessing quality of medical death certification: Concordance between gold standard diagnosis and underlying cause of death in selected Mexican hospitals

BackgroundIn Mexico, the vital registration system relies on information collected from death certificates to generate official mortality figures. Although the death certificate has high coverage across the country, there is little information regarding its validity. The objective of this study was to assess the concordance between the underlying cause of death in official statistics obtained from death certificates and a gold standard diagnosis of the same deaths derived from medical records of hospitals.MethodsThe study sample consisted of 1,589 deaths that occurred in 34 public hospitals in the Federal District and the state of Morelos, Mexico in 2009. Neonatal, child, and adult cases were selected for causes of death that included infectious diseases, noncommunicable diseases, and injuries. We compared the underlying cause of death, obtained from medical death certificates, against a gold standard diagnosis derived from a review of medical records developed by the Population Health Metrics Research Consortium. We used chance-corrected concordance and accuracy as metrics to evaluate the quality of performance of the death certificate.ResultsAnalysis considering only the underlying cause of death resulted in a median chance-corrected concordance between the cause of death in medical death certificates versus the gold standard of 54.3% (95% uncertainty interval [UI]: 52.2, 55.6) for neonates, 38.5% (37.0, 40.0) for children, and 66.5% (65.9, 66.9) for adults. The accuracy resulting from the same analysis was 0.756 (0.747, 0.769) for neonates, 0.683 (0.663, 0.701) for children, and 0.780 (0.774, 0.785) for adults. Median chance-corrected concordance and accuracy increased when considering the mention of any cause of death in the death certificate, not just the underlying cause. Concordance varied substantially depending on cause of death, and accuracy varied depending on the true cause-specific mortality fraction composition.ConclusionsAlthough we cannot generalize our conclusions to Mexico as a whole, the results demonstrate important problems with the quality of the main source of information for causes of death used by decision-makers in settings with highly technological vital registration systems. It is necessary to improve death certification procedures, especially in the case of child and neonatal deaths. This requires an important commitment from the health system and health institutions.

Performance of physician-certified verbal autopsies: multisite validation study using clinical diagnostic gold standards

BackgroundPhysician review of a verbal autopsy (VA) and completion of a death certificate remains the most widely used approach for VA analysis. This study provides new evidence about the performance of physician-certified verbal autopsy (PCVA) using defined clinical diagnostic criteria as a gold standard for a multisite sample of 12,542 VAs. The study was also designed to analyze issues related to PCVA, such as the impact of a second physician reader on the cause of death assigned, the variation in performance with and without household recall of health care experience (HCE), and the importance of local information for physicians reading VAs.MethodsThe certification was performed by 24 physicians. The assignment of VA was random and blinded. Each VA was certified by one physician. Half of the VAs were reviewed by a different physician with household recall of health care experience included. The completed death certificate was processed for automated ICD-10 coding of the underlying cause of death. PCVA was compared to gold standard cause of death assignment based on strictly defined clinical diagnostic criteria that are part of the Population Health Metrics Research Consortium (PHMRC) gold standard verbal autopsy study.ResultsFor individual cause assignment, the overall chance-corrected concordance for PCVA against the gold standard cause of death is less than 50%, with substantial variability by cause and physician. Physicians assign the correct cause around 30% of the time without HCE, and addition of HCE improves performance in adults to 45% and slightly higher in children to 48%. Physicians estimate cause-specific mortality fractions (CSMFs) with considerable error for adults, children, and neonates. Only for neonates for a cause list of six causes with HCE is accuracy above 0.7. In all three age groups, CSMF accuracy improves when household recall of health care experience is available.ConclusionsResults show that physician coding for cause of death assignment may not be as robust as previously thought. The time and cost required to initially collect the verbal autopsies must be considered in addition to the analysis, as well as the impact of diverting physicians from servicing immediate health needs in a population to review VAs. All of these considerations highlight the importance and urgency of developing better methods to more reliably analyze past and future verbal autopsies to obtain the highest quality mortality data from populations without reliable death certification.

Direct estimation of cause-specific mortality fractions from verbal autopsies: multisite validation study using clinical diagnostic gold standards

BackgroundVerbal autopsy (VA) is used to estimate the causes of death in areas with incomplete vital registration systems. The King and Lu method (KL) for direct estimation of cause-specific mortality fractions (CSMFs) from VA studies is an analysis technique that estimates CSMFs in a population without predicting individual-level cause of death as an intermediate step. In previous studies, KL has shown promise as an alternative to physician-certified verbal autopsy (PCVA). However, it has previously been impossible to validate KL with a large dataset of VAs for which the underlying cause of death is known to meet rigorous clinical diagnostic criteria.MethodsWe applied the KL method to adult, child, and neonatal VA datasets from the Population Health Metrics Research Consortium gold standard verbal autopsy validation study, a multisite sample of 12,542 VAs where gold standard cause of death was established using strict clinical diagnostic criteria. To emulate real-world populations with varying CSMFs, we evaluated the KL estimations for 500 different test datasets of varying cause distribution. We assessed the quality of these estimates in terms of CSMF accuracy as well as linear regression and compared this with the results of PCVA.ResultsKL performance is similar to PCVA in terms of CSMF accuracy, attaining values of 0.669, 0.698, and 0.795 for adult, child, and neonatal age groups, respectively, when health care experience (HCE) items were included. We found that the length of the cause list has a dramatic effect on KL estimation quality, with CSMF accuracy decreasing substantially as the length of the cause list increases. We found that KL is not reliant on HCE the way PCVA is, and without HCE, KL outperforms PCVA for all age groups.ConclusionsLike all computer methods for VA analysis, KL is faster and cheaper than PCVA. Since it is a direct estimation technique, though, it does not produce individual-level predictions. KL estimates are of similar quality to PCVA and slightly better in most cases. Compared to other recently developed methods, however, KL would only be the preferred technique when the cause list is short and individual-level predictions are not needed.

Synergism of verbal autopsy and diagnostic pathology autopsy for improved accuracy of mortality data

Commentary This series provides an important opportunity to consider how diagnostic pathology autopsy could be used in conjunction with verbal autopsy to provide more accurate cause of death and mortality data in all countries, and specifically in those countries with inadequate or nonexistent death registrations systems. For the purposes of this commentary, the term “autopsy” will denote the medical-pathology diagnostic procedure, in contrast to “verbal autopsy.” The term “autopsy” means “to see or observe for oneself,” but traditional use has been reserved for the postmortem examination of a (dead) body by a physician/ pathologist, in order to identify diseases and injuries and determine the cause(s) of death. This medical-diagnostic pathology procedure integrates trained observation of the external and internal body with dissection or other invasive procedures, in order to obtain tissue samples, which are evaluated by microscopy and other specialized laboratory modalities, including chemical, toxicologic, genetic, and molecular biologic analyses. Used more broadly, the term “autopsy” reflects the aggregate of procedures used for postmortem medical diagnosis or death investigation, including investigative procedures that identify information about the deceased’s medical history and the circumstances and scene of his/her death. To most pathologists and physicians, the term “verbal autopsy” seems a contradiction in terms. However, it is a clearly defined procedure, which allows classification of cause of death and cause-s pecific mortality by the analysis of data derived from structured interviews of family, friends, and caretakers, as well as review of any available medical records [1]. As more than two-thirds of the world’s population lives and dies in countries that lack functional vital registration systems, and in which most deaths occur outside of medical facilities and are neither enumerated nor classified by cause, verbal autopsy has become the primary methodology for determining population-based cause-specific mortality [2,3]. The development of computerized algorithmic systems for determination of cause of death by analysis of verbal autopsy data is a major focus of health metrics research, and emphasis is currently focused on using the recentlycompleted dataset from the Population Health Metrics Research Consortium (PHMRC) project that will allow analysis of verbal autopsy data collected from more than 12,000 hospitalized patients with causes of death established by rigorous clinical criteria.