Cause-specific Mortality Fraction Accuracy Research Articles

Agreement between cause of death assignment by computer-coded verbal autopsy methods and physician coding of verbal autopsy interviews in South Africa

ABSTRACT Background The South African national cause of death validation (NCODV 2017/18) project collected a national sample of verbal autopsies (VA) with cause of death (COD) assignment by physician-coded VA (PCVA) and computer-coded VA (CCVA). Objective The performance of three CCVA algorithms (InterVA-5, InSilicoVA and Tariff 2.0) in assigning a COD was compared with PCVA (reference standard). Methods Seven performance metrics assessed individual and population level agreement of COD assignment by age, sex and place of death subgroups. Positive predictive value (PPV), sensitivity, overall agreement, kappa, and chance corrected concordance (CCC) assessed individual level agreement. Cause-specific mortality fraction (CSMF) accuracy and Spearman’s rank correlation assessed population level agreement. Results A total of 5386 VA records were analysed. PCVA and CCVAs all identified HIV/AIDS as the leading COD. CCVA PPV and sensitivity, based on confidence intervals, were comparable except for HIV/AIDS, TB, maternal, diabetes mellitus, other cancers, and some injuries. CCVAs performed well for identifying perinatal deaths, road traffic accidents, suicide and homicide but poorly for pneumonia, other infectious diseases and renal failure. Overall agreement between CCVAs and PCVA for the top single cause (48.2–51.6) indicated comparable weak agreement between methods. Overall agreement, for the top three causes showed moderate agreement for InterVA (70.9) and InSilicoVA (73.8). Agreement based on kappa (−0.05–0.49)and CCC (0.06–0.43) was weak to none for all algorithms and groups. CCVAs had moderate to strong agreement for CSMF accuracy, with InterVA-5 highest for neonates (0.90), Tariff 2.0 highest for adults (0.89) and males (0.84), and InSilicoVA highest for females (0.88), elders (0.83) and out-of-facility deaths (0.85). Rank correlation indicated moderate agreement for adults (0.75–0.79). Conclusions Whilst CCVAs identified HIV/AIDS as the leading COD, consistent with PCVA, there is scope for improving the algorithms for use in South Africa.

Evaluation of methods for assigning causes of death from verbal autopsies in India.

Physician-coded verbal autopsy (PCVA) is the most widely used method to determine causes of death (COD) in countries where medical certification of death is low. Computer-coded verbal autopsy (CCVA), an alternative method to PCVA for assigning the COD is considered to be efficient and cost-effective. However, the performance of CCVA as compared to PCVA is yet to be established in the Indian context. We evaluated the performance of PCVA and three CCVA methods i.e., InterVA 5, InSilico, and Tariff 2.0 on verbal autopsies done using the WHO 2016 VA tool on 2,120 reference standard cases developed from five tertiary care hospitals of Delhi. PCVA methodology involved dual independent review with adjudication, where required. Metrics to assess performance were Cause Specific Mortality Fraction (CSMF), sensitivity, positive predictive value (PPV), CSMF Accuracy, and Kappa statistic. In terms of the measures of the overall performance of COD assignment methods, for CSMF Accuracy, the PCVA method achieved the highest score of 0.79, followed by 0.67 for Tariff_2.0, 0.66 for Inter-VA and 0.62 for InSilicoVA. The PCVA method also achieved the highest agreement (57%) and Kappa scores (0.54). The PCVA method showed the highest sensitivity for 15 out of 20 causes of death. Our study found that the PCVA method had the best performance out of all the four COD assignment methods that were tested in our study sample. In order to improve the performance of CCVA methods, multicentric studies with larger sample sizes need to be conducted using the WHO VA tool.

Assessment of the quality of the vital registration system for under-5 mortality in Yucatan, Mexico

IntroductionVital registration is an important element in health information systems which can inform policy and strengthen health systems. Mexico has a well-functioning vital registration system; however, there is still room for improvement, especially for deaths of children under 5. This study assesses the quality of the vital registration system in capturing deaths and evaluates the quality of cause of death certification in under-5 deaths in Yucatan, Mexico.MethodsWe collected information on under-5 deaths that occurred in 2015 and 2016 in Yucatan, Mexico. We calculated the Vital Statistics Performance Index (VSPI) to have a general assessment of the vital registration performance. We examined the agreement between vital registration records and medical records at the individual and population levels using the chance-corrected concordance (CCC) and cause-specific mortality fraction (CSMF) accuracy as quality metrics.ResultsWe identified 966 records from the vital registry for all under-5 deaths, and 390 were linked to medical records of deaths occurring at public hospitals. The Yucatan vital registration system captured 94.8% of the expected under-5 deaths, with an overall VSPI score of 87.2%. Concordance between underlying cause of death listed in the vital registry and the cause determined by the medical record review varied substantially across causes, with a mean overall chance-corrected concordance across causes of 6.9% for neonates and 46.9% for children. Children had the highest concordance for digestive diseases, and neonates had the highest concordance for meningitis/sepsis. At the population level, the CSMF accuracy for identifying the underlying cause listed was 35.3% for neonates and 67.7% for children.ConclusionsAlthough the vital registration system has overall good performance, there are still problems in information about causes of death for children under 5 that are related mostly to certification of the causes of death. The accuracy of information can vary substantially across age groups and causes, with causes reported for neonates being generally less reliable than those for older children. Results highlight the need to implement strategies to improve the certification of causes of death in this population.

Are cause of death data for Shanghai fit for purpose? A retrospective study of medical records

ObjectivesTo assess the quality of cause of death reporting in Shanghai for both hospital and home deaths.Design and settingMedical records review (MRR) to independently establish a reference data set against...

Validation of SmartVA using conventional autopsy: A study of adult deaths in Brazil

Accurate cause of death data are essential to guide health policy. However, mortality surveillance is limited in many low-income countries. In such settings, verbal autopsy (VA) is increasingly used to provide population-level cause of death data. VAs are now widely interpreted using the automated algorithms SmartVA and InterVA. Here we use conventional autopsy as the gold standard to validate SmartVA methodology. This study included adult deaths from natural causes in São Paulo and Recife for which conventional autopsy was indicated. VA was conducted with a relative of the deceased using an amended version of the SmartVA instrument to suit the local context. Causes of death from VA were produced using the SmartVA-Analyze program. Physician coded verbal autopsy (PCVA), conducted on the same questionnaires, and Global Burden of Disease Study data were used as additional comparators. Cause of death data were grouped into 10 broad causes for the validation due to the real-world utility of VA lying in identifying broad population cause of death patterns. The study included 2,060 deaths in São Paulo and 1,079 in Recife. The cause specific mortality fractions (CSMFs) estimated using SmartVA were broadly similar to conventional autopsy for: cardiovascular diseases (46.8% vs 54.0%, respectively), cancers (10.6% vs 11.4%), infections (7.0% vs 10.4%) and chronic respiratory disease (4.1% vs 3.7%), causes accounting for 76.1% of the autopsy dataset. The SmartVA CSMF estimates were lower than autopsy for "Other NCDs" (7.8% vs 14.6%) and higher for diabetes (13.0% vs 6.6%). CSMF accuracy of SmartVA compared to autopsy was 84.5%. CSMF accuracy for PCVA was 93.0%. The results suggest that SmartVA can, with reasonable accuracy, predict the broad cause of death groups important to assess a population's epidemiological transition. VA remains a useful tool for understanding causes of death where medical certification is not possible.

Limitations to current methods to estimate cause of death: a validation study of a verbal autopsy model.

Background: Accurate information on causes of death (CoD) is essential to estimate burden of disease, track global progress, prioritize cost-effective interventions, and inform policies to reduce mortality. In low-income settings, where a significant proportion of deaths take place at home or in poorly-resourced peripheral health facilities, data on CoD often relies on verbal autopsies (VAs). Validations of VAs have been performed against clinical diagnosis, but never before against an acceptable gold standard: the complete diagnostic autopsy (CDA). Methods: We have validated a computer-coded verbal autopsy method -the InterVA- using individual and population metrics to determine CoD against the CDA, in 316 deceased patients of different age groups who died in a tertiary-level hospital in Maputo, Mozambique between 2013 and 2015. Results: We found a low agreement of the model across all age groups at the individual (kappa statistic ranging from -0.030 to 0.232, lowest in stillbirths and highest in adults) and population levels (chance-corrected cause-specific mortality fraction accuracy ranging from -1.00 to 0.62, lowest in stillbirths, highest in children). The sensitivity in identifying infectious diseases was low (0% for tuberculosis, diarrhea, and disseminated infections, 32% for HIV-related infections, 33% for malaria and 36% for pneumonia). Of maternal deaths, 26 were assigned to eclampsia but only four patients actually died of eclampsia. Conclusions: These findings do not lead to building confidence in current estimates of CoD. They also call to the need to implement autopsy methods where they may be feasible, and to improve the quality and performance of current VA techniques.

Limitations to current methods to estimate cause of death: a validation study of a verbal autopsy model.

Background: Accurate information on causes of death (CoD) is essential to estimate burden of disease, track global progress, prioritize cost-effective interventions, and inform policies to reduce mortality. In low-income settings, where a significant proportion of deaths take place at home or in poorly-resourced peripheral health facilities, data on CoD often relies on verbal autopsies (VAs). Validations of VAs have been performed against clinical diagnosis, but never before against an acceptable gold standard: the complete diagnostic autopsy (CDA). Methods: We have validated a computer-coded verbal autopsy method -the InterVA- using individual and population metrics to determine CoD against the CDA, in 316 deceased patients of different age groups who died in a tertiary-level hospital in Maputo, Mozambique between 2013 and 2015. Results: We found a low agreement of the model across all age groups at the individual (kappa statistic ranging from -0.030 to 0.232, lowest in stillbirths and highest in adults) and population levels (chance-corrected cause-specific mortality fraction accuracy ranging from -1.00 to 0.62, lowest in stillbirths, highest in children). The sensitivity in identifying infectious diseases was low (0% for tuberculosis, diarrhea, and disseminated infections, 32% for HIV-related infections, 33% for malaria and 36% for pneumonia). Of maternal deaths, 26 were assigned to eclampsia but only four patients actually died of eclampsia. Conclusions: These findings do not lead to building confidence in current estimates of CoD. They also call to the need to implement autopsy methods where they may be feasible, and to improve the quality and performance of current VA techniques.

Comparative performance of verbal autopsy methods in identifying causes of adult mortality: A case study in India.

Background & objectives:Cause of death assignment from verbal autopsy (VA) questionnaires is conventionally accomplished through physician review. However, since recently, computer softwares have been developed to assign the cause of death. The present study evaluated the performance of computer software in assigning the cause of death from the VA, as compared to physician review.Methods:VA of 600 adult deaths was conducted using open- and close-ended questionnaires in Nandpur Kalour Block of Punjab, India. Entire VA forms were used by two physicians independently to assign the cause of death using the International Statistical Classification of Diseases and Related Health Problems (ICD)-10 codes. In case of disagreement between them, reconciliation was done, and in cases of persistent disagreements finally, adjudication was done by a third physician. InterVA-4-generated causes from close-ended questionnaires were compared using Kappa statistics with causes assigned by physicians using a questionnaire having both open- and close-ended questions. At the population level, Cause-Specific Mortality Fraction (CSMF) accuracy and P-value from McNemar’s paired Chi-square were calculated. CSMF accuracy indicates the absolute deviation of a set of proportions of causes of death out of the total number of deaths between the two methods.Results:The overall agreement between InterVA-4 and physician coding was ‘fair’ (κ=0.42; 95% confidence interval 0.38, 0.46). CSMF accuracy was found to be 0.71. The differences in proportions from the two methods were statistically different as per McNemar’s paired Chi-square test for ischaemic heart diseases, liver cirrhosis and maternal deaths.Interpretation & conclusions:In comparison to physicians, assignment of causes of death by InterVA- 4 was only ‘fair’. Hence, it may be appropriate to continue with physician review as the optimal option available in the current scenario.

Limitations to current methods to estimate cause of death: a validation study of a verbal autopsy model

Background: Accurate information on causes of death (CoD) is essential to estimate burden of disease, track global progress, prioritize cost-effective interventions, and inform policies to reduce mortality. In low-income settings, where a significant proportion of deaths take place at home or in poorly-resourced peripheral health facilities, data on CoD often relies on verbal autopsies (VAs). Validations of VAs have been performed against clinical diagnosis, but never before against an acceptable gold standard: the complete diagnostic autopsy (CDA). Methods: We have validated a computer-coded verbal autopsy method –the InterVA- using individual and population metrics to determine CoD against the CDA, in 316 deceased patients of different age groups who died in a tertiary-level hospital in Maputo, Mozambique between 2013 and 2015. Results: We found a low agreement of the model across all age groups at the individual (kappa statistic ranging from -0.030 to 0.232, lowest in stillbirths and highest in adults) and population levels (chance-corrected cause-specific mortality fraction accuracy ranging from -1.00 to 0.62, lowest in stillbirths, highest in children). The sensitivity in identifying infectious diseases was low (0% for tuberculosis, diarrhea, and disseminated infections, 32% for HIV-related infections, 33% for malaria and 36% for pneumonia). Of maternal deaths, 26 were assigned to eclampsia but only four patients actually died of eclampsia. Conclusions: These findings do not lead to building confidence in current estimates of CoD. They also call to the need to implement autopsy methods where they may be feasible, and to improve the quality and performance of current VA techniques.

Using health facility deaths to estimate population causes of neonatal and child mortality in four African countries

BackgroundVerbal autopsy is the main method used in countries with weak civil registration systems for estimating community causes of neonatal and 1–59-month-old deaths. However, validation studies of verbal autopsy methods are limited and assessment has been dependent on hospital-based studies, with uncertain implications for its validity in community settings. If the distribution of community deaths by cause was similar to that of facility deaths, or could be adjusted according to related demographic factors, then the causes of facility deaths could be used to estimate population causes.MethodsCauses of neonatal and 1–59-month-old deaths from verbal/social autopsy (VASA) surveys in four African countries were estimated using expert algorithms (EAVA) and physician coding (PCVA). Differences between facility and community deaths in individual causes and cause distributions were examined using chi-square and cause-specific mortality fractions (CSMF) accuracy, respectively. Multinomial logistic regression and random forest models including factors from the VASA studies that are commonly available in Demographic and Health Surveys were built to predict population causes from facility deaths.ResultsLevels of facility and community deaths in the four countries differed for one to four of 10 EAVA or PCVA neonatal causes and zero to three of 12 child causes. CSMF accuracy for facility compared to community deaths in the four countries ranged from 0.74 to 0.87 for neonates and 0.85 to 0.95 for 1–59-month-olds. Crude CSMF accuracy in the prediction models averaged 0.86 to 0.88 for neonates and 0.93 for 1–59-month-olds. Adjusted random forest prediction models increased average CSMF accuracy for neonates to, at most, 0.90, based on small increases in all countries.ConclusionsThere were few differences in facility and community causes of neonatal and 1–59-month-old deaths in the four countries, and it was possible to project the population CSMF from facility deaths with accuracy greater than the validity of verbal autopsy diagnoses. Confirmation of these findings in additional settings would warrant research into how medical causes of deaths in a representative sample of health facilities can be utilized to estimate the population causes of child death.

Limitations to current methods to estimate cause of death: a validation study of a verbal autopsy model.

Background: Accurate information on causes of death (CoD) is essential to estimate burden of disease, track global progress, prioritize cost-effective interventions, and inform policies to reduce mortality. In low-income settings, where a significant proportion of deaths take place at home or in poorly-resourced peripheral health facilities, data on CoD often relies on verbal autopsies (VAs). Validations of VAs have been performed against clinical diagnosis, but never before against an acceptable gold standard: the complete diagnostic autopsy (CDA). Methods: We have validated a computer-coded verbal autopsy method -the InterVA- using individual and population metrics to determine CoD against the CDA, in 316 deceased patients of different age groups who died in a tertiary-level hospital in Maputo, Mozambique between 2013 and 2015. Results: We found a low agreement of the model across all age groups at the individual (kappa statistic ranging from -0.030 to 0.232, lowest in stillbirths and highest in adults) and population levels (chance-corrected cause-specific mortality fraction accuracy ranging from -1.00 to 0.62, lowest in stillbirths, highest in children). The sensitivity in identifying infectious diseases was low (0% for tuberculosis, diarrhea, and disseminated infections, 32% for HIV-related infections, 33% for malaria and 36% for pneumonia). Of maternal deaths, 26 were assigned to eclampsia but only four patients actually died of eclampsia. Conclusions: These findings do not lead to building confidence in current estimates of CoD. They also call to the need to implement autopsy methods where they may be feasible, and to improve the quality and performance of current VA techniques.

Automatically determining cause of death from verbal autopsy narratives

BackgroundA verbal autopsy (VA) is a post-hoc written interview report of the symptoms preceding a person’s death in cases where no official cause of death (CoD) was determined by a physician. Current leading automated VA coding methods primarily use structured data from VAs to assign a CoD category. We present a method to automatically determine CoD categories from VA free-text narratives alone.MethodsAfter preprocessing and spelling correction, our method extracts word frequency counts from the narratives and uses them as input to four different machine learning classifiers: naïve Bayes, random forest, support vector machines, and a neural network.ResultsFor individual CoD classification, our best classifier achieves a sensitivity of.770 for adult deaths for 15 CoD categories (as compared to the current best reported sensitivity of.57), and.662 with 48 WHO categories. When predicting the CoD distribution at the population level, our best classifier achieves.962 cause-specific mortality fraction accuracy for 15 categories and.908 for 48 categories, which is on par with leading CoD distribution estimation methods.ConclusionsOur narrative-based machine learning classifier performs as well as classifiers based on structured data at the individual level. Moreover, our method demonstrates that VA narratives provide important information that can be used by a machine learning system for automated CoD classification. Unlike the structured questionnaire-based methods, this method can be applied to any verbal autopsy dataset, regardless of the collection process or country of origin.

Assessing the quality of medical death certification: a case study of concordance between national statistics and results from a medical record review in a regional hospital in the Philippines

BackgroundMedical certificates of cause of death (MCCOD) issued by hospital physicians are a key input to vital registration systems. Deaths certified by hospital physicians have been implicitly considered to be of high quality, but recent evidence suggests otherwise. We conducted a medical record review (MRR) of hospital MCCOD in the Philippines and compared the cause of death concordance with certificates coded by the Philippines Statistics Authority (PSA).MethodsMCCOD for adult deaths in Bohol Regional Hospital (BRH) in 2007–2008 and 2011 were collected and reviewed by a team of study physicians. Corresponding MCCOD coded by the PSA were linked by a hospital identifier. The study physicians wrote a new MCCOD using the patient medical record, noted the quality of the medical record to produce a cause of death, and indicated whether it was necessary to change the underlying cause of death (UCOD). Chance-corrected concordance, cause-specific mortality fraction (CSMF) accuracy, and chance-corrected CSMF were used to examine the concordance between the MRR and PSA.ResultsA total of 1052 adult deaths were linked between the MRR and PSA. Median chance-corrected concordance was 0.73, CSMF accuracy was 0.85, and chance-corrected CSMF accuracy was 0.58. 74.8% of medical records were deemed to be of high enough quality to assign a cause of death, yet study physicians indicated that it was necessary to change the UCOD in 41% of deaths, 82% of which required addition of a new UCOD.ConclusionsMedical records were generally of sufficient quality to assign a cause of death and concordance between the PSA and MRR was reasonably high, suggesting that routine mortality statistics data are reasonably accurate for describing population level causes of death in Bohol. While overall agreement between the PSA and MRR in major cause groups was sufficient for public health purposes, improvements in death certification practices are recommended to help physicians differentiate between treatable (immediate) COD and COD that are important for public health surveillance.

The Limits of the Verbal Autopsy for Cause of Death Determination

Background: Accurate information on causes of death (CoD) is essential to reduce mortality. In low-income settings, data on CoD is obtained using imprecise methods and knowledge of what people die of is based on assumptions, preventing to achieve full effectiveness of health interventions, monitor their impact, and thus targeting effective policies to reduce mortality. Methods: We have validated a computer-coded verbal autopsy method -the InterVA-4- using individual and population metrics to determine CoD against the gold standard, the complete diagnostic autopsy (CDA), in 316 deceased patients of different age groups who died in a tertiary-level hospital in Maputo, Mozambique between 2013 and 2015. Findings: We found a low agreement of the InterVA-4 model across all age groups at the individual (Kappa statistics ranging from -0·030 to 0·232, lowest in stillbirths and highest in adults) and population levels (chance-corrected cause-specific mortality fraction accuracy ranging from -1·00 to 0·62, lowest in stillbirths, highest in children). The sensitivity of the InterVA-4 model in identifying infectious diseases was low for all infectious diseases (0% for tuberculosis, diarrhea, and disseminated infections, 32% for HIV-related infections, 33% for malaria and 36% for pneumonia). Twenty-six maternal deaths were assigned to eclampsia but only four patients were actually eclampsia. Interpretation: The agreement of the InterVA-4 model against the CDA was low at the individual and at the population level. These findings are key to revise current tools used to establish CoD in high-burden mortality areas, and will help in improving VA techniques and developing more precise methods. Funding Statement: This work has been funded through grants from the Bill and Melinda Gates Foundation and the Spanish Ministry of Health. Declaration of Interests: The authors do not have any conflict of interest. Ethics Approval Statement: This study received approval by the Clinical Research Ethics Committee of the Hospital Clinic of Barcelona, Spain (File 2013/8677) and the National Bioethics Committee of Mozambique (Ref. 342/CNBS/13).

Measuring mortality due to HIV-associated tuberculosis among adults in South Africa: Comparing verbal autopsy, minimally-invasive autopsy, and research data.

BackgroundThe World Health Organization (WHO) aims to reduce tuberculosis (TB) deaths by 95% by 2035; tracking progress requires accurate measurement of TB mortality. International Classification of Diseases (ICD) codes do not differentiate between HIV-associated TB and HIV more generally. Verbal autopsy (VA) is used to estimate cause of death (CoD) patterns but has mostly been validated against a suboptimal gold standard for HIV and TB. This study, conducted among HIV-positive adults, aimed to estimate the accuracy of VA in ascertaining TB and HIV CoD when compared to a reference standard derived from a variety of clinical sources including, in some, minimally-invasive autopsy (MIA).Methods and findingsDecedents were enrolled into a trial of empirical TB treatment or a cohort exploring diagnostic algorithms for TB in South Africa. The WHO 2012 instrument was used; VA CoD were assigned using physician-certified VA (PCVA), InterVA-4, and SmartVA-Analyze. Reference CoD were assigned using MIA, research, and health facility data, as available. 259 VAs were completed: 147 (57%) decedents were female; median age was 39 (interquartile range [IQR] 33–47) years and CD4 count 51 (IQR 22–102) cells/μL. Compared to reference CoD that included MIA (n = 34), VA underestimated mortality due to HIV/AIDS (94% reference, 74% PCVA, 47% InterVA-4, and 41% SmartVA-Analyze; chance-corrected concordance [CCC] 0.71, 0.42, and 0.31, respectively) and HIV-associated TB (41% reference, 32% PCVA; CCC 0.23). For individual decedents, all VA methods agreed poorly with reference CoD that did not include MIA (n = 259; overall CCC 0.14, 0.06, and 0.15 for PCVA, InterVA-4, and SmartVA-Analyze); agreement was better at population level (cause-specific mortality fraction accuracy 0.78, 0.61, and 0.57, for the three methods, respectively).ConclusionsCurrent VA methods underestimate mortality due to HIV-associated TB. ICD and VA methods need modifications that allow for more specific evaluation of HIV-related deaths and direct estimation of mortality due to HIV-associated TB.

Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death.

BackgroundPhysician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA–4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method is that it requires a training data set from a prior validation study, while InterVA relies on clinically specified conditional probabilities. We undertook to validate the hierarchical expert algorithm analysis of VA data, an automated, intuitive, deterministic method that does not require a training data set.MethodsUsing Population Health Metrics Research Consortium study hospital source data, we compared the primary causes of 1629 neonatal and 1456 1–59 month–old child deaths from VA expert algorithms arranged in a hierarchy to their reference standard causes. The expert algorithms were held constant, while five prior and one new “compromise” neonatal hierarchy, and three former child hierarchies were tested. For each comparison, the reference standard data were resampled 1000 times within the range of cause–specific mortality fractions (CSMF) for one of three approximated community scenarios in the 2013 WHO global causes of death, plus one random mortality cause proportions scenario. We utilized CSMF accuracy to assess overall population–level validity, and the absolute difference between VA and reference standard CSMFs to examine particular causes. Chance–corrected concordance (CCC) and Cohen’s kappa were used to evaluate individual–level cause assignment.ResultsOverall CSMF accuracy for the best–performing expert algorithm hierarchy was 0.80 (range 0.57–0.96) for neonatal deaths and 0.76 (0.50–0.97) for child deaths. Performance for particular causes of death varied, with fairly flat estimated CSMF over a range of reference values for several causes. Performance at the individual diagnosis level was also less favorable than that for overall CSMF (neonatal: best CCC = 0.23, range 0.16–0.33; best kappa = 0.29, 0.23–0.35; child: best CCC = 0.40, 0.19–0.45; best kappa = 0.29, 0.07–0.35).ConclusionsExpert algorithms in a hierarchy offer an accessible, automated method for assigning VA causes of death. Overall population–level accuracy is similar to that of more complex machine learning methods, but without need for a training data set from a prior validation study.

A shortened verbal autopsy instrument for use in routine mortality surveillance systems.

BackgroundVerbal autopsy (VA) is recognized as the only feasible alternative to comprehensive medical certification of deaths in settings with no or unreliable vital registration systems. However, a barrier to its use by national registration systems has been the amount of time and cost needed for data collection. Therefore, a short VA instrument (VAI) is needed. In this paper we describe a shortened version of the VAI developed for the Population Health Metrics Research Consortium (PHMRC) Gold Standard Verbal Autopsy Validation Study using a systematic approach.MethodsWe used data from the PHMRC validation study. Using the Tariff 2.0 method, we first established a rank order of individual questions in the PHMRC VAI according to their importance in predicting causes of death. Second, we reduced the size of the instrument by dropping questions in reverse order of their importance. We assessed the predictive performance of the instrument as questions were removed at the individual level by calculating chance-corrected concordance and at the population level with cause-specific mortality fraction (CSMF) accuracy. Finally, the optimum size of the shortened instrument was determined using a first derivative analysis of the decline in performance as the size of the VA instrument decreased for adults, children, and neonates.ResultsThe full PHMRC VAI had 183, 127, and 149 questions for adult, child, and neonatal deaths, respectively. The shortened instrument developed had 109, 69, and 67 questions, respectively, representing a decrease in the total number of questions of 40-55 %. The shortened instrument, with text, showed non-significant declines in CSMF accuracy from the full instrument with text of 0.4 %, 0.0 %, and 0.6 % for the adult, child, and neonatal modules, respectively.ConclusionsWe developed a shortened VAI using a systematic approach, and assessed its performance when administered using hand-held electronic tablets and analyzed using Tariff 2.0. The length of a VA questionnaire was shortened by almost 50 % without a significant drop in performance. The shortened VAI developed reduces the burden of time and resources required for data collection and analysis of cause of death data in civil registration systems.Electronic supplementary materialThe online version of this article (doi:10.1186/s12916-015-0528-8) contains supplementary material, which is available to authorized users.

Improving performance of the Tariff Method for assigning causes of death to verbal autopsies.

BackgroundReliable data on the distribution of causes of death (COD) in a population are fundamental to good public health practice. In the absence of comprehensive medical certification of deaths, the only feasible way to collect essential mortality data is verbal autopsy (VA). The Tariff Method was developed by the Population Health Metrics Research Consortium (PHMRC) to ascertain COD from VA information. Given its potential for improving information about COD, there is interest in refining the method. We describe the further development of the Tariff Method.MethodsThis study uses data from the PHMRC and the National Health and Medical Research Council (NHMRC) of Australia studies. Gold standard clinical diagnostic criteria for hospital deaths were specified for a target cause list. VAs were collected from families using the PHMRC verbal autopsy instrument including health care experience (HCE). The original Tariff Method (Tariff 1.0) was trained using the validated PHMRC database for which VAs had been collected for deaths with hospital records fulfilling the gold standard criteria (validated VAs). In this study, the performance of Tariff 1.0 was tested using VAs from household surveys (community VAs) collected for the PHMRC and NHMRC studies. We then corrected the model to account for the previous observed biases of the model, and Tariff 2.0 was developed. The performance of Tariff 2.0 was measured at individual and population levels using the validated PHMRC database.ResultsFor median chance-corrected concordance (CCC) and mean cause-specific mortality fraction (CSMF) accuracy, and for each of three modules with and without HCE, Tariff 2.0 performs significantly better than the Tariff 1.0, especially in children and neonates. Improvement in CSMF accuracy with HCE was 2.5 %, 7.4 %, and 14.9 % for adults, children, and neonates, respectively, and for median CCC with HCE it was 6.0 %, 13.5 %, and 21.2 %, respectively. Similar levels of improvement are seen in analyses without HCE.ConclusionsTariff 2.0 addresses the main shortcomings of the application of the Tariff Method to analyze data from VAs in community settings. It provides an estimation of COD from VAs with better performance at the individual and population level than the previous version of this method, and it is publicly available for use.Electronic supplementary materialThe online version of this article (doi:10.1186/s12916-015-0527-9) contains supplementary material, which is available to authorized users.

Measuring causes of death in populations: a new metric that corrects cause-specific mortality fractions for chance.

BackgroundVerbal autopsy is gaining increasing acceptance as a method for determining the underlying cause of death when the cause of death given on death certificates is unavailable or unreliable, and there are now a number of alternative approaches for mapping from verbal autopsy interviews to the underlying cause of death. For public health applications, the population-level aggregates of the underlying causes are of primary interest, expressed as the cause-specific mortality fractions (CSMFs) for a mutually exclusive, collectively exhaustive cause list. Until now, CSMF Accuracy is the primary metric that has been used for measuring the quality of CSMF estimation methods. Although it allows for relative comparisons of alternative methods, CSMF Accuracy provides misleading numbers in absolute terms, because even random allocation of underlying causes yields relatively high CSMF accuracy. Therefore, the objective of this study was to develop and test a measure of CSMF that corrects this problem.MethodsWe developed a baseline approach of random allocation and measured its performance analytically and through Monte Carlo simulation. We used this to develop a new metric of population-level estimation accuracy, the Chance Corrected CSMF Accuracy (CCCSMF Accuracy), which has value near zero for random guessing, and negative quality values for estimation methods that are worse than random at the population level.ResultsThe CCCSMF Accuracy formula was found to be CCSMF Accuracy = (CSMF Accuracy - 0.632) / (1 - 0.632), which indicates that, at the population-level, some existing and commonly used VA methods perform worse than random guessing.ConclusionsCCCSMF Accuracy should be used instead of CSMF Accuracy when assessing VA estimation methods because it provides a more easily interpreted measure of the quality of population-level estimates.Electronic supplementary materialThe online version of this article (doi:10.1186/s12963-015-0061-1) contains supplementary material, which is available to authorized users.

Correction: Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies.

Verbal autopsy (VA) is an important method for obtaining cause of death information in settings without vital registration and medical certification of causes of death. An array of methods, including physician review and computer-automated methods, have been proposed and used. Choosing the best method for VA requires the appropriate metrics for assessing performance. Currently used metrics such as sensitivity, specificity, and cause-specific mortality fraction (CSMF) errors do not provide a robust basis for comparison. We use simple simulations of populations with three causes of death to demonstrate that most metrics used in VA validation studies are extremely sensitive to the CSMF composition of the test dataset. Simulations also demonstrate that an inferior method can appear to have better performance than an alternative due strictly to the CSMF composition of the test set. VA methods need to be evaluated across a set of test datasets with widely varying CSMF compositions. We propose two metrics for assessing the performance of a proposed VA method. For assessing how well a method does at individual cause of death assignment, we recommend the average chance-corrected concordance across causes. This metric is insensitive to the CSMF composition of the test sets and corrects for the degree to which a method will get the cause correct due strictly to chance. For the evaluation of CSMF estimation, we propose CSMF accuracy. CSMF accuracy is defined as one minus the sum of all absolute CSMF errors across causes divided by the maximum total error. It is scaled from zero to one and can generalize a method's CSMF estimation capability regardless of the number of causes. Performance of a VA method for CSMF estimation by cause can be assessed by examining the relationship across test datasets between the estimated CSMF and the true CSMF. With an increasing range of VA methods available, it will be critical to objectively assess their performance in assigning cause of death. Chance-corrected concordance and CSMF accuracy assessed across a large number of test datasets with widely varying CSMF composition provide a robust strategy for this assessment.