Most Orthopaedic Studies Using the National Inpatient Sample Fail to Adhere to Recommended Research Practices: A Systematic Review.

Abstract

The National Inpatient Sample (NIS) database is accessible, inexpensive, and increasingly used in orthopaedic research, but it has complex design features that require nuanced methodological considerations for appropriate use and interpretation. A recent study showed poor adherence to recommended research practices for the NIS across a broad spectrum of medical and surgical fields, but the degree and patterns of nonadherence among orthopaedic publications remain unclear. In this study, we sought: (1) to quantify nonadherence to recommended research practices provided by the Agency for Healthcare Research and Quality (AHRQ) for using the NIS data in orthopaedic studies from 2016-2017; and, (2) to identify the most common nonadherence practices. We evaluated all 136 manuscripts published across the 74 orthopaedic journals listed on Scimago Journal & Country Rank that used the NIS from January 2016 through December 2017. Of those studies, 2% (3 of 136) were excluded because NIS was not used for analysis. The studies were evaluated for adherence to seven recommended research practices by the AHRQ: (1) identifying observations as hospitalization events rather than unique patients; (2) not performing state-level analyses; (3) limiting hospital-level analyses to data from year 1988-2011; (4) not performing physician-level analyses; (5) avoiding the use of nonspecific secondary diagnosis codes to infer in-hospital events; (6) using survey-specific analysis methods that account for clustering, stratification, and weighting; and (7) accounting for data changes in trend analyses spanning major transition periods in the data set (1997-1998 and 2011-2012). Overall, 93% (124 of 133) of the studies did not adhere to one or more practices. For each of the research practices assessed, 80% (106 of 133) of the studies did not account for the clustering and stratification in survey design; 56% (75 of 133) implied records were unique patients rather than hospitalization events; 41% (54 of 133) inappropriately used secondary diagnosis codes to infer in-hospital events. Nearly all manuscripts published in orthopaedic journals using the NIS database in 2016 and 2017 failed to adhere to recommended research practices. Future research quantifying variations in study results on the basis of adherence to recommended research practices would be of value. With the ubiquitous presence of large-database studies in orthopaedic journals, our work points to the importance of rigorous methodological appraisal when evaluating results, and encourages journals to require the use of the methodology checklists upon submission of such studies. More research is needed to determine whether deviations from recommended research practices actually lead to biased conclusions that affect patient care and policy-related decisions.