Healthcare Performance Evaluation Research Articles

Enhancing Disease Detection in Electronic Medical Records: Integrating Human Expertise and Large Language Models with Application to Diabetes, Hypertension, and Acute Myocardial Infarction

ObjectiveElectronic medical records (EMR) are widely available to complement administrative data-based disease surveillance and healthcare performance evaluation. Defining conditions from EMR is labour-intensive, requiring advanced medical informatics knowledge, and is challenging without effective data extraction tools. This study developed a high-throughput pipeline to detect diseases in EMRs. MethodsWe developed a pipeline that leverages a generative large language model (LLM) to analyze, understand, and interpret EMR notes by following clinical experts’ designed prompts. The pipeline was applied to detect diabetes, hypertension, and acute myocardial infarction (AMI) from the EMRs for a cardiac patient cohort in Calgary, Canada. The performance was compared against clinician-validated diagnoses as the reference standard. ResultsThe cohort consisted of 3,413 patients with 551,095 clinical notes. The prevalence was 27.8%, 66.3%, and 54.3% for diabetes, hypertension, and AMI, respectively. The performance for detecting conditions varied: diabetes had 90.5% sensitivity, 83% specificity, and 67% positive predictive value (PPV); hypertension had 94.2% sensitivity, 30.2% specificity, and 73.8% PPV; and AMI had 86.4% sensitivity, 61% specificity, and 75.3% PPV. The monthly prevalence trends between the detected cases and reference standard showed similar patterns. ConclusionThe proposed pipeline demonstrated reasonable accuracy and high efficiency in disease detection without manually curated labels, indicating the potential for automated real-time disease surveillance using EMRs. ImplicationVariations of documentation practices in clinical note can impact the detection performance of different diseases. Hence, an automated pipeline integrating LLMs with expert knowledge may improve detection accuracy with reduced labour costs while indicating documentation quality.

Measuring performance in allied health professional role substitution models of care: a clinician survey.

Professional role substitution models of care have emerged as a key strategy to address increasing healthcare demand. Gaining insights from those actively engaged in the process of these models' implementation and evaluation is pivotal to ensuring sustainability and further successful implementation. The purpose of this study was to describe allied-health clinicians' perceptions, practice, and experiences of healthcare performance evaluation in professional role substitution models of care. Data were collected via an online platform between 22 June - 22 July 2022 using a combination of convenience and network-based sampling of allied-health clinicians involved or interested in the implementation and evaluation of professional role substitution models of care. Clinicians answered 25 questions which consisted of demographic and targeted questions regarding performance evaluation across six domains of healthcare quality (effectiveness, safety, appropriateness, access & equity, continuity of care, and cost, efficiency, productivity & sustainability). A total of 102 clinicians accessed the survey, with 72 providing complete survey data. Eleven allied-health professions were represented, working across twelve specialities in thirteen hospital and health services. Whilst most allied-health clinicians (93-100%) supported measuring performance in each of the six healthcare quality domains, only 26-58% were measuring these domains in practice. Allied-health leadership support (62.5%), clinician drive (62.5%), consumer engagement (50%) and medical support (46%) were enablers whilst a lack of resources (human, time, financial (47%)), healthcare performance frameworks and/or policies (40%) were identified as barriers. Given the opportunity, clinicians would invest the most financial resources in digital solutions as a core strategy to improve performance evaluation. Allied-health professionals expressed strong support for principles of performance evaluation, however in practice, performance evaluation is still in its infancy in professional role substitution models of care. Organisations can implement strategies that maximise the enablers whilst addressing barriers identified to improve performance evaluation in these models of care.

Comparative Analysis of Additive and Multiplicative BoD Models in Healthcare Performance Evaluation

Comparative Analysis of Additive and Multiplicative BoD Models in Healthcare Performance Evaluation

Evaluating Healthcare Performance in Low- and Middle-Income Countries: A Pilot Study on Selected Settings in Ethiopia, Tanzania, and Uganda.

The literature reports some experiences regarding the design of integrated healthcare Performance Evaluation Systems (PES) applied in Low- and Middle-income Countries (LMIC). This study describes the design of an integrated and bottom-up PES aimed at evaluating healthcare services delivery in rural settings. The analysis involved four hospitals and their relative health districts in Ethiopia, Tanzania, and Uganda. The evaluation process was undertaken for those indicators that could be evaluated using the same reference standard. The evaluation scores were determined through the international standards identified in the literature or through benchmarking assessment. Both administrative and health data were extracted from the hospitals' registers and District Health Information Systems (DHIS) from 2017 to 2020. We defined 128 indicators: 88 were calculated at the hospital level and 40 at the health district level. The evaluation process was undertaken for 48 indicators. The evaluated indicators are represented using effective graphical tools. In settings characterised by multiple healthcare providers, this framework may contribute to achieving good governance through performance evaluation, benchmarking, and accountability. It may promote evidence-based decision-making in the planning and allocation of resources, thus ultimately fostering quality improvement processes and practices, both at the hospital and health district level.

1450Exeperiential Note on Enhancing Epidemiological Studies through Supervision of Postgraduate Research on Scaffolded Thesis-by-Publication

Abstract Focus of Presentation Most researchers are based in universities, and their research performance site is often metropolitan and urban cities. The effect is that scope of epidemiological studies has largely been without keen interest at the primary healthcare level. A pathway that this could have been addressed is teaching-research nexus in students’ research supervision. Hence in 2013, a research program was started and focused in primary health promotion. The program involved novel application of a known idea, whereby students do scaffold thesis-by-publication. Over the years, observations of students’ behaviour regarding the program are being made. This report narrates 7 years’ experience from Nigeria. Findings In the 7 years, 44 research students (3 PhDs, 20 Masters and 21 Hons) have passed through the program. Among the 21 Hons, 1 has volunteered to remain a collaborator and ‘unpaid’ Research Assistant in the program. Currently, 11 PhD candidates including 8/20 (40%) Masters’ degree graduates who are scaling up their work. In the year 2020, 4 of the PhD candidates have gone on to epidemiological study as well as primary healthcare performance evaluation of their local government areas in terms of public health problem of research interest. All students expressed being motivated by the publications. Conclusions/Implications Taking research students through thesis-by-publication journey is one approach to advance epidemiological studies. Making the students do the work on their own community advances health promotion, outreach to rural areas and primary health perspectives. Key messages Teaching-research nexus buoyed with thesis-by-publication can advance teaching and communication of epidemiology.

An integrated quality and resilience engineering framework in healthcare with Z-number data envelopment analysis.

Supplier selection for medical equipment is a major challenge for hospitals in healthcare supply chains. The primary reason for measuring medical equipment supplier efficiency is to achieve the highest level of overall performance and productivity in healthcare supply chains. This study presents an integrated quality and resilience engineering (QRE) framework for evaluating medical equipment suppliers' performance using structural equation modeling and Z-number data envelopment analysis (Z-DEA). Noise analysis is used to select the best α-cut for the Z-DEA model, and fuzzy data are used to handle uncertainties. We show that flexibility, conformance to standards, redundancy, cost, quality certifications, and delivery time significantly affect the medical equipment suppliers' performance. In addition, we demonstrate that the proposed integrated QRE framework is more efficient and informative than stand-alone quality engineering or resiliency engineering. We present a case study in a cardiovascular hospital to illustrate the applicability of the proposed framework for medical equipment supplier evaluation and selection. To the best of our knowledge, this is the first study to integrate QRE and Z-DEA for supplier performance evaluation in healthcare.

Co-Production Performance Evaluation in Healthcare. A Systematic Review of Methods, Tools and Metrics.

Co-produced practices and publications in the healthcare sector are gaining momentum, since they can be a useful tool in addressing the sustainability and resilience challenges of health systems. However, the investigation of positive and, mainly, negative outcomes is still confused and fragmented, and above all, a comprehensive knowledge of the metrics used to assess these outcomes is lacking. To fill this gap, this study aims to systematically review the extant literature to map the methods, tools and metrics used to empirically evaluate co-production in health services. The search took place in six databases: Scopus, Web of Science, Psych INFO, PubMed, Cochrane and CINAHL. A total of 2311 articles were screened and 203 articles were included in the analysis, according to PRISMA guidelines. Findings show that outcomes are mainly investigated through qualitative methods and from the lay actor or provider perspective. Moreover, the detailed categorisation of the quantitative measures found offers a multidimensional performance measurement system and highlights the impact areas where research is needed to develop and test new measures. Findings should also promote improvements in empirical data collection on the multiple faceted co-produced activities and spur the consciousness of the adoption of sustainable co-productive initiatives.

Performance measurement indicators in the healthcare industry: a systematic review

This systematic review addresses a gap in the literature on performance appraisal systems for healthcare workers. The study explores the efficiency of employee performance measurement systems in providing accurate measurement of the performance of healthcare workers across various job types with a particular focus on performance indicators associated with soft skills. Cochrane guidelines for performing a systematic search in disciplinespecific and in multidisciplinary databases were supplemented with an independent manual search of relevant studies. Twenty-three articles were identified and classified into three categories: performance measurement systems, evaluation and development of measuring tools, and problems of measuring performance of medical practitioners. The review revealed that there is no comprehensive approach to performance evaluation in healthcare. The identification of soft skills as a performance indicator was lacking and none of the implemented performance measurement systems were reported to be effective. Implications for further research to focus on the development of performance measurements for all workers in the healthcare industry were highlighted.

The online communication of healthcare organisations in the 'post-truth' era: An analysis of 167 websites in Italy

Introduction: The advent of the Internet and the availability of health information online has posed new challenges and opportunities to the healthcare sector. The amount of people who surf the Internet for health-related purposes is dramatically increasing. Actually, the ‘post-truth’ phenomenon had interested also the healthcare field, where the quality of the online information is really variable. In addition, the health literacy of people and their consequent capacity to correctly understand and therefore use health-related information is crucial. The quality and usability of information provided by the healthcare organizations online appear very important, in order to avoid the risks of fake news as well as of a misunderstanding of information. The websites can be used as a powerful channel for improving the access to care, the use of healthcare services, the health literacy and the empowerment of people. Methods: By analysing the websites of 167 healthcare organizations from 13 Italian Regions, we verified whatever online communication of Italian public hospitals and local healthcare authorities was readable and comprehensible for people with different level of education and different level of literacy, by using the Gulpease Index and by measuring the adoption of words from the Italian Fundamental Vocabulary the data refers to December 2015. Results: The results showed that, in Italy, the healthcare organisations’ online communication is still not structured by taking into consideration literacy and health literacy of people who may surf the Internet. In fact, the websites were difficult to read and understand for people with a low educational level according to both the indexes used. The healthcare organisation’s websites appeared built on the healthcare organisations’ needs, rather than on those of the potential readers. The healthcare management of the Regions recognized the importance of these findings. For this reason, the Collaborative of the Italian Regions, coordinated by the MeS Laboratory of Sant’Anna School, decided to monitor and evaluate these aspects by introducing specific indicators in the healthcare Performance Evaluation System they voluntarily adopted. Implications: In the ‘post-truth’ era, it is important to consider the risks of more readable and comprehensible information available on other than the healthcare organisations’ websites. This suggests an urgent consideration of the health literacy of people, in addition to their information needs. The availability of our findings was fundamental to inform managers of Italian regional healthcare systems and support them in addressing this issue and working for an equal access to good quality but also comprehensible and readable information online.

The Application of Data Envelopment Analysis in Healthcare Performance Evaluation of Rehabilitation Departments in Hungary

Purpose: Performance evaluation is a general problem both in production and service systems. Generally, operation performance is determined based on input resource utilization and on outputs related data. Performance evaluation is especially complicated when both financial and nonfinancial indicators must be considered in the evaluation of the efficiency of healthcare system. The purpose of this paper is to apply data envelopment analysis (DEA) in order to measure the efficiency of rehabilitation departments curing musculoskeletal diseases.Methodology/Approach: The evaluation of the efficiency of rehabilitation departments includes several parameters. Performance evaluation becomes complicated when several evaluation criteria must be taken into consideration at the same time. In these cases, scoring methods are generally used, which transform performance data into a common scale and an aggregate score is calculated with subjective weights. Using DEA the subjective element of evaluation is eliminated when the weights of inputs and outputs are determined.Findings: The applied DEA model evaluates the performance of rehabilitation departments. The presented analysis highlights the differences between the efficiency of the studied departments, and explores inefficiencies related to economies of scale. The slack values directly show the operational shortcomings in specific areas, and indicate the exact amount of the required changes.Research Limitation/implication: The applied DEA model evaluates the performance of rehabilitation departments. The presented analysis highlights the differences between the efficiency of the studied departments, and explores inefficiencies related to economies of scale. The slack values directly show the operational shortcomings in specific areas, and indicate the exact amount of the required changes.Originality/Value of paper: The originality of the paper lies on the identification of inputs and outputs for the applied DEA model as only nonfinancial indicators were taken into consideration. The analysis involves all rehabilitation departments of the Hungarian healthcare system; consequently, conclusions related to the general state of this area can be drawn.

Rethinking Healthcare Performance Evaluation Systems towards the People-Centredness Approach: Their Pathways, their Experience, their Evaluation.

Patient experience should be the starting point to achieve a high quality of care. Coherently, healthcare performance evaluation systems, driving the change in line with the main strategic goals, should be designed considering the patient perspective. Instead, they are traditionally defined according to the healthcare service provider's point of view. Consequently, they reproduce a "silo-vision" characterized by a clear separation of responsibilities limited to a specific setting of care or to a single organization. This commentary discusses the importance of using patient-reported measures together with indicators based on administrative data to evaluate cross-setting healthcare services within a multidimensional healthcare performance evaluation system. The experience of the Tuscany regional healthcare Performance Measurement System (PMS), implemented more than 10 years ago and in continuous evolution, represents an innovative example of how to measure the quality of the whole care pathway including patient experience. This new approach is based on a systematic, systemic and standardized collection of patient-reported experience measures in several healthcare pathways and evaluating them using a coherent graphical representation. Targets, incentives and other managerial tools are fixed, overcoming organizational boundaries and integrating the patient point of view with the goal of moving the healthcare system towards a patient-centredness approach to care.

Imprecise DEA framework for evaluating health-care performance of districts

PurposeThis paper aims to address performance measurement in the health-care sector, which gains increasing importance for most countries because growing health expenditures and increased quality and competition in the health sector require hospitals to use their resources efficiently. Health policy-makers and health-care managers stress the need for developing a robust performance evaluation methodology for health-care organizations.Design/methodology/approachThis paper presents an imprecise data envelopment analysis (DEA) framework for evaluating the health-care performance of 26 districts in Istanbul, a metropolis with nearly 15 million inhabitants. The proposed methodology takes into account both quantitative and qualitative data represented as linguistic variables for performance evaluation. Moreover, this study reckons that weight flexibility in DEA assessments can lead to unrealistic weighting schemes for some inputs and outputs, which are likely to result in overstated efficiency scores for a number of decision-making units (in here, districts). To overcome this problem, a weight restricted imprecise DEA model that constrains weight flexibility in DEA is proposed.FindingsThe proposed imprecise DEA approach sets forth a more realistic decision methodology for evaluating the relative health-care performance and also enables to determine the best district in terms of health-care performance in Istanbul.Originality/valueThis paper includes the quality dimension, which has been overlooked in previous studies, into the health-care performance evaluation of districts. Moreover, it circumvents unrealistic weight flexibility which may distort the relative evaluation of health-care performance.

Linking Swedish health data registers to establish a research database and a shared decision-making tool in hip replacement

BackgroundSweden offers a unique opportunity to researchers to construct comprehensive databases that encompass a wide variety of healthcare related data. Statistics Sweden and the National Board of Health and Welfare collect individual level data for all Swedish residents that ranges from medical diagnoses to socioeconomic information. In addition to the information collected by governmental agencies the medical profession has initiated nationwide Quality Registers that collect data on specific diagnoses and interventions. The Quality Registers analyze activity within healthcare institutions, with the aims of improving clinical care and fostering clinical research.Main bodyThe Swedish Hip Arthroplasty Register (SHAR) has been collecting data since 1979. Joint replacement in general and hip replacement in particular is considered a success story with low mortality and complication rate. It is credited to the pioneering work of the SHAR that the revision rate following hip replacement surgery in Sweden is amongst the lowest in the world. This has been accomplished by the diligent follow-up of patients with feedback of outcomes to the providers of the healthcare along with post market surveillance of individual implant performance. During its existence SHAR has experienced a constant organic growth. One major development was the introduction of the Patient Reported Outcome Measures program, giving a voice to the patients in healthcare performance evaluation. The next aim for SHAR is to integrate patients’ wishes and expectations with the surgeons’ expertise in the form of a Shared Decision-Making (SDM) instrument. The first step in building such an instrument is to assemble the necessary data. This involves linking the SHARs database with the two aforementioned governmental agencies. The linkage is done by the 10-digit personal identity number assigned at birth (or immigration) for every Swedish resident. The anonymized data is stored on encrypted serves and can only be accessed after double identification.ConclusionThis data will serve as starting point for several research projects and clinical improvement work.

Assisting allied health in performance evaluation: a systematic review.

BackgroundPerformance evaluation raises several challenges to allied health practitioners and there is no agreed approach to measuring or monitoring allied health service performance. The aim of this review was to examine the literature on performance evaluation in healthcare to assist in the establishment of a framework that can guide the measurement and evaluation of allied health clinical service performance. This review determined the core elements of a performance evaluation system, tools for evaluating performance, and barriers to the implementation of performance evaluation.MethodsA systematic review of the literature was undertaken. Five electronic databases were used to search for relevant articles: MEDLINE, Embase, CINAHL, PsychInfo, and Academic Search Premier. Articles which focussed on any allied health performance evaluation or those which examined performance in health care in general were considered in the review. Content analysis was used to synthesise the findings from individual articles.ResultsA total of 37 articles were included in the review. The literature suggests there are core elements involved in performance evaluation which include prioritising clinical areas for measurement, setting goals, selecting performance measures, identifying sources of feedback, undertaking performance measurement, and reporting the results to relevant stakeholders. The literature describes performance evaluation as multi-dimensional, requiring information or data from more than one perspective to provide a rich assessment of performance. A range of tools or instruments are available to capture various perspectives and gather a comprehensive picture of health care quality.ConclusionsEvery allied health care delivery system has different performance needs and will therefore require different approaches. However, there are core processes that can be used as a framework to evaluate allied health performance. A careful examination of barriers to performance evaluation and subsequent tailoring of strategies to overcome these barriers should be undertaken to achieve the aims of performance evaluation. The findings of this review should inform the development of a standardised framework that can be used to measure and evaluate allied health performance. Future research should explore the utility and overall impact of such framework in allied health service delivery.

Performance evaluation in healthcare: the experience of maternity pathway from Tuscany to the Italian network of regions

Background Considering that performance measurement is largely recognised as a key tool for quality improvement, which kind of performance evaluation can be useful for healthcare professional, general managers and regional policy makers to improve the mother and child healthcare services? The presentation of the experience in assessing the maternity pathway (MP), made initially by Tuscany and successively shared within a network of Italian regions, will contribute to answer this question.

Advancing the use of performance evaluation in health care.

The purpose of this paper is to develop a framework for health care performance evaluation that enables decision makers to identify areas indicative of corrective actions. The framework should provide information on strategic pro-/regress in an operational context that justifies the need for organizational adjustments. The study adopts qualitative methods for constructing the framework, subsequently implementing the framework in a Danish magnetic resonance imaging (MRI) unit. Workshops and interviews form the basis of the qualitative construction phase, and two internal and five external databases are used for a quantitative data collection. By aggregating performance outcomes, collective measures of performance are achieved. This enables easy and intuitive identification of areas not strategically aligned. In general, the framework has proven helpful in an MRI unit, where operational decision makers have been struggling with extensive amounts of performance information. The implementation of the framework in a single case in a public and highly political environment restricts the generalizing potential. The authors acknowledge that there may be more suitable approaches in organizations with different settings. The strength of the framework lies in the identification of performance problems prior to decision making. The quality of decisions is directly related to the individual decision maker. The only function of the framework is to support these decisions. The study demonstrates a more refined and transparent use of performance reporting by combining strategic weight assignment and performance aggregation in hierarchies. In this way, the framework accentuates performance as a function of strategic progress or regress, thus assisting decision makers in exerting operational effort in pursuit of strategic alignment.

Potential pros and cons of external healthcare performance evaluation systems: real-life perspectives on Iranian hospital evaluation and accreditation program.

Performance evaluation is essential to quality improvement in healthcare. The current study has identified the potential pros and cons of external healthcare evaluation programs, utilizing them subsequently to look into the merits of a similar case in a developing country. A mixed method study employing both qualitative and quantitative data collection and analysis techniques was adopted to achieve the study end. Subject Matter Experts (SMEs) and professionals were approached for two-stage process of data collection. Potential advantages included greater attractiveness of high accreditation rank healthcare organizations to their customers/purchasers and boosted morale of their personnel. Downsides, as such, comprised the programs' over-reliance on value judgment of surveyors, routinization and incurring undue cost on the organizations. In addition, the improved, standardized care processes as well as the judgmental nature of program survey were associated, as pros and cons, to the program investigated by the professionals. Besides rendering a tentative assessment of Iranian hospital evaluation program, the study provides those running external performance evaluations with a lens to scrutinize the virtues of their own evaluation systems through identifying the potential advantages and drawbacks of such programs. Moreover, the approach followed could be utilized for performance assessment of similar evaluation programs.

Building China's municipal healthcare performance evaluation system: a Tuscan perspective

Regional healthcare performance evaluation systems can help optimize healthcare resources on regional basis and improve the performance of healthcare services provided. The Tuscany region in Italy is a good example of an institution which meets these requirements. China has yet to build such a system based on international experience. In this paper, based on comparative studies between Tuscany and China, we propose that the managing institutions in China's experimental cities can select and commission a third-party agency to, respectively, evaluate the performance of their affiliated hospitals and community health service centers. Following some features of the Tuscan experience, the Chinese municipal healthcare performance evaluation system can be built by focusing on the selection of an appropriate performance evaluation agency, the design of an adequate performance evaluation mechanism and the formulation of a complete set of laws, rules and regulations. When a performance evaluation system at city level is formed, the provincial government can extend the successful experience to other cities.

Healthcare Assessment and Performance: Using Simulation

Simulation as an evaluative tool has been subject to problems validating it as a gold standard for performance evaluation in health care. The simulator can be effective only when coupled with an effective curriculum. Simulation in itself raises several important questions: (a) What aspects of professional competence can be better assessed by simulation technique than by more traditional evaluative measures? (b) What are the extra costs of simulations in terms of money, time, and other resources? (c) What are the long-and short-term ramifications of simulation exercises for examinees, simulators, and observers? (d) How can the ethical requirements of informed consent be met without jeopardizing the reality of the simulation? This article will address the ongoing dilemmas around performance assessment and will begin to frame the ongoing debate in the field of simulation and education research.

Book Reviews: Donald N. Lombardi: Handbook of Personnel Selection and Performance Evaluation in Healthcare. Guidelines for Hourly, Professional, and Managerial Employees

Book Reviews: Donald N. Lombardi: Handbook of Personnel Selection and Performance Evaluation in Healthcare. Guidelines for Hourly, Professional, and Managerial Employees