Systematic reviews (SRs) are essential for evidence-based practice but remain labor-intensive, especially during abstract screening. This study evaluates whether multiple large language models (multi-LLMs) collaboration can improve the efficiency and reduce costs for abstract screening. Abstract screening was framed as a question-answering (QA) task using cost-effective LLMs. Three multi-LLM collaboration strategies were evaluated, including majority voting by averaging opinions of peers, multi-agent debate for answer refinement, and LLM-based adjudication against answers of individual QA baselines. These strategies were evaluated on 28 SRs of the CLEF eHealth 2019 technology-assisted review benchmark using standard performance metrics such as mean average precision (MAP) and work saved over sampling at 95% recall (work saved over sampling WSS@95%). Multi-LLM collaboration significantly outperformed QA baselines. Majority voting was overall the best strategy, achieving the highest MAP 0.462 and 0.341 on subsets of SRs about clinical intervention and diagnostic technology assessment, respectively, with WSS@95% 0.606 and 0.680, enabling in theory up to 68% workload reduction at 95% recall of all relevant studies. Multi-agent debate improved weaker models most. Our own adjudicator-as-a-ranker method was the second strongest approach, surpassing adjudicator-as-a-judge, but at a significantly higher cost than majority voting and debating. Multi-LLM collaboration substantially improves abstract screening efficiency, and the success lies in model diversity. Making the best use of diversity, majority voting stands out in terms of both excellent performance and low cost compared to adjudication. Despite context-dependent gains and diminishing model diversity, multi-agent debate is still a cost-effective strategy and a potential direction of further research.

A Value Framework for the Assessment of Diagnostic Technologies: A Proposal Based on a Targeted Systematic Review and a Multistakeholder Deliberative Process in Latin America

Insights into UK NICE diagnostic-technology assessment decisions

PNS132 VALUE FRAMEWORK FOR ASSESSMENT OF DIAGNOSTIC TECHNOLOGIES: A LATIN AMERICAN CASE

Adoption of new technologies, including diagnostic tests, is often considered not to deliver the expected return on investment. The reasons for this poor link between expectation and outcome include lack of evidence, variation in use of the technology, and an inability of the health system to manage the balance between investment and disinvestment associated with the change in care pathway. The challenges lie in the complex nature of healthcare provision where the investment is likely to be made in the jurisdiction of one stakeholder while the benefits (as well as dis-benefits) accrue to the other stakeholders. A prime example is found in the field of laboratory medicine and the use of diagnostic tests. The current economic tools employed in healthcare are primarily used to make policy and strategic decisions, particularly across health systems, and in purchaser and provider domains. These tools primarily involve cost effectiveness and budget impact analyses, both of which have been applied in health technology assessment of diagnostic technologies. However, they lack the granularity to translate findings down to the financial management and operational decision making at the provider department level. We propose an approach to translational health economics based on information derived from service line management and time-driven activity-based costing, identifying the resource utilisation for each of the units involved in the delivery of a care pathway, before and after adoption of new technology. This will inform investment and disinvestment decisions, along with identifying where the benefits, and dis-benefits, can be achieved for all stakeholders.

Introducción. El tabaquismo es causa de enfermedad respiratoria y factor de riesgo de enfermedades cardiovasculares, cáncer y otras enfermedades. Las intervenciones desarrolladas están orientadas al conocimiento del tabaquismo en fumadores, orientando un diagnóstico basado en la utilización de diversas pruebas que se han convertido en instrumento de evaluación, lo cual ha generado la necesidad de conocer las propiedades psicométricas de las pruebas utilizadas.
 Objetivo. Validación del test de Fagerström para adicción a la nicotina.
 Materiales y métodos. Se llevó a cabo un estudio transversal y descriptivo de evaluación de tecnología diagnóstica, y muestreo no probabilístico por conveniencia. Se incluyó una población de 200 fumadores activos de la Universidad de Boyacá. Se evaluaron seis preguntas que conforman el test, mediante la prueba de esfericidad de Bartlett, Kaiser-Meyer-Olkin y alfa de Cronbach.
 Resultados. La prueba de esfericidad de Bartlett resultó estadísticamente significativa, lo cual sugiere la existencia de correlaciones dentro de la matriz de seis preguntas. La prueba de comunalidad es de utilidad en la determinación de los grupos o los factores que se describen en el test para establecer la adicción física a la nicotina. La variación total explicable de las seis preguntas que corresponden a la versión corta del test de Fagerström dio cuenta del 47,7 % de variabilidad.
 Conclusiones. Las propiedades psicométricas del test permiten su utilización en el diagnóstico de tabaquismo según la adicción a la nicotina del fumador, sin presentar diferencias en la comprensión por parte de los fumadores y teniendo resultado fiables.
 Palabras clave: validez de las pruebas, reproducibilidad de resultados, nicotina.

Setting: Three pediatric pneumatologist offices in Bucaramanga, Colombia. Objective: To establish the concordance between medical criteria and the Childhood-Asthma Control Test (cACT). Design: Study of the assessment of diagnostic technology using transverse sampling. 127 asthma patients aged between 4 and 11 years and their parents filled before clinical evaluation made by a pediatric pneumologist. Patients were classified as controlled or not controlled. Criteria validity was established comparing this classification using Cohen’s kappa and performance indicators according to ROC analysis. Results: 78% of the patients were controlled; patients who are not controlled have a higher score in cACT (mean difference: 3.25 points). Concordance among cACT subscales is acceptable (ρ = 0.554). cACT’s sensitivity was 53.6%, specificity 78.8%, positive likelihood ratio (LR+) 2.53, and negative likelihood ratio (LR-) 0.59. The best cut-off point is 15, with 98% sensitivity, a specificity of 14.3%, LR+ in 1.14, and LR- in 0.14. Conclusions: cACT is a valid tool to determine asthma control in children without replacing medical criteria or other clinical tests. In populations with difficult access to high complexity services, it is useful to decide whether urgent referral to the specialist is necessary.

The inclusion of patient-reported outcome measures (PROMs) in the routine clinical care of chronically ill patients has the potential to add valuable information about the impact of the disease and its treatment and promotes effective patient self-management in which patients become more active participants in their own care. PROMs provide clinicians with timely information on patients' symptoms as well as functional and emotional status. PROMs are a useful tool for enhancing patient-clinician communication. We develop a conceptual framework describing the potential effects of the use of PROMs in chronic care management. The framework summarizes insights from the methods for evaluating the clinical effectiveness and methods for the health technology assessment of diagnostic technologies and results from the relevant studies. The framework describes potential effects, from proximal to distal, including communication (patient-clinician, patient-relative, clinician-clinician, and clinician-relative), engaging patients in shared clinical decision making, patient management (clinician management and patient self-management), and patient outcomes. Important potential effects also include enhancement in patient activation as well as improvements in clinician and patient satisfaction, and patient adherence to recommended treatment. Previous frameworks have described patient-physician communication, patient satisfaction, and health outcomes. Our framework adds unique domains, including patient engagement, patient activation, shared clinical decision making, and patient self-management. The framework can be used as a tool to guide the development of interventions to improve chronic care management through the use of PROMs.

Determining the validity and reliability of a user satisfaction instrument applied to patients attending the emergency department and clinic who were suffering from acute febrile syndrome in Bucaramanga's Public Health Network. A diagnostic technology assessment study was conducted between 2008 and 2009 on patients suffering from acute febrile syndrome. A telephone questionnaire was designed using 3 instruments. Content validity was assessed by experts and health professionals and face validity was assessed in a pilot study involving 30 users. Reproducibility was tested in questionnaires independently applied by two nurses to 19 patients. Data from 146 surveys was collected; 75.3 % were answered by women, 33.6 % were patients and 52.7 % were their mothers when patients were under 15. Three factors were identified from factor analysis: satisfaction with medical care, nursing and medical centre organisation. The final model included 16 of the 26 items. Cronbach's Alpha for each factor was 0.89 (0.86-0.93 95 %CI), 0.78 (0.67- 0.84 95 %CI) and 0.88 (0.85-0.91 95 %CI), respectively. The Kappa coefficient was 0.90 (0.77-0.97 95 %CI) and intra-class correlation coefficient was 0.96 (0.86-0.99 95 %CI). The average difference of the sum of all items was -1.3 (SD: 8.5) and Bland and Altman limits of agreement were -18.9 and 15.3, respectively. This study established the validity and reliability of a new instrument concerning user satisfaction used with patients suffering from acute febrile syndrome attending emergency health care which that could be applied in the Colombian Public Health System.

How to Use an Article About a Diagnostic Test

A Simple Method for Selecting Urine Samples that Need Culturing

Measuring and incorporating patient preferences and utilities in the assessment of diagnostic technology

Measuring and incorporating patient preferences and utilities in the assessment of diagnostic technology: Discussion for session 7

Incorporating patient preferences and utilities in the assessment of diagnostic technology: Response

A Hierarchical Outcomes Approach to Test Assessment

We cannot choose to ignore outcomes in the clinical assessment of diagnostic technologies

Challenges and opportunities in economic evaluations of diagnostic tests and procedures

To determine the incremental cost-effectiveness of magnetic resonance imaging (MRI) and computed tomography (CT) in young adults presenting with equivocal neurological signs and symptoms. A decision analysis of long-term survival using accuracy data from a diagnostic technology assessment of MRI and CT in patients with suspected multiple sclerosis, information from the medical literature, and clinical assumptions. In the baseline analysis, at 30% likelihood of an underlying neurologic disease, MRI use has an incremental cost of $101,670 for each additional quality-adjusted life-year saved compared with $20,290 for CT use. As the probability of disease increases, further MRI use becomes a cost-effective alternative costing $30,000 for each quality-adjusted life-year saved. If a negative MRI result provides reassurance, the incremental costs of immediate MRI use decreases and falls below $25,000 for each quality-adjusted life-year saved no matter the likelihood of disease. For most individuals with neurological symptoms or signs, CT imaging is cost-effective while MR imaging is not. The cost-effectiveness of MRI use, however, improves as the likelihood of an underlying neurological disease increases. For selected patients who highly value diagnostic information, MRI is a reasonable and cost-effective use of medical resources when even the likelihood of disease is quite low (5%).

To present criteria to aid intensive care workers in the assessment of diagnostic technologies, using the example of bronchoalveolar lavage for the evaluation of ventilator-associated pneumonia. MEDLINE was used to search for articles published from 1969 to the present that concerned diagnostic tests, diagnostic technology, pneumonia, and critically ill patients. Clinical investigations, case control studies, case series, and experimental data on the use of bronchoalveolar lavage. Studies of diagnostic technology were also included. We extracted relevant data in duplicate, independently. Diagnostic technology assessment should begin by establishing the capability of the technology under ideal or laboratory conditions, followed by an exploration of the range of possible uses as well as the accuracy of the test. Bronchoalveolar lavage is a well-established technology for the diagnosis of pneumonia in immunocompromised patients. Studies of the accuracy of bronchoalveolar lavage in ventilator-dependent but nonimmunocompromised patients have shown promising diagnostic accuracy. Accuracy, however, is insufficient for dissemination of a test; an evaluation of the impact of a test on management decisions and, most importantly, on patient outcome, is required. Investigators have not addressed the full impact of bronchoalveolar lavage, and, even if the test is accurate, there are reasons to doubt whether patients will be better off if the test becomes part of routine clinical practice. We present guidelines for the assessment of diagnostic technology, and apply them to bronchoalveolar lavage for the evaluation of ventilator-associated pneumonia. Bronchoalveolar lavage has been studied in both the laboratory and clinical setting, and the diagnostic sensitivity and specificity of this technique are high. Further randomized trials evaluating management decisions and patient benefit would facilitate decisions regarding the appropriate dissemination of bronchoalveolar lavage.

Suspected pedal osteomyelitis in a diabetic patient poses a management dilemma. The available diagnostic tools vary in sensitivity, specificity, cost, and risk. Researchers from the Diagnostic Technology Assessment Consortium used decision analysis to compare the cost- effectiveness of four strategies in assessing and managing this problem. The strategies were: …