Domain-specific Criteria Research Articles

Standardized assessment of evidence supporting the adoption of mobile health solutions: A Clinical Consensus Statement of the ESC Regulatory Affairs Committee

Abstract Mobile health (mHealth) solutions have the potential to improve self-management and clinical care. For successful integration into routine clinical practice, healthcare professionals (HCPs) need accepted criteria helping the mHealth solutions’ selection, while patients require transparency to trust their use. Information about their evidence, safety and security may be hard to obtain and consensus is lacking on the level of required evidence. The new Medical Device Regulation is more stringent than its predecessor, yet its scope does not span all intended uses and several difficulties remain. The European Society of Cardiology Regulatory Affairs Committee set up a Task Force to explore existing assessment frameworks and clinical and cost-effectiveness evidence. This knowledge was used to propose criteria with which HCPs could evaluate mHealth solutions spanning diagnostic support, therapeutics, remote follow-up and education, specifically for cardiac rhythm management, heart failure and preventive cardiology. While curated national libraries of health apps may be helpful, their requirements and rigour in initial and follow-up assessments may vary significantly. The recently developed CEN-ISO/TS 82304-2 health app quality assessment framework has the potential to address this issue and to become a widely used and efficient tool to help drive decision-making internationally. The Task Force would like to stress the importance of co-development of solutions with relevant stakeholders, and maintenance of health information in apps to ensure these remain evidence-based and consistent with best practice. Several general and domain-specific criteria are advised to assist HCPs in their assessment of clinical evidence to provide informed advice to patients about mHealth utilization.

Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study

Artificial intelligence (AI) has repeatedly been shown to encode historical inequities in healthcare. We aimed to develop a framework to quantitatively assess the performance equity of health AI technologies and to illustrate its utility via a case study. Here, we propose a methodology to assess whether health AI technologies prioritise performance for patient populations experiencing worse outcomes, that is complementary to existing fairness metrics. We developed the Health Equity Assessment of machine Learning performance (HEAL) framework designed to quantitatively assess the performance equity of health AI technologies via a four-step interdisciplinary process to understand and quantify domain-specific criteria, and the resulting HEAL metric. As an illustrative case study (analysis conducted between October 2022 and January 2023), we applied the HEAL framework to a dermatology AI model. A set of 5420 teledermatology cases (store-and-forward cases from patients of 20 years or older, submitted from primary care providers in the USA and skin cancer clinics in Australia), enriched for diversity in age, sex and race/ethnicity, was used to retrospectively evaluate the AI model's HEAL metric, defined as the likelihood that the AI model performs better for subpopulations with worse average health outcomes as compared to others. The likelihood that AI performance was anticorrelated to pre-existing health outcomes was estimated using bootstrap methods as the probability that the negated Spearman's rank correlation coefficient (i.e., "R") was greater than zero. Positive values of R suggest that subpopulations with poorer health outcomes have better AI model performance. Thus, the HEAL metric, defined as p (R>0), measures how likely the AI technology is to prioritise performance for subpopulations with worse average health outcomes as compared to others (presented as a percentage below). Health outcomes were quantified as disability-adjusted life years (DALYs) when grouping by sex and age, and years of life lost (YLLs) when grouping by race/ethnicity. AI performance was measured as top-3 agreement with the reference diagnosis from a panel of 3 dermatologists per case. Across all dermatologic conditions, the HEAL metric was 80.5% for prioritizing AI performance of racial/ethnic subpopulations based on YLLs, and 92.1% and 0.0% respectively for prioritizing AI performance of sex and age subpopulations based on DALYs. Certain dermatologic conditions were significantly associated with greater AI model performance compared to a reference category of less common conditions. For skin cancer conditions, the HEAL metric was 73.8% for prioritizing AI performance of age subpopulations based on DALYs. Analysis using the proposed HEAL framework showed that the dermatology AI model prioritised performance for race/ethnicity, sex (all conditions) and age (cancer conditions) subpopulations with respect to pre-existing health disparities. More work is needed to investigate ways of promoting equitable AI performance across age for non-cancer conditions and to better understand how AI models can contribute towards improving equity in health outcomes. Google LLC.

Mining Utilities Itemsets based on social network

Mining utility item sets based on social network data involves extracting meaningful patterns and associations from user interactions. In this paper, the process begins by collecting and preprocessing data from platforms like Facebook, Twitter, or LinkedIn. Utility measures are defined based on frequency of occurrence, user engagement metrics, or other domain-specific criteria. Itemsets that meet certain thresholds are identified using techniques like frequent itemset mining or advanced algorithms like Apriori or FP-growth. Additional analyses, such as association rule mining, uncover relationships between different itemsets or user segments, providing valuable insights for personalized recommendations, targeted advertising, and decision-making processes.

GUM: The generalized upper model

GUM is a linguistically-motivated ontology originally developed to support natural language processing systems by offering a level of representation intermediate between linguistic forms and domain knowledge. Whereas modeling decisions for individual domains may need to be responsive to domain-specific criteria, a linguistically-motivated ontology offers a characterization that generalizes across domains because its design criteria are derived independently both of domain and of application. With respect to this mediating role, the use of GUM resembles (and partially predates) the adoption of upper ontologies as tools for mediating across domains and for supporting domain modeling. This paper briefly introduces the ontology, setting out its origins, design principles and applications. The example cases for this special issue are then described, illustrating particularly some of the principal differences and similarities of GUM to non-linguistically motivated upper ontologies.

Knowledge-Based Stable Roommates Problem: A Real-World Application

AbstractThe Stable Roommates problem with Ties and Incomplete lists (SRTI) is a matching problem characterized by the preferences of agents over other agents as roommates, where the preferences may have ties or be incomplete. SRTI asks for a matching that is stable and, sometimes, optimizes a domain-independent fairness criterion (e.g. Egalitarian). However, in real-world applications (e.g. assigning students as roommates at a dormitory), we usually consider a variety of domain-specific criteria depending on preferences over the habits and desires of the agents. With this motivation, we introduce a knowledge-based method to SRTI considering domain-specific knowledge and investigate its real-world application for assigning students as roommates at a university dormitory.

Health timeline: an insight-based study of a timeline visualization of clinical data

BackgroundThe increasing complexity and volume of clinical data poses a challenge in the decision-making process. Data visualizations can assist in this process by speeding up the time required to analyze and understand clinical data. Even though empirical experiments show that visualizations facilitate clinical data understanding, a consistent method to assess their effectiveness is still missing.MethodsThe insight-based methodology determines the quality of insights a user acquires from the visualization. Insights receive a value from one to five points based on a domain-specific criteria. Five professional psychiatrists took part in the study using real de-identified clinical data spanning 4 years of medical history.ResultsA total of 50 assessments were transcribed and analyzed. Comparing a total of 558 insights using Health Timeline and 576 without, the mean value using the Timeline (1.7) was higher than without (1.26; p<0.01), similarly the cumulative value with the Timeline (11.87) was higher than without (10.96: p<0.01). The average time required to formulate the first insight with the Timeline was higher (13.16 s) than without (7 s; p<0.01). Seven insights achieved the highest possible value using Health Timeline while none were obtained without it.ConclusionsThe Health Timeline effectively improved understanding of clinical data and helped participants recognize complex patterns from the data. By applying the insight-based methodology, the effectiveness of the Health Timeline was quantified, documented and demonstrated. As an outcome of this exercise, we propose the use of such methodologies to measure the effectiveness of visualizations that assist the clinical decision-making process.

Developing a dynamic HIV transmission model for 6 U.S. cities: An evidence synthesis.

BackgroundDynamic HIV transmission models can provide evidence-based guidance on optimal combination implementation strategies to treat and prevent HIV/AIDS. However, these models can be extremely data intensive, and the availability of good-quality data characterizing regional microepidemics varies substantially within and across countries. We aim to provide a comprehensive and transparent description of an evidence synthesis process and reporting framework employed to populate and calibrate a dynamic, compartmental HIV transmission model for six US cities.MethodsWe executed a mixed-method evidence synthesis strategy to populate model parameters in six categories: (i) initial HIV-negative and HIV-infected populations; (ii) parameters used to calculate the probability of HIV transmission; (iii) screening, diagnosis, treatment and HIV disease progression; (iv) HIV prevention programs; (v) the costs of medical care; and (vi) health utility weights for each stage of HIV disease progression. We identified parameters that required city-specific data and stratification by gender, risk group and race/ethnicity a priori and sought out databases for primary analysis to augment our evidence synthesis. We ranked the quality of each parameter using context- and domain-specific criteria and verified sources and assumptions with our scientific advisory committee.FindingsTo inform the 1,667 parameters needed to populate our model, we synthesized evidence from 59 peer-reviewed publications and 24 public health and surveillance reports and executed primary analyses using 11 data sets. Of these 1,667 parameters, 1,517 (91%) were city-specific and 150 (9%) were common for all cities. Notably, 1,074 (64%), 201 (12%) and 312 (19%) parameters corresponded to categories (i), (ii) and (iii), respectively. Parameters ranked as best- to moderate-quality evidence comprised 39% of the common parameters and ranged from 56%-60% across cities for the city-specific parameters. We identified variation in parameter values across cities as well as within cities across risk and race/ethnic groups.ConclusionsBetter integration of modelling in decision making can be achieved by systematically reporting on the evidence synthesis process that is used to populate models, and by explicitly assessing the quality of data entered into the model. The effective communication of this process can help prioritize data collection of the most informative components of local HIV prevention and care services in order to reduce decision uncertainty and strengthen model conclusions.

Domain-specific model selection for structural identification of the Rab5-Rab7 dynamics in endocytosis.

BackgroundGiven its recent rapid development and the central role that modeling plays in the discipline, systems biology clearly needs methods for automated modeling of dynamical systems. Process-based modeling focuses on explanatory models of dynamical systems; it constructs such models from measured time-course data and formalized modeling knowledge. In this paper, we apply process-based modeling to the practically relevant task of modeling the Rab5-Rab7 conversion switch in endocytosis. The task is difficult due to the limited observability of the system variables and the noisy measurements, which pose serious challenges to the process of model selection. To address these issues, we propose a domain-specific model selection criteria that take into account knowledge about the necessary properties of the simulated model behavior.ResultsIn a series of modeling experiments, we compare the results of process-based modeling obtained with different model selection criteria. The first is the standard maximum likelihood criterion based solely on least-squares model error. The second one is a parsimony-based criterion that also takes into account model complexity. We also introduce three domain-specific criteria based on domain expert expectations about the simulated behavior of an endocytosis model. According to the first criterion, 90 of the candidate models are indistinguishable. Furthermore, taking into account the complexity of the model does not lead to better model selection. However, the use of domain-specific criteria results in a remarkable improvement over the other two model selection criteria.ConclusionsWe demonstrate the applicability of process-based modeling to the task of modeling the Rab5-Rab7 dynamics in endocytosis. Our experiments show that the domain-specific criteria outperform the standard domain-independent criteria for model selection. We also find that some of the model structures discarded as implausible in previous studies lead to the expected Rab5-Rab7 switch behavior.Electronic supplementary materialThe online version of this article (doi:10.1186/s12918-015-0175-x) contains supplementary material, which is available to authorized users.

The INTERGROWTH-21st Project Neurodevelopment Package: a novel method for the multi-dimensional assessment of neurodevelopment in pre-school age children.

BackgroundThe International Fetal and Newborn Growth Consortium for the 21st Century (INTERGROWTH-21st) Project is a population-based, longitudinal study describing early growth and development in an optimally healthy cohort of 4607 mothers and newborns. At 24 months, children are assessed for neurodevelopmental outcomes with the INTERGROWTH-21st Neurodevelopment Package. This paper describes neurodevelopment tools for preschoolers and the systematic approach leading to the development of the Package.MethodsAn advisory panel shortlisted project-specific criteria (such as multi-dimensional assessments and suitability for international populations) to be fulfilled by a neurodevelopment instrument. A literature review of well-established tools for preschoolers revealed 47 candidates, none of which fulfilled all the project's criteria. A multi-dimensional assessment was, therefore, compiled using a package-based approach by: (i) categorizing desired outcomes into domains, (ii) devising domain-specific criteria for tool selection, and (iii) selecting the most appropriate measure for each domain.ResultsThe Package measures vision (Cardiff tests); cortical auditory processing (auditory evoked potentials to a novelty oddball paradigm); and cognition, language skills, behavior, motor skills and attention (the INTERGROWTH-21st Neurodevelopment Assessment) in 35–45 minutes. Sleep-wake patterns (actigraphy) are also assessed. Tablet-based applications with integrated quality checks and automated, wireless electroencephalography make the Package easy to administer in the field by non-specialist staff. The Package is in use in Brazil, India, Italy, Kenya and the United Kingdom.ConclusionsThe INTERGROWTH-21st Neurodevelopment Package is a multi-dimensional instrument measuring early child development (ECD). Its developmental approach may be useful to those involved in large-scale ECD research and surveillance efforts.

Evaluation of electronic maritime services using fuzzy logic techniques

Our paper proposes an innovative methodological approach in order to evaluate e-business services. We assume a Multicriteria Decision Analysis (MCDA) framework, where sets of general and domain-specific criteria are used to judge the relative performance of e-services under evaluation. In this paper we present the main characteristics of the evaluation framework, including the assessment of the evaluation criteria and their relative importance, the evaluation of the performance and the final ranking of e-services. The proposed methodology has the innovative feature of embodying techniques of fuzzy logic theory into the classical multicriteria decision analysis. We present an application of our methodology in the maritime sector where we assume an e-services platform model, which supports operational and commercial activities of shipping companies on the internet. In the presented case study, we apply the proposed methodology in order to evaluate six electronic maritime services using totally 16 criteria.

The criteria nurses use in assessing acute trauma in military emergency care

Emergency medical care for seriously injured patients in war or warlike situations is highly important when it comes to soldiers' survival and morale. The Swedish Armed Forces sends nurses, who have limited experience of caring for injured personnel in the field, on a variety of international missions. The aim of this investigation was to identify the kind of criteria nurses rely on when assessing acute trauma and what factors are affecting the emergency care of injured soldiers. A phenomenographic research approach based on interviews was used. The database for the study consists of twelve nurses who served in Bosnia in 1994-1996. The criteria nurses rely on, when assessing acute trauma in emergency care, could be described in terms of domain-specific criteria such as a physiological, an anatomical, a causal and a holistic approach as well as contextual criteria such as being able to communicate, having a sense of belonging, the military environment, the conscript medical orderly and familiarity with health-caring activity. The present study shows that the specific contextual factors affecting emergency care in the field must also be practised before the nurse faces military emergency care situations. This calls for realistic exercises and training programs, where experience from civilian emergency care is interwoven with the knowledge specific to military medical care.

THE EVALUATION OF INFORMATION TECHNOLOGY PROJECTS: A FUZZY MULTICRITERIA DECISION-MAKING APPROACH

This paper presents a fuzzy set-based approach to the evaluation of information technology (IT) projects. We assume a multi-criteria decision-making framework, where sets of general and domain-specific criteria are used to judge the relative performance of alternative technologies. The methodology was originally developed for DIAS.net, an EU project aiming at the development of the Information Society in insular and isolated regions of Europe. In this paper, we present many aspects of our evaluation framework, including the synthesis of evaluation teams, the assessment of the importance of criteria, the evaluation of the performance of the alternatives and the final ranking and selection of projects. The methodology presented has the innovative feature of embodying techniques of fuzzy sets theory into the classical multi-criteria decision analysis. This combination enables us to handle efficiently the subjectiveness that often characterizes expert judgements on a decision problem. Fuzzy linguistic terms, such as "poor," "fair," "very important," etc. are proposed for assessing the relative merit of alternatives and criteria. The paper concludes by exploring the potentiality of the above methodology in providing a flexible and robust IT evaluation framework.