Domain Expert User Research Articles

Creating a biomedical knowledge base by addressing GPT inaccurate responses and benchmarking context.

We created GNQA, a generative pre-trained transformer (GPT) knowledge base driven by a performant retrieval augmented generation (RAG) with a focus on aging, dementia, Alzheimer's and diabetes. We uploaded a corpus of three thousand peer reviewed publications on these topics into the RAG. To address concerns about inaccurate responses and GPT 'hallucinations', we implemented a context provenance tracking mechanism that enables researchers to validate responses against the original material and to get references to the original papers. To assess the effectiveness of contextual information we collected evaluations and feedback from both domain expert users and 'citizen scientists' on the relevance of GPT responses. A key innovation of our study is automated evaluation by way of a RAG assessment system (RAGAS). RAGAS combines human expert assessment with AI-driven evaluation to measure the effectiveness of RAG systems. When evaluating the responses to their questions, human respondents give a "thumbs-up" 76% of the time. Meanwhile, RAGAS scores 90% on answer relevance on questions posed by experts. And when GPT-generates questions, RAGAS scores 74% on answer relevance. With RAGAS we created a benchmark that can be used to continuously assess the performance of our knowledge base. Full GNQA functionality is embedded in the free GeneNetwork.org web service, an open-source system containing over 25 years of experimental data on model organisms and human. The code developed for this study is published under a free and open-source software license at https://git.genenetwork.org/gn-ai/tree/README.md.

Selection-Bias-Corrected Visualization via Dynamic Reweighting.

The collection and visual analysis of large-scale data from complex systems, such as electronic health records or clickstream data, has become increasingly common across a wide range of industries. This type of retrospective visual analysis, however, is prone to a variety of selection bias effects, especially for high-dimensional data where only a subset of dimensions is visualized at any given time. The risk of selection bias is even higher when analysts dynamically apply filters or perform grouping operations during ad hoc analyses. These bias effects threaten the validity and generalizability of insights discovered during visual analysis as the basis for decision making. Past work has focused on bias transparency, helping users understand when selection bias may have occurred. However, countering the effects of selection bias via bias mitigation is typically left for the user to accomplish as a separate process. Dynamic reweighting (DR) is a novel computational approach to selection bias mitigation that helps users craft bias-corrected visualizations. This paper describes the DR workflow, introduces key DR visualization designs, and presents statistical methods that support the DR process. Use cases from the medical domain, as well as findings from domain expert user interviews, are also reported.

Development of a Custom Spell-Checker for Emergency Department Data

ObjectiveTo share progress on a custom spell-checker for emergency department chief complaint free-text data and demonstrate a spell-checker validation Shiny application.IntroductionEmergency department (ED) syndromic surveillance relies on a chief complaint, which is often a free-text field, and may contain misspelled words, syntactic errors, and healthcare-specific and/or facility-specific abbreviations. Cleaning of the chief complaint field may improve syndrome capture sensitivity and reduce misclassification of syndromes. We are building a spell-checker, customized with language found in ED corpora, as our first step in cleaning our chief complaint field. This exercise would elucidate the value of pre-processing text and would lend itself to future work using natural language processing (NLP) techniques, such as topic modeling. Such a tool could be extensible to other datasets that contain free-text fields, including electronic reportable disease lab and case reporting.MethodsChief complaints may contain words that are incorrect if they are misspelled (e.g.,“patient has herpertension”), or, if the word yields a syntactically incorrect phrase (e.g., the word “huts” in the phrase: “my toe huts”).We are developing a spell-checker tool for chief complaint text using the R and Python programming languages. The first stage in the development of the spell-checker is the identifying and handling of misspellings; future work will address syntactic errors. Known abbreviations are identified using regular expressions, and unknown abbreviations are addressed by the spell-checker. The spell checker performs 4 steps on chief complaint data: identification of misspellings, generation of a substitute candidate word list, word sense disambiguation to identify replacement word, and replacement of the misspelled word, based on methods found in the literature.[1] As the spell-checker requires a dictionary of correctly spelled, healthcare-specific terms including all terms that would appear in an ED corpus, we used vocabularies from the Unified Medical Language System, ED-specific terminology, and domain expert user input. Dictionary construction, misspelling identification algorithms, and word list generation algorithms are in the development stage.Simultaneously, we are building an R Shiny interactive web application for syndromic surveillance analysts to manually correct a subset of misspelled words, which we will use to validate and evaluate the performance of the spell-checker tool.[1] Tolentino HD, Matters MD, Walop W, et al. A UMLS-based spell checker for natural language processing in vaccine safety. BMC Medical Informatics and Decision Making. 2007;7(1). doi:10.1186/1472-6947-7-3.ResultsProject still in development phase.ConclusionsThe audience will learn about important considerations for developing a spell-checker, including those for data structure of a dictionary and algorithms for identification of misplaced words and identification of candidate replacement words. We will demonstrate our word list generation algorithm and the Shiny application which uses these words for spell-checker validation. We will share relevant code; after our presentation, audience members should able to apply code and lessons to their own projects and/or to collaborate with the NYC Department of Health and Mental Hygiene.

Selection Bias Tracking and Detailed Subset Comparison for High-Dimensional Data.

The collection of large, complex datasets has become common across a wide variety of domains. Visual analytics tools increasingly play a key role in exploring and answering complex questions about these large datasets. However, many visualizations are not designed to concurrently visualize the large number of dimensions present in complex datasets (e.g. tens of thousands of distinct codes in an electronic health record system). This fact, combined with the ability of many visual analytics systems to enable rapid, ad-hoc specification of groups, or cohorts, of individuals based on a small subset of visualized dimensions, leads to the possibility of introducing selection bias-when the user creates a cohort based on a specified set of dimensions, differences across many other unseen dimensions may also be introduced. These unintended side effects may result in the cohort no longer being representative of the larger population intended to be studied, which can negatively affect the validity of subsequent analyses. We present techniques for selection bias tracking and visualization that can be incorporated into high-dimensional exploratory visual analytics systems, with a focus on medical data with existing data hierarchies. These techniques include: (1) tree-based cohort provenance and visualization, including a user-specified baseline cohort that all other cohorts are compared against, and visual encoding of cohort "drift", which indicates where selection bias may have occurred, and (2) a set of visualizations, including a novel icicle-plot based visualization, to compare in detail the per-dimension differences between the baseline and a user-specified focus cohort. These techniques are integrated into a medical temporal event sequence visual analytics tool. We present example use cases and report findings from domain expert user interviews.

Visual Progression Analysis of Event Sequence Data.

Event sequence data is common to a broad range of application domains, from security to health care to scholarly communication. This form of data captures information about the progression of events for an individual entity (e.g., a computer network device; a patient; an author) in the form of a series of time-stamped observations. Moreover, each event is associated with an event type (e.g., a computer login attempt, or a hospital discharge). Analyses of event sequence data have been shown to help reveal important temporal patterns, such as clinical paths resulting in improved outcomes, or an understanding of common career trajectories for scholars. Moreover, recent research has demonstrated a variety of techniques designed to overcome methodological challenges such as large volumes of data and high dimensionality. However, the effective identification and analysis of latent stages of progression, which can allow for variation within different but similarly evolving event sequences, remain a significant challenge with important real-world motivations. In this paper, we propose an unsupervised stage analysis algorithm to identify semantically meaningful progression stages as well as the critical events which help define those stages. The algorithm follows three key steps: (1) event representation estimation, (2) event sequence warping and alignment, and (3) sequence segmentation. We also present a novel visualization system, ET2, which interactively illustrates the results of the stage analysis algorithm to help reveal evolution patterns across stages. Finally, we report three forms of evaluation for ET2: (1) case studies with two real-world datasets, (2) interviews with domain expert users, and (3) a performance evaluation on the progression analysis algorithm and the visualization design.

Snorkel MeTaL: Weak Supervision for Multi-Task Learning.

Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user specifies a problem consisting of multiple, hierarchically-related sub-tasks-for example, classifying a document at multiple levels of granularity-and then provides labeling functions for each sub-task as weak supervision. MeTaL learns a re-weighted model of these labeling functions, and uses the combined signal to train a hierarchical multi-task network which is automatically compiled from the structure of the sub-tasks. Using MeTaL on a radiology report triage task and a fine-grained news classification task, we achieve average gains of 11.2 accuracy points over a baseline supervised approach and 9.5 accuracy points over the predictions of the user-provided labeling functions.

Software development environments and tools in MDE

Abstract. Model-Driven Engineering (MDE) is the notion that we can construct a model of a system that we can then transform into the real thing. The development of software in MDE using Domain-Specific Languages (DSLs) has two phases. First, the development of artifacts such as DSLs and transformation mechanisms by the modeling experts. Second, people non-technical experts (domain expert or end user) using the artifacts created develop applications simply because of the high level of abstraction allowed by technology. Several factors are considered to limit the use of MDE. One of them, is the lack of knowledge the tools and the development activities with MDE. To support the MDE initiative, the present work makes a description of the theoretical foundations of MDE, also describes the main activities to build several MDE artifacts with some of the tools most known in this technology.

Multiple-view strategies for enhanced understanding of dynamic tourist activity through geovisualization at regional and national scales

This article is part of research exploring various ways to visualize the complexity of sentient movement in space and time. In this case, visualization focuses on the movement of tourists on the West Coast of the South Island of New Zealand, with some commentary on national tourist activity. Working at the atomistic level of the individual we adopt space, time, and activity (s,t,a) as the three main dimensions of sentient movement, but we emphasize activity as a privileged dimension where visualization is involved. The rationale for this is that when interpreting movements of individuals or crowds, most itineraries either reflect resources in the environment that encourage travel or avoid risk or cause delay, resources ranging from food and water to incoming sunshine or rain. Activity and process is at the heart of most rich movement data-sets, and we suggest that using multiple-view visualization to decode (s,t,a) and environmental patterns opens further avenues for interpreting and communicating dynamic human and animal geographies. To this end, this article focuses on investigating multiple-view geovisualizations through the lens of six groups of multiple views, each of them demonstrating data in a unique way, in order to reveal complex (s,t,a) patterns of rich tourist movement data. An insight-based domain expert user evaluation is discussed at the end of this article.

GOALS: generator of adaptive learning scenarios

The problem of generating personalised learning activities for learners is a difficult task. The difficulty is compounded if the learning activity is mediated or presented through a serious game. In this paper, we present a system, called generator of adaptive learning scenarios GOALS that is capable of generating learning scenarios taking into account the learners' interaction traces, pedagogical objectives and the specificities of serious games. The generator we propose aims to be generic, i.e., independent of the application domain and serious games. To achieve this, we propose an architecture that organises the knowledge in three layers: domain concepts, pedagogical resources and serious game resources. This work has been conducted in the context of Cognitive Linguistic Elements Stimulations Project CLES. This project targets the development of an online serious game dedicated to persons with cognitive disabilities. In order to validate our approaches, we conducted experiments in the context of CLES project. These experiments are based on comparative method that compares the results generated by our system with that of an expert. The results of the valuations, conducted with a domain expert and end users, are also presented.

Pyroxene

Medium term management of territorial risk increase requires decision support tools able to represent and simulate spatial dynamics. For forest fire risk, simulators of spatial dynamics of both fuel zones and vulnerable zones are produced by specialised disciplines, like ecology and geography. In order to integrate simulators of spatial dynamics of different themes at different scale levels, this paper proposes a spatial agent and GIS based software infrastructure called Pyroxene. This infrastructure is designed to implement “models for models integration”, specified by domain expert users. To do so, end users have to use a specification framework, specifically designed for spatio-dynamic models integration. The purpose of the implemented platform is to execute and synchronise integrated simulations. To operate the semantic and syntax requirements of such integration, the platform is structured in an HLA-like architecture, and implemented as a multi-agents system for models integration, compliant with the FIPA specification. It is also organised around a GIS. First steps of validation confirm the validity of the system at the functional level and precise the limits of using the approach on the operational level.

An initial agent behaviour modelling and definition methodology as applied to unmanned aerial vehicle simulations

In some agent-based modelling systems, it is difficult for domain-expert users to define and implement agent behaviours without possessing extensive programming knowledge. This research developed a process through which non-programmer, domain experts can effectively interact with a modelling environment to define, develop and implement agent-based behaviours. This paper presents a new methodology for the definition and implementation of UAV agent behaviours based on diagramming an agent's controller state and using diagram syntax to generate the executable algorithm. No programming knowledge is required to reuse modular behaviours and trigger. After describing the methodology, a sample scenario is presented as a proof-of-concept.

Domain expert user development

The European Union's SmartGov project transforms public-sector employees into developers of the government e-services used directly by the public.

Environments to support context and emotion aware visual interaction

The interaction with software systems is often affected by many types of hurdles that induce users to make errors and mistakes, and to break the continuity of their reasoning while carrying out a working task with the computer. As a consequence, negative emotional states, such as frustration, dissatisfaction, and anxiety, may arise. In this paper, we illustrate how the Software Shaping Workshop (SSW) methodology can represent a solution to the problem of developing interactive systems that are correctly perceived and interpreted by end-users, thus becoming more acceptable and favouring positive emotional states. In the methodology, a key role is played by domain-expert users, that is, experts in a specific domain, not necessarily experts in computer science. Domain-expert users’ skills and background, including their knowledge of the domain and users’ needs and habits, are exploited to create context and emotion aware visual interactive systems. Examples of these systems are illustrated by referring to a case study in the automation field.

Assessing Expert Systems Impact on Users’ Jobs

A comprehensive list of ten major expert systems (ES) related factors likely to affect users' jobs has been defined, including problem importance, problem difficulty, developer skill, domain expert quality, user characteristics, user satisfaction, shell quality, user involvement, management support, and system usage. Impact on the job has been defined in terms of eleven items dealing with changes in job importance, amount of work, accuracy requirements, skills needed, job appeal, feed-back about performance, freedom in how to do the job, opportunity for advancement, job security, relation with peers, and job satisfaction. Data were collected on sixtynine expert systems developed through IBM's Corporate Manufacturing Expert Systems Project Center in San Jose, California. The results show that the major variables having the most impact on users' jobs are problem importance, problem difficulty, domain expert quality, user satisfaction with the ES, shell quality, and user involvement in ES development. Based on the results, recommendations are made for corporate and ES development managers to increase the likelihood that ES will have a desirable impact on users' jobs.