Active Learning Setting Research Articles

FoodAtlas: Automated knowledge extraction of food and chemicals from literature

Automated generation of knowledge graphs that accurately capture published information can help with knowledge organization and access, which have the potential to accelerate discovery and innovation. Here, we present an integrated pipeline to construct a large-scale knowledge graph using large language models in an active learning setting. We apply our pipeline to the association of raw food, ingredients, and chemicals, a domain that lacks such knowledge resources. By using an iterative active learning approach of 4120 manually curated premise-hypothesis pairs as training data for ten consecutive cycles, the entailment model extracted 230,848 food-chemical composition relationships from 155,260 scientific papers, with 106,082 (46.0 %) of them never been reported in any published database. To augment the knowledge incorporated in the knowledge graph, we further incorporated information from 5 external databases and ontology sources. We then applied a link prediction model to identify putative food-chemical relationships that were not part of the constructed knowledge graph. Validation of the 443 hypotheses generated by the link prediction model resulted in 355 new food-chemical relationships, while results show that the model score correlates well (R2 = 0.70) with the probability of a novel finding. This work demonstrates how automated learning from literature at scale can accelerate discovery and support practical applications through reproducible, evidence-based capture of latent interactions of diverse entities, such as food and chemicals.

Simulation experiment design for calibration via active learning

Simulation models often have parameters as input and return outputs to understand the behavior of complex systems. Calibration is the process of estimating the values of the parameters in a simulation model in light of observed data from the system that is being simulated. When simulation models are expensive, emulators are built with simulation data as a computationally efficient approximation of an expensive model. An emulator then can be used to predict model outputs, instead of repeatedly running an expensive simulation model during the calibration process. Sequential design with an intelligent selection criterion can guide the process of collecting simulation data to build an emulator, making the calibration process more efficient and effective. This article proposes two novel criteria for sequentially acquiring new simulation data in an active learning setting by considering uncertainties on the posterior density of parameters. Analysis of several simulation experiments and real-data simulation experiments from epidemiology demonstrates that proposed approaches result in improved posterior and field predictions.

Maintaining the impact of action-oriented entrepreneurship training: the role of error mastery orientation for training transfer in an active learning setting

ABSTRACT Action-oriented entrepreneurship training can boost entrepreneurship, but positive short-term effects often decay over time. Based on training transfer theory, we argue that error mastery orientation is a personal characteristic that affects the maintenance of training outcomes over time in the transfer context of entrepreneurship. We implemented an action-oriented entrepreneurship training at a university in Uganda and collected data from training participants before (NT0 = 175) and after the training programme (NT1 = 164) as well as at a 1-year follow-up measurement (NT2 = 52). Growth curve models showed that the participants experienced an increase in entrepreneurial action and new venture creation over time. Furthermore, error mastery orientation influenced the maintenance of entrepreneurial action and new venture creation over time. The study contributes to theories of entrepreneurship training and training transfer by demonstrating that error mastery orientation is a boundary condition of the long-term effectiveness of active learning interventions, such as action-oriented entrepreneurship training.

The System Architecture-Function-Outcome Framework for Fostering and Assessing Systems Thinking in First-Year STEM Education and Its Potential Applications in Case-Based Learning

Systems thinking is crucial for understanding and solving complex problems and is considered an important thinking skill in engineering. Active learning is considered an effective approach for fostering STEM students’ systems thinking. However, viable methods for teaching and assessing systems thinking with active learning across STEM disciplines, particularly in first-year undergraduate education, are still under-researched. In this paper, we introduce a research-based framework named System Architecture-Function-Outcome to help first-year STEM instructors both foster and assess students’ introductory systems thinking. To conduct an initial evaluation of the framework’s suitability in active learning settings, we designed a directed case-based learning assignment with an adapted article and a rubric for assessing ‘introductory systems thinking’, as defined in the framework. We deployed the assignment among 84 first-year STEM students and successfully tested its inter-rater reliability, with 75–100% inter-rater agreement across all assessment criteria. We discuss the implications of our results on fostering and assessing first-year STEM students’ systems thinking, and outline examples for potential applications of the framework, pending further validation, in case-based learning settings of varying degrees of learner autonomy, from lecture-based to problem-based learning.

A glimpse of social presence during an organic chemistry group activity

IntroductionThe US must strengthen the STEM workforce to provide solutions that support a resilient society and robust economy. Such solutions require a range of perspectives from individuals with diverse identities. However, women and individuals who identify as a racial or ethnic minority are woefully underrepresented in the STEM workforce. The reality doubly impacts those with intersecting identities, such as Black women. Efforts to increase the representation of Black women in STEM disciplines include interventions that improve their success in undergraduate courses. Paramount to this is expanding understanding of the experiences of Black women in the academic setting. The current work is situated within the social presence of the Community of Inquiry (CoI) framework, examining social talk among Black women in the learning environment. The CoI model holds that the interconnection between teaching, cognitive, and social presences influences how students engage to master course content. More specifically, each presence is critical for the student’s ability to connect with their peers and instructor and engage in discourse related to the application of information.MethodsThe characterization of social presence was captured through audio and visual recordings. The videos were examined for academic and non-academic social talk. The group examined consisted of four Black women at a HBCU.ResultsInstances of social talk were captured that aligned with affective and emotional expression, open communication, and group cohesion. The dialog was both academic in nature, corresponding to the task, and non-academic, unrelated to the task.DiscussionFew studies have been done to characterize CoI in academic spaces consisting only of Black women. Disaggregating the stories of Black women from the broader context provides the visibility needed to understand their experiences and imagine a STEM paradigm responsive to their existence. Such knowledge is critical to understanding better how women, particularly women of African descent, navigate the learning environment. The article provides a preliminary look at Black women in an active learning setting to shed light on the experiences of these individuals in the STEM context.

The Interactive Effect of Personality Types and Learning Setting on Performance in Accounting Courses

ABSTRACT This paper examines whether students’ personality types influence their performance in two masters-level accounting courses: one traditional lecture setting (Financial Accounting) and one active learning setting (Accounting Research). Research has shown that accounting student performance is related to personality types (Nourayi and Cherry 1993; Lawrence and Taylor 2000). However, the accounting profession is pushing for active learning to emphasize the importance of communication and critical thinking skills (Pincus, Stout, Sorensen, Stocks, and Lawson 2017). Our study is the first to relate personality types to performance in learning settings in the accounting area. We also extend prior personality type research to examine masters-level accounting student performance. We find that Extrovert students outperformed Introvert students in the active learning setting and Sensing students outperformed Intuitive students in the traditional lecture setting. Combining personality types into four personality temperaments and four MBTI cognitive function groups also resulted in significant effects on student performance. Data Availability: Data are available from the authors upon request.

GANDALF: Graph-based transformer and Data Augmentation Active Learning Framework with interpretable features for multi-label chest Xray classification

Informative sample selection in an active learning (AL) setting helps a machine learning system attain optimum performance with minimum labeled samples, thus reducing annotation costs and boosting performance of computer-aided diagnosis systems in the presence of limited labeled data. Another effective technique to enlarge datasets in a small labeled data regime is data augmentation. An intuitive active learning approach thus consists of combining informative sample selection and data augmentation to leverage their respective advantages and improve the performance of AL systems. In this paper, we propose a novel approach called GANDALF (Graph-based TrANsformer and Data Augmentation Active Learning Framework) to combine sample selection and data augmentation in a multi-label setting. Conventional sample selection approaches in AL have mostly focused on the single-label setting where a sample has only one disease label. These approaches do not perform optimally when a sample can have multiple disease labels (e.g., in chest X-ray images). We improve upon state-of-the-art multi-label active learning techniques by representing disease labels as graph nodes and use graph attention transformers (GAT) to learn more effective inter-label relationships. We identify the most informative samples by aggregating GAT representations. Subsequently, we generate transformations of these informative samples by sampling from a learned latent space. From these generated samples, we identify informative samples via a novel multi-label informativeness score, which beyond the state of the art, ensures that (i) generated samples are not redundant with respect to the training data and (ii) make important contributions to the training stage. We apply our method to two public chest X-ray datasets, as well as breast, dermatology, retina and kidney tissue microscopy MedMNIST datasets, and report improved results over state-of-the-art multi-label AL techniques in terms of model performance, learning rates, and robustness.

Highly Efficient Active Learning With Tracklet-Aware Co-Cooperative Annotators for Person Re-Identification.

Supervised person re-identification (ReID) has attracted widespread attentions in the computer vision community due to its great potential in real-world applications. However, the demand of human annotation heavily limits the application as it is costly to annotate identical pedestrians appearing from different cameras. Thus, how to reduce the annotation cost while preserving the performance remains challenging and has been studied extensively. In this article, we propose a tracklet-aware co-cooperative annotators' framework to reduce the demand of human annotation. Specifically, we partition the training samples into different clusters and associate adjacent images in each cluster to produce the robust tracklet which decreases the annotation requirements significantly. Besides, to further reduce the cost, we introduce a powerful teacher model in our framework to implement the active learning strategy and select the most informative tracklets for human annotator, the teacher model itself, in our setting, also acts as an annotator to label the relatively certain tracklets. Thus, our final model could be well-trained with both confident pseudo-labels and human-given annotations. Extensive experiments on three popular person ReID datasets demonstrate that our approach could achieve competitive performance compared with state-of-the-art methods in both active learning and unsupervised learning (USL) settings.

Evaluation of uncertainty quantification methods in multi-label classification: A case study with automatic diagnosis of electrocardiogram

Artificial Intelligence (AI) use in automated Electrocardiogram (ECG) classification has continuously attracted the research community’s interest, motivated by their promising results. Despite their great promise, limited attention has been paid to the robustness of their results, which is a key element for their implementation in clinical practice. Uncertainty Quantification (UQ) is a critical for trustworthy and reliable AI, particularly in safety-critical domains such as medicine. Estimating uncertainty in Machine Learning (ML) model predictions has been extensively used for Out-of-Distribution (OOD) detection under single-label tasks. However, the use of UQ methods in multi-label classification remains underexplored.This study goes beyond developing highly accurate models comparing five uncertainty quantification methods using the same Deep Neural Network (DNN) architecture across various validation scenarios, including internal and external validation as well as OOD detection, taking multi-label ECG classification as the example domain. We show the importance of external validation and its impact on classification performance, uncertainty estimates quality, and calibration. Ensemble-based methods yield more robust uncertainty estimations than single network or stochastic methods. Although current methods still have limitations in accurately quantifying uncertainty, particularly in the case of dataset shift, incorporating uncertainty estimates with a classification with a rejection option improves the ability to detect such changes. Moreover, we show that using uncertainty estimates as a criterion for sample selection in active learning setting results in greater improvements in classification performance compared to random sampling.

A Semi-parametric Model for Decision Making in High-Dimensional Sensory Discrimination Tasks

Psychometric functions typically characterize binary sensory decisions along a single stimulus dimension. However, real-life sensory tasks vary along a greater variety of dimensions (e.g. color, contrast and luminance for visual stimuli). Approaches to characterizing high-dimensional sensory spaces either require strong parametric assumptions about these additional contextual dimensions, or fail to leverage known properties of classical psychometric curves. We overcome both limitations by introducing a semi-parametric model of sensory discrimination that applies traditional psychophysical models along a stimulus intensity dimension, but puts Gaussian process (GP) priors on the parameters of these models with respect to the remaining dimensions. By combining the flexibility of the GP with the deep literature on parametric psychophysics, our semi-parametric models achieve good performance with much less data than baselines on both synthetic and real-world, high-dimensional psychophysics datasets. We additionally show strong performance in a Bayesian active learning setting, and present a novel active learning paradigm for the semi-parametric model.

Fast uncertainty estimates in deep learning interatomic potentials.

Deep learning has emerged as a promising paradigm to give access to highly accurate predictions of molecular and material properties. A common short-coming shared by current approaches, however, is that neural networks only give point estimates of their predictions and do not come with predictive uncertainties associated with these estimates. Existing uncertainty quantification efforts have primarily leveraged the standard deviation of predictions across an ensemble of independently trained neural networks. This incurs a large computational overhead in both training and prediction, resulting in order-of-magnitude more expensive predictions. Here, we propose a method to estimate the predictive uncertainty based on a single neural network without the need for an ensemble. This allows us to obtain uncertainty estimates with virtually no additional computational overhead over standard training and inference. We demonstrate that the quality of the uncertainty estimates matches those obtained from deep ensembles. We further examine the uncertainty estimates of our methods and deep ensembles across the configuration space of our test system and compare the uncertainties to the potential energy surface. Finally, we study the efficacy of the method in an active learning setting and find the results to match an ensemble-based strategy at order-of-magnitude reduced computational cost.

Active learning and novel model calibration measurements for automated visual inspection in manufacturing

Quality control is a crucial activity performed by manufacturing enterprises to ensure that their products meet quality standards and avoid potential damage to the brand’s reputation. The decreased cost of sensors and connectivity enabled increasing digitalization of manufacturing. In addition, artificial intelligence enables higher degrees of automation, reducing overall costs and time required for defect inspection. This research compares three active learning approaches, having single and multiple oracles, to visual inspection. Six new metrics are proposed to assess the quality of calibration without the need for ground truth. Furthermore, this research explores whether existing calibrators can improve performance by leveraging an approximate ground truth to enlarge the calibration set. The experiments were performed on real-world data provided by Philips Consumer Lifestyle BV. Our results show that the explored active learning settings can reduce the data labeling effort by between three and four percent without detriment to the overall quality goals, considering a threshold of p = 0.95. Furthermore, the results show that the proposed calibration metrics successfully capture relevant information otherwise available to metrics used up to date only through ground truth data. Therefore, the proposed metrics can be used to estimate the quality of models’ probability calibration without committing to a labeling effort to obtain ground truth data.

Delay-Adaptive Learning in Generalized Linear Contextual Bandits

In this paper, we consider online learning in generalized linear contextual bandits where rewards are not immediately observed. Instead, rewards are available to the decision maker only after some delay, which is unknown and stochastic. Such delayed feedback occurs in several active learning settings, including product recommendation, personalized medical treatment selection, bidding in first-price auctions, and bond trading in over-the-counter markets. We study the performance of two well-known algorithms adapted to this delayed setting: one based on upper confidence bounds and the other based on Thompson sampling. We describe modifications on how these two algorithms should be adapted to handle delays and give regret characterizations for both algorithms. To the best of our knowledge, our regret bounds provide the first theoretical characterizations in generalized linear contextual bandits with large delays. Our results contribute to the broad landscape of contextual bandits literature by establishing that both algorithms can be made to be robust to delays, thereby helping clarify and reaffirm the empirical success of these two algorithms, which are widely deployed in modern recommendation engines. Funding: This work was supported by the National Science Foundation [Grants 2118199, 1915967, and CCF-2106508], the Air Force Office of Scientific Research [Award FA9550-20-1-0397], a Digital Twin research grant from Bain & Company, and a faculty research grant from New York University’s Center for Global Economy and Business.

The effect of flipped learning on blood pressure knowledge and self-directed learning skills of first-year nursing students: A randomized controlled trial

AimThis study investigated the effect of flipped learning on first-year nursing students’ blood pressure knowledge levels and self-directed learning skills. BackgroundFlipped learning is an innovative approach that helps nursing students learn about blood pressure and how to measure how to measure blood pressure accurately. Flipped learning also promotes active and student-centered learning settings and encourages nursing students to develop self-directed skills. DesignThis study adopted a pretest-posttest open-label randomized controlled trial. MethodThe sample consisted of 94 first-year nursing students randomized into experimental (n = 48) and control groups (n = 46). The experimental group participants were trained using the flipped learning model. Data were collected using a personal information form, the Blood Pressure Knowledge Test (BPKT) and the Self-Directed Learning Skills Scale (SDLSS). ResultsThere was no significant difference in pretest BPKT scores between the experimental and control groups. However, there was a significant difference in posttest BPKT scores between the experimental and control groups (p = 0.011). Moreover, there was a significant difference between the experimental group's mean pretest, posttest and follow-up SDLSS scores (p = 0.009). ConclusionThe experimental group had a significantly higher mean posttest BPKT score than the pretest score. They had significantly higher mean posttest SDLSS total and “self-monitoring,” “motivation,” and “self-confidence” subscale scores than the pretest score.

Embracing discomfort in active learning and technology-rich higher education settings: sensemaking through reflexive inquiry

AbstractPreparing students for professional life in a Volatile, Uncertain, Complex and Ambiguous twenty-first century world has seen tertiary institutions eschew the traditional ‘lecture-tutorial’ model in favour of active learning approaches. However, implementing these approaches is not unproblematic. This paper explores how we navigated the tensions of cultivating twenty-first century skills in our students—first-year preservice teachers—through a purposely designed approach to active learning in an educational technology course. We illustrate how deploying Bakhtinian precepts through reflexive inquiry supported sensemaking of discomfort, leveraging this sensemaking to reinvigorate practice.

Impact of combination of short lecture and group discussion on the learning of physiology by nonmajor undergraduates.

This study assessed the impact of an "active learning" strategy employed alone or in combination with traditional lectures on the learning of mammalian physiology by undergraduate students. The study investigated the impact of three teaching strategies, namely 1) traditional lecture, 2) group discussion alone, and 3) combination of lecture and group discussion. For all strategies, students were given homework in a textbook and they completed written assignments before each session. Every student led the discussion of at least one assigned theme during each group session. The students had no access to the textbook or notes during group discussions. Four examinations (3 in-semester and a final) assessed the students' knowledge of fundamental concepts of physiology of specific organ systems. Part of the final examination reassessed knowledge of previously tested topics. The results show that the teaching modality employed to introduce physiology topics influenced students' learning. The average marginal effect of the lecture + discussion modality (average improvement linked to lecture + discussion strategy) on students' performance was 6.45% [95% confidence interval (CI95) (4.73, 8.16), P = 1.74 × 10-13], and the average improvement associated with the discussion-only modality was 5.5% [CI95 (3.84, 7.16), P = 7.84 × 10-11]. On average, all class ranks performed better on materials covered under active learning settings than under lecture-only conditions. Moreover, students' performance under combined lecture and discussion conditions is predictive of their overall performance in the course. The results support the positive effect of student-centered learning and demonstrate the efficacy of a combination of lectures and group discussions on learning of physiology by nonmajor students.NEW & NOTEWORTHY The purpose of this study was to evaluate the effect of group discussion on the learning of mammalian physiology by nonmajor undergraduate students. Combining traditional lectures with group discussions increased the active participation of students in class and improved their learning of physiology, as measured by the results of in-semester and final examinations. The active learning technique benefited all class ranks on average.

An active learning method for diabetic retinopathy classification with uncertainty quantification.

In recent years, deep learning (DL) techniques have provided state-of-the-art performance in medical imaging. However, good quality (annotated) medical data is in general hard to find due to the usually high cost of medical images, limited availability of expert annotators (e.g., radiologists), and the amount of time required for annotation. In addition, DL is data-hungry and its training requires extensive computational resources. Furthermore, DL being a black-box method lacks transparency on its inner working and lacks fundamental understanding behind decisions made by the model and consequently, this notion enhances the uncertainty on its predictions. To this end, we address these challenges by proposing a hybrid model, which uses a Bayesian convolutional neural network (BCNN) for uncertainty quantification, and an active learning approach for annotating the unlabeled data. The BCNN is used as a feature descriptor and these features are then used for training a model, in an active learning setting. We evaluate the proposed framework for diabetic retinopathy classification problem and demonstrate state-of-the-art performance in terms of different metrics.

Batch Active Learning with Graph Neural Networks via Multi-Agent Deep Reinforcement Learning

Graph neural networks (GNNs) have achieved tremendous success in many graph learning tasks such as node classification, graph classification and link prediction. For the classification task, GNNs' performance often highly depends on the number of labeled nodes and thus could be significantly hampered due to the expensive annotation cost. The sparse literature on active learning for GNNs has primarily focused on selecting only one sample each iteration, which becomes inefficient for large scale datasets. In this paper, we study the batch active learning setting for GNNs where the learning agent can acquire labels of multiple samples at each time. We formulate batch active learning as a cooperative multi-agent reinforcement learning problem and present a novel reinforced batch-mode active learning framework BiGeNe. To avoid the combinatorial explosion of the joint action space, we introduce a value decomposition method that factorizes the total Q-value into the average of individual Q-values. Moreover, we propose a novel multi-agent Q-network consisting of a graph convolutional network (GCN) component and a gated recurrent unit (GRU) component. The GCN component takes both the informativeness and inter-dependences between nodes into account and the GRU component enables the agent to consider interactions between selected nodes in the same batch. Experimental results on multiple public datasets demonstrate the effectiveness and efficiency of our proposed method.

Active learning of causal structures with deep reinforcement learning

We study the problem of experiment design to learn causal structures from interventional data. We consider an active learning setting in which the experimenter decides to intervene on one of the variables in the system in each step and uses the results of the intervention to recover further causal relationships among the variables. The goal is to fully identify the causal structures with minimum number of interventions. We present the first deep reinforcement learning based solution for the problem of experiment design. In the proposed method, we embed input graphs to vectors using a graph neural network and feed them to another neural network which outputs a variable for performing intervention in each step. Both networks are trained jointly via a Q-iteration algorithm. Experimental results show that the proposed method achieves competitive performance in recovering causal structures with respect to previous works, while significantly reducing execution time in dense graphs.

Active Learning: A Shift from Passive Learning to Student Engagement Improves Understanding and Contextualization of Nutrition and Community Health

Challenging the notion of “teaching by telling,” active learning utilizes a student-oriented approach by emphasizing the concept of knowledge retention through peer interaction. To further examine the potential of active learning, we created a workshop based on didactic education and student collaboration. Participants included undergraduate students from traditionally underrepresented and disadvantaged backgrounds. The workshop was part of our summer academic enrichment program run in an urban, medically underserved community. The workshop focused on clinical and biochemical nutrition, wherein students synthesized information by discussing dietary choices and the socioeconomic aspects of nutrition. Student reception of the workshop was adjudged by anonymous surveys. The survey questions were designed to gauge how the workshop objectives were achieved. Cronbach alpha (0.276) confirmed that there was more than a single theme contained in the questions. The majority of students (97%) agreed that the workshop met the learning objectives: (1) acquire basic clinical knowledge, (2) gain a better understanding of nutrition, (3) formulate a linkage between clinical nutrition and disease, and (4) benefit from peer interaction. Students’ performance in the post-quiz (100% correct answers) had improved significantly compared to the pre-quiz (25% correct answers) suggesting acquisition, understanding and application of nutrition aspects taught in the workshop. Overall, the present study demonstrated the engagement and understanding of students with respect to learning about nutrition and community health in an active learning setting. These types of active-learning-based sessions may have broad applicability for any academic discipline to improve student engagement and knowledge retention.