Confidence Measure Research Articles

Piloting online training in the Pacific-Ophelia project for child and adolescent mental health.

In June 2020, St Vincent's Mental Health, Fiji National University, and the Royal Australian and New Zealand College of Psychiatrists, Faculty of Child and Adolescent Psychiatry collaborated to deliver online, specialized child and adolescent mental health (CAMH) training to Pacific-based healthcare workers. This accompanying research aimed to understand the telehealth model and structures that would sustain an engaged community of practice and support the development of professional networks across the Pacific. Quantitative and qualitative feedback was analyzed to understand participation and self-rated measures of skills, knowledge, and confidence in providing health care for children and young people, as well as experiences of training, including access, engagement, and applicability of the initiative to the Pacific Islands health care organizations. Ophelia Training was able to meet the stated learning objectives. The data from all stakeholders identifies the value of a telehealth initiative incorporating training, technical assistance, knowledge networks, and professional coaching as a capacity building approach. This program offers an integration of research and practice. This regional approach to understanding telehealth capacity for Pacific Island mental health services is valuable for informing decision-making with respect to clinical care, management, workforce training and policy. It also provided an opportunity to improve health inequalities, by improving access to CAMH training via telehealth.

Sensors, Students, and Self: Exploring Knowledge, Self-Efficacy, and Interest Impact of Ocean Sensor Learning on High School Marine Science Students.

This study examined the effect of online technical lessons of how ocean sensors function on student interest in ocean science technology, as well as knowledge gain outcomes. Additionally, the study contributes novel findings to sensor-based learning literature by measuring changes to self-efficacy and confidence gains stemming from sensor-based learning, as well as changes in interest in ocean careers. The area of educational focus was also novel—focusing on how the sensors themselves function, not just what they do. Precipitated by COVID-19 pandemic constraints, the team used a remote learning approach to provide lessons on sensors at a distance, providing an additional opportunity to contrast this approach with previously studied hands-on learning modes. A sample of students from four high school marine science classes completed two assessments both before and after a series of lessons on ocean sensors. This included a self-reported survey (N = 48), and an open-ended knowledge assessment (N = 40). Results showed modest gains in knowledge assessments, and students experienced statistically significant gains in confidence in their ability to explain what sensors are, confidence in their ability to use sensors and understand resulting data, and confidence in accuracy of sensor data (p < 0.05). No changes were observed for several measures of interest in ocean technology, nor were there changes in an already high belief that understanding these sensors is important to marine science careers. Notably, these findings measure a positive shift in several measures of self-efficacy and confidence, which is a new finding for sensor-based learning. The findings also contrast with prior related work that included hands-on activities with sensors, which reported an increase in interest after working with sensors, whereas this intervention did not. This suggests a hands-on component is key to increasing interest in ocean technology.

Phase-Based Displacement Sensor With Improved Spatial Frequency Estimation and Data Fusion Strategy

Phase-based vibration measurement has attra- cted a lot of attention due to its advantages of wireless, non-contact, and full-field measurement capabilities. The phase shifts across the different video frames correspond to the motion of the structure, which makes it possible to measure the vibration with the phase. However, the decoding of the phase shifts and the presence of the phase singularities make the phase-based method noise-sensitive and non-robust when concerning vibration measurement. Within this paper, an improved framework is proposed to enhance the performance of the phase-based technique for vibration measurement. A double filtering approach combined with the O&#x2019;Shea refinement is introduced to decrease the phase noise to obtain accurate spatial frequency estimates. Then a confidence measure index is constructed to evaluate the reliability of phase responses to avoid the outliers induced by the phase singularities. Finally, an information fusion strategy over multiple scales is developed to enhance the accuracy and robustness of the motion estimation. Both simulated and experimental results verify the effectiveness and accuracy of the proposed framework.

Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation

Abstract Discourse parsing has been studied for decades. However, it still remains challenging to utilize discourse parsing for real-world applications because the parsing accuracy degrades significantly on out-of-domain text. In this paper, we report and discuss the effectiveness and limitations of bootstrapping methods for adapting modern BERT-based discourse dependency parsers to out-of-domain text without relying on additional human supervision. Specifically, we investigate self-training, co-training, tri-training, and asymmetric tri-training of graph-based and transition-based discourse dependency parsing models, as well as confidence measures and sample selection criteria in two adaptation scenarios: monologue adaptation between scientific disciplines and dialogue genre adaptation. We also release COVID-19 Discourse Dependency Treebank (COVID19-DTB), a new manually annotated resource for discourse dependency parsing of biomedical paper abstracts. The experimental results show that bootstrapping is significantly and consistently effective for unsupervised domain adaptation of discourse dependency parsing, but the low coverage of accurately predicted pseudo labels is a bottleneck for further improvement. We show that active learning can mitigate this limitation.

Deep Active Learning via Open-Set Recognition.

In many applications, data is easy to acquire but expensive and time-consuming to label, prominent examples include medical imaging and NLP. This disparity has only grown in recent years as our ability to collect data improves. Under these constraints, it makes sense to select only the most informative instances from the unlabeled pool and request an oracle (e.g., a human expert) to provide labels for those samples. The goal of active learning is to infer the informativeness of unlabeled samples so as to minimize the number of requests to the oracle. Here, we formulate active learning as an open-set recognition problem. In this paradigm, only some of the inputs belong to known classes; the classifier must identify the rest as unknown. More specifically, we leverage variational neural networks (VNNs), which produce high-confidence (i.e., low-entropy) predictions only for inputs that closely resemble the training data. We use the inverse of this confidence measure to select the samples that the oracle should label. Intuitively, unlabeled samples that the VNN is uncertain about contain features that the network has not been exposed to; thus they are more informative for future training. We carried out an extensive evaluation of our novel, probabilistic formulation of active learning, achieving state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and FashionMNIST. Additionally, unlike current active learning methods, our algorithm can learn even in the presence of out-of-distribution outliers. As our experiments show, when the unlabeled pool consists of a mixture of samples from multiple datasets, our approach can automatically distinguish between samples from seen vs. unseen datasets. Overall, our results show that high-quality uncertainty measures are key for pool-based active learning.

Safe to speak: Fostering psychological safety among incoming predoctoral dental students.

To describe the implementation and impact of an introductory session on psychological safety for incoming first-year Doctor of Dental Surgery (DDS) learners. In August 2020, first-year DDS students participated in a 2-h online instructional session focused on defining psychological safety, the behaviors that promote and disrupt it, and setting expectations for learner and faculty behavior to foster it. Learners worked in small groups to discuss the various components of psychological safety and document their discussion in a collaborative workspace before a large group debriefs. Participants were requested to complete an electronic survey before and after the session to evaluate their change in knowledge, confidence in performing tasks to foster psychological safety, and perceptions about their level of control. Six months after the session, learners were requested to participate in a focus group for additional feedback and reflections. Seventy-four participants completed the before and after experience survey. There was a statistically significant increase in knowledge of the components of psychological safety and perceptions of control. In addition, self-reported measures of confidence to perform the tasks increased from an average rating of 36% confident before the session to 86% confident afterward. In the experience, learners were active in their discussion and generated many ideas about behaviors that promote and disrupt psychological safety, which informed the creation of learner and faculty classroom expectations. Engaging learners in discussions about psychological safety can be a useful activity to establish expectations and create group norms that may support collaborative learning environments.

Research on the Whole Teaching of Vocal Music Course in University Music Performance Major Based on Multimedia Technology

MT (multimedia technology) music teaching has changed the rigid, unitary, and boring teaching form of traditional music teaching, which has made a comprehensive breakthrough and impact on traditional university music education and has shown a brand-new teaching form to contemporary college students, greatly increasing their strong interest in learning music. In this paper, an objective evaluation method of vocal music quality based on the comparison of sound parameter characteristics is proposed by using the technology of sound signal parameter analysis and extraction. Through confidence measure, reliable data are selected for online adaptive updating of Gaussian mixture model, and the recognition results are smoothed to further remove the instantaneous mutation error. The main melody of singing is discriminated by fundamental frequency discrimination model. Experiments show that the overall accuracy rate of the main melody of vocal music extracted by this algorithm is 86.24%.

Bayesian robust hankel matrix completion with uncertainty modeling for synchrophasor data recovery

Synchrophasor data suffer from quality issues like missing and bad data. Exploiting the low-rankness of the Hankel matrix of the synchrophasor data, this paper formulates the data recovery problem as a robust low-rank Hankel matrix completion problem and proposes a Bayesian data recovery method that estimates the posterior distribution of synchrophasor data from partial observations. In contrast to the deterministic approaches, our proposed Bayesian method provides an uncertainty index to evaluate the confidence of each estimation. To the best of our knowledge, this is the first method that provides confidence measure for synchrophasor data recovery. Numerical experiments on synthetic data and recorded synchrophasor data demonstrate that our method outperforms existing low-rank matrix completion methods.

Weight Annotation in Information Extraction

The framework of document spanners abstracts the task of information extraction from text as a function that maps every document (a string) into a relation over the document's spans (intervals identified by their start and end indices). For instance, the regular spanners are the closure under the Relational Algebra (RA) of the regular expressions with capture variables, and the expressive power of the regular spanners is precisely captured by the class of VSet-automata -- a restricted class of transducers that mark the endpoints of selected spans. In this work, we embark on the investigation of document spanners that can annotate extractions with auxiliary information such as confidence, support, and confidentiality measures. To this end, we adopt the abstraction of provenance semirings by Green et al., where tuples of a relation are annotated with the elements of a commutative semiring, and where the annotation propagates through the positive RA operators via the semiring operators. Hence, the proposed spanner extension, referred to as an annotator, maps every string into an annotated relation over the spans. As a specific instantiation, we explore weighted VSet-automata that, similarly to weighted automata and transducers, attach semiring elements to transitions. We investigate key aspects of expressiveness, such as the closure under the positive RA, and key aspects of computational complexity, such as the enumeration of annotated answers and their ranked enumeration in the case of ordered semirings. For a number of these problems, fundamental properties of the underlying semiring, such as positivity, are crucial for establishing tractability.

Distinguishing Biological Trends from Adaptation

Models of evolution commonly decompose change into deterministic and stochastic components. Different models, however, produce different concepts of signal and noise. Excursion tests correct for two distinct types of noise, sampling error and phenotypic drift, resulting in two distinct types of signal or trend. A minimal trend is a signal of historical change after correcting for sampling error; a directed trend is a model dependent measure of confidence that selection has occurred (correcting for both sampling error and drift). The history of random walks and excursion tests in paleobiology highlights the conceptual middle ground between historical pattern and underlying processes. In paleobiology, both minimal and directed trends reflect a causally agnostic, trait-level product of evolutionary and environmental processes. Directed trends provide evidence for directional causes, but the identity and relative strength of those causes requires a deeper understanding. Thus, the magnitude of minimal and directed trends should not be interpreted as the strength of selection for a specified trait. They can be a useful stepping stone in that direction when paired with a map of the middle ground and agreement about how genetic, macroevolutionary, and environmental processes contribute to evolutionary change.

CellPhy: accurate and fast probabilistic inference of single-cell phylogenies from scDNA-seq data

We introduce CellPhy, a maximum likelihood framework for inferring phylogenetic trees from somatic single-cell single-nucleotide variants. CellPhy leverages a finite-site Markov genotype model with 16 diploid states and considers amplification error and allelic dropout. We implement CellPhy into RAxML-NG, a widely used phylogenetic inference package that provides statistical confidence measurements and scales well on large datasets with hundreds or thousands of cells. Comprehensive simulations suggest that CellPhy is more robust to single-cell genomics errors and outperforms state-of-the-art methods under realistic scenarios, both in accuracy and speed. CellPhy is freely available at https://github.com/amkozlov/cellphy.

Teaching brief motivational interviewing to medical students using a pedagogical framework

ObjectiveMedical schools are charged with assisting medical students to acquire the confidence, knowledge and skills for behavior change conversations in primary healthcare. The present study evaluated teaching brief motivational interviewing (MI) to pre-clinical medical students. MethodsForty-six students participated in an educational intervention premised on the Learn, See, Practice, Prove, Do, Maintain pedagogical framework, comprising 2 × 2-h lectures, a 2-h role-play triad session, and 3 × 2-h small group simulated patient encounters supported by scaffolding strategies. Measures of brief MI knowledge (MI Knowledge and Attitudes Test & Multiple-Choice Knowledge Test) and confidence (MI Confidence Scale) were taken at baseline, post-training, and 3-month follow-up, and skills (Behavior Change Counseling Index) were assessed at three intervals during simulated patient encounters. ResultsStudents who received brief MI training improved in knowledge and confidence from baseline to post-training and gains remained at 3-months. Brief MI skills improved across the simulation sessions. ConclusionPre-clinical medical students can attain knowledge, confidence and skills in brief MI after participation in a short intervention and improvements are sustainable. Practice implicationsOur results support the use of an evidence-based pedagogical framework for teaching brief MI in pre-clinical years of medical curricula and our scaffolding strategy affords promise.

Neural Models for Measuring Confidence on Interactive Machine Translation Systems

Reducing the human effort performed with the use of interactive-predictive neural machine translation (IPNMT) systems is one of the main goals in this sub-field of machine translation (MT). Prior works have focused on changing the human–machine interaction method and simplifying the feedback performed. Applying confidence measures (CM) to an IPNMT system helps decrease the number of words that the user has to check through the translation session, reducing the human effort needed, although this supposes losing a few points in the quality of the translations. The effort reduction comes from decreasing the number of words that the translator has to review—it only has to check the ones with a score lower than the threshold set. In this paper, we studied the performance of four confidence measures based on the most used metrics on MT. We trained four recurrent neural network (RNN) models to approximate the scores from the metrics: Bleu, Meteor, Chr-f, and TER. In the experiments, we simulated the user interaction with the system to obtain and compare the quality of the translations generated with the effort reduction. We also compare the performance of the four models between them to see which of them obtains the best results. The results achieved showed a reduction of 48% with a Bleu score of 70 points—a significant effort reduction to translations almost perfect.

Web-Based Problem-solving Training With and Without Peer Support in Veterans With Unmet Mental Health Needs: Pilot Study of Feasibility, User Acceptability, and Participant Engagement.

BackgroundeHealth tools have the potential to meet the mental health needs of individuals who experience barriers to accessing in-person treatment. However, most users have less than optimal engagement with eHealth tools. Coaching from peer specialists may increase their engagement with eHealth.ObjectiveThis pilot study aims to test the feasibility and acceptability of a novel, completely automated web-based system to recruit, screen, enroll, assess, randomize, and then deliver an intervention to a national sample of military veterans with unmet mental health needs; investigate whether phone-based peer support increases the use of web-based problem-solving training compared with self-directed use; and generate hypotheses about potential mechanisms of action for problem-solving and peer support for future full-scale research.MethodsVeterans (N=81) with unmet mental health needs were recruited via social media advertising and enrolled and randomized to the self-directed use of a web-based problem-solving training called Moving Forward (28/81, 35%), peer-supported Moving Forward (27/81, 33%), or waitlist control (26/81, 32%). The objective use of Moving Forward was measured with the number of log-ins. Participants completed pre- and poststudy measures of mental health symptoms and problem-solving confidence. Satisfaction was also assessed post treatment.ResultsAutomated recruitment, enrollment, and initial assessment methods were feasible and resulted in a diverse sample of veterans with unmet mental health needs from 38 states. Automated follow-up methods resulted in 46% (37/81) of participants completing follow-up assessments. Peer support was delivered with high fidelity and was associated with favorable participant satisfaction. Participants randomized to receive peer support had significantly more Moving Forward log-ins than those of self-directed Moving Forward participants, and those who received peer support had a greater decrease in depression. Problem-solving confidence was associated with greater Moving Forward use and improvements in mental health symptoms among participants both with and without peer support.ConclusionsEnrolling and assessing individuals in eHealth studies without human contact is feasible; however, different methods or designs are necessary to achieve acceptable participant engagement and follow-up rates. Peer support shows potential for increasing engagement in web-based interventions and reducing symptoms. Future research should investigate when and for whom peer support for eHealth is helpful. Problem-solving confidence should be further investigated as a mechanism of action for web-based problem-solving training.Trial RegistrationClinicalTrials.gov NCT03555435; http://clinicaltrials.gov/ct2/show/NCT03555435

Speech Emotion Recognition Based on Self-Attention Weight Correction for Acoustic and Text Features

Speech emotion recognition (SER) is essential for understanding a speaker’s intention. Recently, some groups have attempted to improve SER performance using a bidirectional long short-term memory (BLSTM) to extract features from speech sequences and a self-attention mechanism to focus on the important parts of the speech sequences. SER also benefits from combining the information in speech with text, which can be accomplished automatically using an automatic speech recognizer (ASR), further improving its performance. However, ASR performance deteriorates in the presence of emotion in speech. Although there is a method to improve ASR performance in the presence of emotional speech, it requires the fine-tuning of ASR, which has a high computational cost and leads to the loss of cues important for determining the presence of emotion in speech segments, which can be helpful in SER. To solve these problems, we propose a BLSTM-and-self-attention-based SER method using self-attention weight correction (SAWC) with confidence measures. This method is applied to acoustic and text feature extractors in SER to adjust the importance weights of speech segments and words with a high possibility of ASR error. Our proposed SAWC reduces the importance of words with speech recognition error in the text feature while emphasizing the importance of speech segments containing these words in acoustic features. Our experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset reveal that our proposed method achieves a weighted average accuracy of 76.6%, outperforming other state-of-the-art methods. Furthermore, we investigated the behavior of our proposed SAWC in each of the feature extractors.

Efficient non-contrast enhanced 3D Cartesian cardiovascular magnetic resonance angiography of the thoracic aorta in 3\xa0min

BackgroundThe application of cardiovascular magnetic resonance angiography (CMRA) for the assessment of thoracic aortic disease is often associated with prolonged and unpredictable acquisition times and residual motion artefacts. To overcome these limitations, we have integrated undersampled acquisition with image-based navigators and inline non-rigid motion correction to enable a free-breathing, contrast-free Cartesian CMRA framework for the visualization of the thoracic aorta in a short and predictable scan of 3 min.Methods35 patients with thoracic aortic disease (36 ± 13y, 14 female) were prospectively enrolled in this single-center study. The proposed 3D T2-prepared balanced steady state free precession (bSSFP) sequence with image-based navigator (iNAV) was compared to the clinical 3D T2-prepared bSSFP with diaphragmatic-navigator gating (dNAV), in terms of image acquisition time. Three cardiologists blinded to iNAV vs. dNAV acquisition, recorded image quality scores across four aortic segments and their overall diagnostic confidence. Contrast ratio (CR) and relative standard deviation (RSD) of signal intensity (SI) in the corresponding segments were estimated. Co-axial aortic dimensions in six landmarks were measured by two readers to evaluate the agreement between the two methods, along with inter-observer and intra-observer agreement. Kolmogorov–Smirnov test, Mann–Whitney U (MWU), Bland–Altman analysis (BAA), intraclass correlation coefficient (ICC) were used for statistical analysis.ResultsThe scan time for the iNAV-based approach was significantly shorter (3.1 ± 0.5 min vs. 12.0 ± 3.0 min for dNAV, P = 0.005). Reconstruction was performed inline in 3.0 ± 0.3 min. Diagnostic confidence was similar for the proposed iNAV versus dNAV for all three reviewers (Reviewer 1: 3.9 ± 0.3 vs. 3.8 ± 0.4, P = 0.7; Reviewer 2: 4.0 ± 0.2 vs. 3.9 ± 0.3, P = 0.4; Reviewer 3: 3.8 ± 0.4 vs. 3.7 ± 0.6, P = 0.3). The proposed method yielded higher image quality scores in terms of artefacts from respiratory motion, and non-diagnostic images due to signal inhomogeneity were observed less frequently. While the dNAV approach outperformed the iNAV method in the CR assessment, the iNAV sequence showed improved signal homogeneity along the entire thoracic aorta [RSD SI 5.1 (4.4, 6.5) vs. 6.5 (4.6, 8.6), P = 0.002]. BAA showed a mean difference of < 0.05 cm across the 6 landmarks between the two datasets. ICC showed excellent inter- and intra-observer reproducibility.ConclusionsThoracic aortic iNAV-based CMRA with fast acquisition (~ 3 min) and inline reconstruction (3 min) is proposed, resulting in high diagnostic confidence and reproducible aortic measurements.

Localizing Basestations From End-User Timing Advance Measurements

Although mobile communication has become ubiquitous in our modern society, operators typically treat the underlying networking infrastructure in a secretive manner. However, detailed topology information is a key enabler for operator benchmarking and can serve as ground-truth data for system-level simulations or user equipment localization schemes. Still, existing approaches for base station localization that are either based on received signal strength or on the spatial distribution of measurements are not accurate enough for such use cases. Accordingly, we propose a localization scheme that operates on end-user measurements of the 4G &#x0026; 5G timing advance parameter, which acts as a quantized distance measure between the user equipment and the base station. By directly incorporating GPS noise, multipath propagation, and quantization into our stochastic system model, we obtain an estimator that offers a reliable measure of confidence and requires only the configuration of two hyperparameters with a dedicated physical interpretation. We evaluate our approach using a set of drive-test measurements consisting of 190 LTE eNodeBs with ground-truth locations confirmed by an Austrian mobile network operator. Our selective estimator can either operate without prior knowledge, resulting in mean distance errors of below 100 m, or in a classification setup, where it correctly identifies up to 95&#x0025; of eNodeBs from a set of candidate cell tower locations. To allow for reproducibility, we make our dataset and a reference implementation publicly available.

Continuous Measures of Confidence in Direction of Environmental Trends at Site and Other Spatial Scales

Continuous Measures of Confidence in Direction of Environmental Trends at Site and Other Spatial Scales

Increasing Context for Estimating Confidence Scores in Automatic Speech Recognition

Accurate confidence measures for predictions from machine learning techniques play a critical role in the deployment and training of many speech and language processing applications. For example, confidence scores are important when making use of automatically generated transcriptions in training automatic speech recognition (ASR) systems, as well as down-stream applications, such as information retrieval and conversational assistants. Previous work on improving confidence scores for these systems has focused on two main directions: designing features correlated with improved confidence prediction; and employing sequence models to account for the importance of contextual information. Few studies, however, have explored incorporating contextual information more broadly, such as from the future, in addition to the past, or making use of alternative multiple hypotheses in addition to the most likely one. This article introduces two general approaches for encapsulating contextual information from lattices. Experimental results illustrating the importance of increasing contextual information for estimating confidence scores are presented on a range of limited resource languages where word error rates range between 30% and 60%. The results show that the novel approaches provide significant gains in the accuracy of confidence estimation.

Ensemble of Machine Learning Fusion Models for Breast Cancer Detection Based on the Regression Model

Breast cancer is one of the deadliest cancers among women worldwide and one of the main causes of mortality for women in the United States. Breast cancer can be detected earlier and with more accuracy, extending life expectancy at a lower cost. To do this, the efficiency and precision of early breast cancer detection can be increased by evaluating the large data that is currently available utilizing technologies like machine learning fusion-based decision support systems. In this paper, we investigate the prediction performance of various regression models and a decision support system based on these models that provided the predicted category along with a prediction confidence measure. The various machine learning (ML) algorithms applied include decision tree regressor, MLP regressor, SVR, random forest regressor, and K-Neighbors regressor. The models are enhanced by average ensemble and ensemble using K-Neighbors regressor. We used the Breast Cancer Wisconsin Dataset from Wisconsin Prognostic Breast Cancer (WPBC) with 569 digitized images of a fine needle aspirate (FNA) of breast mass and 10 real-valued feature information. Among all five machine learning methods, K-Neighbors regressor had the best performance and ensemble using K-Neighbors regressor gave the best accuracy. The results show that there is a decrease in RMSE, MAE, MBE, R, R2, RRMSE, NSE, and WI when compared to the traditional methods.