Confidence Measure Research Articles

Supporting and Establishing Gender and Sexuality Alliances in Chicago Public Schools.

School nurses are uniquely positioned to support lesbian, gay, bisexual, transgender, and questioning/queer (LGBTQ) students. One approach is involvement in Gender and Sexuality Alliances (GSAs). GSAs have been found to improve outcomes for LGBTQ students. The purposes of this quality improvement project were to build infrastructure for GSAs by implementing a GSA Support Plan including a digital resource binder, GSA advisor trainings, and establishment of an Advisor Leadership Committee and increase the number of schools with a GSA. The binder was distributed electronically to 95 advisors. Forty-nine participants attending virtual trainings. A statistically significant increase in all measures of participant confidence occurred between the pre- and post-training surveys. The Advisor Leadership Committee was established. This paper adds to the literature by showing that support for GSAs is important to aiding LGBTQ students. Further, authors suggest ways in which school nurses can harness their expertise to be involved with GSAs.

Dissociating the pro‐cognitive effects of donepezil and an M1 PAM

AbstractBackgroundDonepezil, an FDA approved drug for dementia in Alzheimer’s disease, increases the extracellular concentration of acetylcholine non‐selectively stimulating cholinergic receptors indiscriminately leading to dose‐limiting side effects. Muscarinic M1 sub‐receptor specific positive allosteric modulators (PAMs) have shown promise in addressing symptoms in Alzheimer’s disease, Schizophrenia and more1. M1 PAMs have little to no agonistic properties, avoiding peripheral side effects, and are mostly centrally located allowing for the modulation of endogenous signaling of the brain2.MethodWe tested non‐human primates in two tasks measuring cognitive flexibility and attentional control3 at varying doses of donepezil and an M1 PAM (VU0453595). Attentional control was measured with a visual search task requiring animals to search for a learned target object among distractors of variable number and similarity to the target object. Cognitive flexibility was measured with a feature‐reward learning task that required animals to learn the value of objects of varying feature complexity by trial and error. Flexibility was indexed as the speed of learning, the degree of latent inhibition and the amount of target perseveration after switching the rewarded feature.ResultDonepezil enhanced attention control at a high dose (0.3 mg/kg) but enhance flexibility only moderately at a lower dose (0.06 mg/kg). In contrast, VU0453595 did not modulate attention, but at 0.1 mg/kg enhanced flexible learning, reduced target perseverations, and increased reaction time measures of decision confidence. At the dose with the most pronounced cognitive benefits donepezil resulted in gastrointestinal side effects, while VU0453595 did not cause side effects.ConclusionThese results suggest that donepezil engages the brain networks underlying cognitive flexibility and interference control at different doses4, while the M1 receptor plays a major role in facilitating flexible behavioral adaptation but with no role in controlling interference. Such receptor specific results can be used to better curate to patient‐specific needs in order to address the most debilitating symptoms with the highest efficacy.1. Melancon, B. J. et al. (2013) Drug Discovery Today https://doi.org/10.1016/j.drudis.2013.09.005.2. Ghoshal, A. et al. (2016) Neuropsychopharmacology doi:10.1038npp.2015.189 3. Womelsdorf, T. et al. (2021) Frontiers in Behavioral Neuroscience https://doi.org/10.3389/fnbeh.2021.721069 4. Hassani, S.A. et al. (2021) Biological Psychiatry Global Open Science https://doi.org/10.1016/j.bpsgos.2021.11.012.

Pairing Bayesian Methods and Systems Theory to Enable Test and Evaluation of Learning‐Based Systems

ABSTRACTModern engineered systems, and learning‐based systems, in particular, provide unprecedented complexity that requires advancement in our methods to achieve confidence in mission success through test and evaluation (T&E). We define learning‐based systems as engineered systems that incorporate a learning algorithm (artificial intelligence) component of the overall system. A part of the unparalleled complexity is the rate at which learning‐based systems change over traditional engineered systems. Where traditional systems are expected to steadily decline (change) in performance due to time (aging), learning‐based systems undergo a constant change which must be better understood to achieve high confidence in mission success. To this end, we propose pairing Bayesian methods with systems theory to quantify changes in operational conditions, changes in adversarial actions, resultant changes in the learning‐based system structure, and resultant confidence measures in mission success. We provide insights, in this article, into our overall goal and progress toward developing a framework for evaluation through an understanding of equivalence of testing.

Consensus automatic speech recognition (CASR) in the California Cognitive Assessment Battery (CCAB).

Recent reports have investigated the use of automatic speech recognition (ASR) to analyze and score verbal responses in cognitive tests. ASR scoring is objective, permits the efficient computerized administration of verbal tests, and generates timestamps that enable the detailed temporal analysis of responses. However, ASR transcription accuracy varies by engine, task, and participant, and ASR can incorrectly score responses from participants with atypical speech patterns. Here we describe the speech-transcription pipeline of the California Cognitive Assessment Battery (CCAB), which incorporates consensus ASR (CASR) to produce more accurate transcripts than possible with any single ASR engine. We also developed a Transcript Review Tool (TRT) which facilitates the manual correction of mis-transcribed words in problem subjects. Figure 1 shows the CCAB speech transcription pipeline. Realtime ASR transcriptions are obtained along with the transcriptions of the digital recordings of responses using six cloud-based ASR engines (e.g., Google, etc.). Individual transcripts are then combined to produce a "consensus" transcript, and a transcription confidence measure based primarily on the agreement between ASR engines (Figure 2). If needed, "consensus" transcripts can be manually corrected using the Transcript Review Tool which enables the review of all words or just those words below a predefined CASR confidence threshold (Figure 3). ASR transcriptions were obtained from 442 healthy adults (mean age = 65.1 ±14.4) who each underwent three days of cognitive testing that included 25 verbal tests. In all, approximately 276 hours of speech were transcribed. Preliminary analyses show that CASR transcription accuracy surpassed 99% for tests with limited response sets (e.g., digit span, verbal list learning, face-name binding, etc.) and exceeded 95% for discursive speech tests (e.g., picture description and logical memory). CASR transcription is more accurate than that of any single ASR engine. When combined with the TRT, "consensus" ASR can produce error-free, timestamped transcripts that enable the detailed analysis of verbal responses from older individuals at risk of cognitive decline.

Is My Driver Observation Model Overconfident? Input-Guided Calibration Networks for Reliable and Interpretable Confidence Estimates

Driver observation models are rarely deployed under perfect conditions. In practice, illumination, camera placement and type differ from the ones present during training and unforeseen behaviours may occur at any time. While observing the human behind the steering wheel leads to more intuitive human-vehicle-interaction and safer driving, it requires recognition algorithms which do not only predict the correct driver state, but also determine their prediction quality through realistic and interpretable confidence measures. Reliable uncertainty estimates are crucial for building trust and are a serious obstacle for deploying activity recognition networks in real driving systems. In this work, we for the first time examine how well the confidence values of modern driver observation models indeed match the probability of the correct outcome and show that raw neural network-based approaches tend to significantly overestimate their prediction quality. To correct this misalignment between the confidence values and the actual uncertainty, we consider two strategies. First, we enhance two activity recognition models often used for driver observation with temperature scaling – an off-the-shelf method for confidence calibration in image classification. Then, we introduce Calibrated Action Recognition with Input Guidance (CARING) – a novel approach leveraging an additional neural network to learn scaling the confidences depending on the video representation. Extensive experiments on the Drive&Act dataset demonstrate that both strategies drastically improve the quality of model confidences, while our CARING model outperforms both, the original architectures and their temperature scaling enhancement, leading to best uncertainty estimates.

Sperm-cell DNA fragmentation prediction using label-free quantitative phase imaging and deep learning.

In intracytoplasmic sperm injection (ICSI), a single sperm cell is selected and injected into an egg. The quality of the chosen sperm and specifically its DNA fragmentation have a significant effect on the fertilization success rate. However, there is no method today to measure the DNA fragmentation of live and unstained cells during ICSI. We present a new method to predict the DNA fragmentation of sperm cells using multi-layer stain-free imaging data, including quantitative phase imaging, and lightweight deep learning architectures. The DNA fragmentation ground truth is achieved by staining the cells with acridine orange and imaging them via fluorescence microscopy. Our prediction model is based on the MobileNet convolutional neural network architecture combined with confidence measurement determined by distances between vectors in the latent space. Our results show that the mean absolute error for cells with high prediction confidence is 0.05 and the 90th percentile mean absolute error is 0.1, where the range of DNA fragmentation score is [0,1]. In the future, this model may be applied to improve cell selection by embryologists during ICSI.

Impact and evaluation of an online culinary nutrition course for health, education and industry professionals to promote vegetable knowledge and consumption.

Poor diet, including inadequate vegetable intake, is a leading risk factor for noncommunicable disease. Culinary and nutrition education provided to trainee and practising health and education professionals is an emerging strategy to promote improved dietary intake, including vegetable consumption. We evaluated the impact and feasibility of an online culinary medicine and nutrition (CM/CN) short course for health, education and vegetable industry professionals. The course aimed to improve participants' skills and confidence to prepare vegetables, knowledge of evidence-based nutrition information and recommendations for improving vegetable consumption and diet quality. A pre-post study consisting of two separate groups participating in two course rounds recruited practising professionals (n = 30) working in health; community, adult and/or culinary education; and the vegetable industry. Evaluation assessed diet quality, vegetable consumption barriers, cooking and food skill confidence, nutrition knowledge and process measures. Seventeen participants (68%) completed the programme. Pre- to postintervention statistically significant increases in vegetables (M 1.3, SD 2.2), fruit (M 1.6, SD 3.1), and breads and cereal (M 1.1, SD 1.7) intakes were observed. Statistically significant increases and large effect sizes for mean food skill confidence scores (M 8.9, SD 15.4, Cohen's d 0.56) and nutrition knowledge scores (M 6.2, SD 15.4, Cohen's d 0.83) were also observed pre- to postintervention. The short online course was feasible and improved diet quality, food skill confidence and nutrition knowledge. Online CM/CN education for practising professionals represents a promising area of research. Future research involving a larger study sample and a more rigorous study design such as a randomised control trial is warranted.

Machine assistance for credit approval? Random wheel can recommend and explain

Approval of credit application is one of the censorious business decisions the bankers usually take on a regular basis. The growing number of new credit applications and the enormous amount of outstanding credit card bills during the recent pandemic make this even more challenging nowadays. Some of the previous studies suggest automating the credit approval process to mitigate this challenge. However, we believe that trustworthy machine assistance could be more beneficial as the effectiveness of complete automation may depend on the richness of the training dataset and model efficiency. A novel classifier, named random wheel, interestingly provides a more interpretable output. In this work, we have proposed an enhanced version of random wheel to facilitate a trustworthy recommendation for credit approval process. It not only produces more accurate and precise recommendations but also provides an interpretable confidence measure. Besides, it explains the machine recommendation for each credit application as well. The availability of recommendation confidence and explanation could bring more trust in the machine provided intelligence which may, in turn, enhance the efficiency of the credit approval process.

‘It’s not so scary anymore. It’s actually exhilarating’: A proof-of-concept study using virtual reality technology for music performance training under pressure

Extended lockdowns associated with the COVID-19 pandemic severely ruptured the capacity of performing artists to connect with peers and colleagues in professional and educational contexts. In this study we test a protocol for the use of immersive performance technologies in tertiary institutions to provide developing and early career musicians to connect with pedagogues and access safe, realistic spaces within which to practice performing under varying degrees of pressure. We investigated the affordances of a VR environment to trigger performance anxiety, and the effectiveness of a synchronous digitally mediated environment for tertiary educators to train a key performance psychology skill. Heart rate, self-reported anxiety and confidence measurements were taken over four levels of situational stress and performance demands. Results revealed that the technology enabled a pre-performance routine to be effectively taught online by an instructor to a musician wearing a VR headset. Notably, this was achieved by both participants being in separate locations without detriment to the teacher-student relationship. This study provides encouraging insight into the capacity for immersive technologies to help students effectively manage the stresses of live performance in both virtual and real worlds.

Using Pixel Counts to Measure Accuracy and Confidence in Representations of Mental Maps

Mental imagery and cognitive maps are difficult to study because these are so subjective and not easily observable. This exploratory study uses pixel counts to measure accuracy and confidence related to mental imagery. In this study 35 fifth grade students received an outline map of the continental United States on a standard-sized sheet of paper and asked to write state abbreviations as large as possible being 100% confident that the abbreviations would be within the boundary of the respective states. The response sheets were scanned. Adobe Photoshop was used to calculate pixel counts of the area of the abbreviations within and outside of the respective state boundaries. The ratio of In and Out pixel counts provided a measurement of Accuracy, while a ratio of In and Total State pixel counts provided a measurement of Confidence. More abbreviations were attempted for US states that had one or more sides present on the US outline map. The girls showed greater accuracy and higher confidence across all conditions. Similarly, there was a linear relationship between the number of reference sides and the confidence outcomes. The results provide proof of concept that pixel count measurements provide value to measuring mental imagery and spatial cognition. Keywords: mental imagery, cognitive maps, spatial cognition, accuracy, confidence index

Region-based evidential deep learning to quantify uncertainty and improve robustness of brain tumor segmentation

Despite recent advances in the accuracy of brain tumor segmentation, the results still suffer from low reliability and robustness. Uncertainty estimation is an efficient solution to this problem, as it provides a measure of confidence in the segmentation results. The current uncertainty estimation methods based on quantile regression, Bayesian neural network, ensemble, and Monte Carlo dropout are limited by their high computational cost and inconsistency. In order to overcome these challenges, Evidential Deep Learning (EDL) was developed in recent work but primarily for natural image classification and showed inferior segmentation results. In this paper, we proposed a region-based EDL segmentation framework that can generate reliable uncertainty maps and accurate segmentation results, which is robust to noise and image corruption. We used the Theory of Evidence to interpret the output of a neural network as evidence values gathered from input features. Following Subjective Logic, evidence was parameterized as a Dirichlet distribution, and predicted probabilities were treated as subjective opinions. To evaluate the performance of our model on segmentation and uncertainty estimation, we conducted quantitative and qualitative experiments on the BraTS 2020 dataset. The results demonstrated the top performance of the proposed method in quantifying segmentation uncertainty and robustly segmenting tumors. Furthermore, our proposed new framework maintained the advantages of low computational cost and easy implementation and showed the potential for clinical application.

Passing the buck!—how credible are self-reported measures of confidence in public institutions?

Abstract Measures of confidence in public institutions are self-reported and therefore, susceptible to heterogeneity arising out of individual perceptions. In this article, I explore the possibility of individuals reporting lower levels of confidence in public institutions depending on their personal economic conditions, for which these institutions cannot be solely held responsible. Using nationally representative Indian data, I exploit exogenous sources of indebtedness in an instrumental variables regression approach to identify the causal effects of personal economic hardships on confidence in public institutions. The identification strategy relies on exogenous variation in the measure of indebtedness generated by a large monetary loss arising out of natural calamities such as accident, fire, or drought. I find that estimated coefficients from two-stage least square regressions are negative and statistically significant suggesting that an increase in indebtedness is associated with an erosion in confidence measures.

Robust methylation-based classification of brain tumours using nanopore sequencing.

DNA methylation-based classification of cancer provides a comprehensive molecular approach to diagnose tumours. In fact, DNA methylation profiling of human brain tumours already profoundly impacts clinical neuro-oncology. However, current implementation using hybridisation microarrays is time consuming and costly. We recently reported on shallow nanopore whole-genome sequencing for rapid and cost-effective generation of genome-wide 5-methylcytosine profiles as input to supervised classification. Here, we demonstrate that this approach allows us to discriminate a wide spectrum of primary brain tumours. Using public reference data of 82 distinct tumour entities, we performed nanopore genome sequencing on 382 tissue samples covering 46 brain tumour (sub)types. Using bootstrap sampling in a cohort of 55 cases, we found that a minimum set of 1000 random CpG features is sufficient for high-confidence classification by ad hoc random forests. We implemented score recalibration as a confidence measure for interpretation in a clinical context and empirically determined a platform-specific threshold in a randomly sampled discovery cohort (N = 185). Applying this cut-off to an independent validation series (n = 184) yielded 148 classifiable cases (sensitivity 80.4%) and demonstrated 100% specificity. Cross-lab validation demonstrated robustness with concordant results across four laboratories in 10/11 (90.9%) cases. In a prospective benchmarking (N = 15), the median time to results was 21.1h. In conclusion, nanopore sequencing allows robust and rapid methylation-based classification across the full spectrum of brain tumours. Platform-specific confidence scores facilitate clinical implementation for which prospective evaluation is warranted and ongoing.

Reassurance seeking test behavior and exam performance

The current studies examine the roles of test anxiety safety behaviors and memory confidence in test performance. In Study 1, 154 college students completed self-report measures, including measures of low memory confidence and reassurance seeking test behavior, and provided access to standardized test scores and information about their academic performance in high school. We found a significant indirect association between low memory confidence and standardized test scores through reassurance seeking test behavior. Reassurance seeking test behavior fully mediated the relationship of low memory confidence with standardized test scores. In Study 2, 310 college students completed self-report measures, including low memory confidence and reassurance seeking test behaviors, and provided access to ACT scores, SAT scores, information regarding their academic performance in high school, and introductory psychology course exam scores. Study 2 found reassurance seeking test behaviors are associated with poorer performance on standardized exams and prospectively predicted poorer performance on course exams. Study 2 failed to replicate Study 1's findings and found only marginal evidence that reassurance seeking test behaviors mediate the association between low memory confidence and performance on exams. Further experimental work to manipulate test anxiety safety behaviors is needed.

Semi-supervised Multi-Source Transfer Learning for Motor Imagery Recognition

In the field of motor imagery (MI) recognition, poor generalization and low recognition performance are major challenges. An MI recognition method based on semi-supervised learning and multi-source transfer learning is proposed. In this approach, samples are transferred from some source domains to the target domain using the multi-source transfer learning method. The source domains selection method based on distribution similarity is designed to select source domains with similar distribution to the target domain, and samples with high information entropy are selected from these source domains for transfer. In this regard, we propose a semi-supervised learning labeling method for labeling the unlabeled samples of the target domain, which utilizes the labeling information from a few labeled samples without increasing the labeling cost. The sample confidence measurement method and the dynamic adjustment mechanism are proposed to ensure labeling accuracy and minimize the influence of mislabeled samples. A fusion classification model can identify the new sample in the target domain. As a measure of the effectiveness of the proposed method, four types of MI from the BCI Competition IV dataset 2A were used to evaluate the recognition ability, and the outcomes confirmed an excellent recognition performance as well as a superior training efficiency when compared with the currently used methods.

Mood and implicit confidence independently fluctuate at different time scales.

Mood is an important ingredient of decision-making. Human beings are immersed into a sea of ​​emotions where episodes of high mood alternate with episodes of low mood. While changes in mood are well characterized, little is known about how these fluctuations interact with metacognition, and in particular with confidence about our decisions. We evaluated how implicit measurements of confidence are related with mood states of human participants through two online longitudinal experiments involving mood self-reports and visual discrimination decision-making tasks. Implicit confidence was assessed on each session by monitoring the proportion of opt-out trials when an opt-out option was available, as well as the median reaction time on standard correct trials as a secondary proxy of confidence. We first report a strong coupling between mood, stress, food enjoyment, and quality of sleep reported by participants in the same session. Second, we confirmed that the proportion of opt-out responses as well as reaction times in non-opt-out trials provided reliable indices of confidence in each session. We introduce a normative measure of overconfidence based on the pattern of opt-out selection and the signal-detection-theory framework. Finally and crucially, we found that mood, sleep quality, food enjoyment, and stress level are not consistently coupled with these implicit confidence markers, but rather they fluctuate at different time scales: mood-related states display faster fluctuations (over one day or half-a-day) than confidence level (two-and-a-half days). Therefore, our findings suggest that spontaneous fluctuations of mood and confidence in decision making are independent in the healthy adult population.

Eye movements reveal that low confidence precedes deliberation.

Contemporary dual-process models of reasoning maintain that there are two types of thinking -intuitive and deliberative -and that low confidence often leads to deliberation. Previous studies examining the confidence -deliberation relationship have been limited by (1) issues of endogeneity and between-subject comparisons, which we address in this study through debias training and (2) measures of confidence that are taken relatively late in the reasoning process, which we address by measuring confidence via real-time eye-tracking. Self-reported and eye-tracked confidence were both negatively related to deliberative thinking. This finding provides new evidence of the timecourse of the confidence -deliberation relationship and reveals that lowered confidence precedes deliberation.

Characterization of merging black holes with two precessing spins

Spin precession in merging black-hole binaries is a treasure trove for both astrophysics and fundamental physics. There are now well-established strategies to infer from gravitational-wave data whether at least one of the two black holes is precessing. In this paper we tackle the next-in-line target, namely the statistical assessment that the observed system has two precessing spins. We find that the recently developed generalization of the effective precession spin parameter $\chi_\mathrm{p}$ is a well-suited estimator to this task. With this estimator, the occurrence of two precessing spins is a necessary (though not sufficient) condition to obtain values $1<\chi_\mathrm{p}\leq 2$. Confident measurements of gravitational-wave sources with $\chi_\mathrm{p}$ values in this range can be taken as a conservative assessment that the binary presents two precessing spins. We investigate this argument using a large set of >100 software injections assuming anticipated LIGO/Virgo sensitivities for the upcoming fourth observing run, O4. Our results are very encouraging, suggesting that, if such binaries exist in nature and merge at a sufficient rate, current interferometers are likely to deliver the first confident detection of merging black holes with two precessing spins. We investigate prior effects and waveform systematics and, though these need to be better investigated, did not find any confident false-positive case among all the configurations we tested. Our assessment should thus be taken as conservative.

Shades of doubt: Measuring and classifying vaccination confidence in Europe

The anti-vaccination movement, vaccine hesitancy, and wavering vaccination confidence have increasingly become matters of public interest, in parallel with an increasing normalization of representations of vaccination as risky. In this study, we used data on vaccination beliefs and behaviors from two Eurobarometer surveys to classify attitudes towards vaccination and to discuss comparability, acquiescence, and other measurement issues. Through cluster analysis, we found that individuals in the European Union (EU27) can be classified into five opinion types, differentiating the poles (“vaccine-trusting” and “vaccine-distrusting”) from the “hesitant & free choice” cluster and from two relatively uncommitted clusters, the “agreeable” (or acquiescent) and the “fence-sitters.” Opinion configurations on vaccination were linked to the broader social structures of age, gender, and educational attainment, to experiences of adult vaccination, and trust in different information sources. We found that trust, distrust, and confusion about vaccination have permeated all social strata in EU countries. The pandemic years have amplified uncertainty concerning vaccine safety and its effectiveness. We also noticed a decrease of trust in the voices of mainstream medical experts during the pandemic period, from about 92 % in 2019 to 73 % in 2021, and a significant increase in people who declared that they “don’t know” whom to trust about vaccine information, ranging from 1 % to about 13 %. Measurements of vaccination confidence in Europe should control for acquiescence, through positively and negatively formulated items, and ensure comparability in time. We strongly recommend the inclusion of a battery of critical items in all future European Commission-funded surveys on vaccination to allow the monitoring of European public confidence in vaccination and in the relevant information sources, including trust in pharmaceutical companies; this will provide an avenue for re-establishing a broader confidence among citizens, health authorities, and specialists.

Active Learning Strategy for Surrogate-Based Quantile Estimation of Field Function

Uncertainty quantification is widely used in engineering domains to provide confidence measures on complex systems. It often requires to accurately estimate extreme statistics on computationally intensive black-box models. In case of spatially or temporally distributed model outputs, one valuable metric results in the estimation of extreme quantile of the output stochastic field. In this paper, a novel active learning surrogate-based method is proposed to determine the quantile of an unidimensional output stochastic process with a confidence measure. This allows to control the error on the estimation of a extreme quantile measure of a stochastic process. The proposed approach combines dimension reduction techniques, Gaussian process and an adaptive refinement strategy to enrich the surrogate model and control the accuracy of the quantile estimation. The proposed methodology is applied on an analytical test case and a realistic aerospace problem for which the estimation of a flight envelop is of prime importance for launch safety reasons in the space industry.