Recall Threshold Research Articles

Commonsense for Zero-Shot Natural Language Video Localization

Zero-shot Natural Language-Video Localization (NLVL) methods have exhibited promising results in training NLVL models exclusively with raw video data by dynamically generating video segments and pseudo-query annotations. However, existing pseudo-queries often lack grounding in the source video, resulting in unstructured and disjointed content. In this paper, we investigate the effectiveness of commonsense reasoning in zero-shot NLVL. Specifically, we present CORONET, a zero-shot NLVL framework that leverages commonsense to bridge the gap between videos and generated pseudo-queries via a commonsense enhancement module. CORONET employs Graph Convolution Networks (GCN) to encode commonsense information extracted from a knowledge graph, conditioned on the video, and cross-attention mechanisms to enhance the encoded video and pseudo-query representations prior to localization. Through empirical evaluations on two benchmark datasets, we demonstrate that CORONET surpasses both zero-shot and weakly supervised baselines, achieving improvements up to 32.13% across various recall thresholds and up to 6.33% in mIoU. These results underscore the significance of leveraging commonsense reasoning for zero-shot NLVL.

Performance of a Breast Cancer Detection AI Algorithm Using the Personal Performance in Mammographic Screening Scheme.

Background The Personal Performance in Mammographic Screening (PERFORMS) scheme is used to assess reader performance. Whether this scheme can assess the performance of artificial intelligence (AI) algorithms is unknown. Purpose To compare the performance of human readers and a commercially available AI algorithm interpreting PERFORMS test sets. Materials and Methods In this retrospective study, two PERFORMS test sets, each consisting of 60 challenging cases, were evaluated by human readers between May 2018 and March 2021 and were evaluated by an AI algorithm in 2022. AI considered each breast separately, assigning a suspicion of malignancy score to features detected. Performance was assessed using the highest score per breast. Performance metrics, including sensitivity, specificity, and area under the receiver operating characteristic curve (AUC), were calculated for AI and humans. The study was powered to detect a medium-sized effect (odds ratio, 3.5 or 0.29) for sensitivity. Results A total of 552 human readers interpreted both PERFORMS test sets, consisting of 161 normal breasts, 70 malignant breasts, and nine benign breasts. No difference was observed at the breast level between the AUC for AI and the AUC for human readers (0.93% and 0.88%, respectively; P = .15). When using the developer's suggested recall score threshold, no difference was observed for AI versus human reader sensitivity (84% and 90%, respectively; P = .34), but the specificity of AI was higher (89%) than that of the human readers (76%, P = .003). However, it was not possible to demonstrate equivalence due to the size of the test sets. When using recall thresholds to match mean human reader performance (90% sensitivity, 76% specificity), AI showed no differences inperformance, with a sensitivity of 91% (P =. 73) and a specificity of 77% (P = .85). Conclusion Diagnostic performance of AI was comparable with that of the average human reader when evaluating cases from two enriched test sets from the PERFORMS scheme. © RSNA, 2023 See also the editorial by Philpotts in this issue.

An Artificial Intelligence-based Mammography Screening Protocol for Breast Cancer: Outcome and Radiologist Workload.

Background Developments in artificial intelligence (AI) systems to assist radiologists in reading mammograms could improve breast cancer screening efficiency. Purpose To investigate whether an AI system could detect normal, moderate-risk, and suspicious mammograms in a screening sample to safely reduce radiologist workload and evaluate across Breast Imaging Reporting and Data System (BI-RADS) densities. Materials and Methods This retrospective simulation study analyzed mammographic examination data consecutively collected from January 2014 to December 2015 in the Danish Capital Region breast cancer screening program. All mammograms were scored from 0 to 10, representing the risk of malignancy, using an AI tool. During simulation, normal mammograms (score < 5) would be excluded from radiologist reading and suspicious mammograms (score > recall threshold [RT]) would be recalled. Two radiologists read the remaining mammograms. The RT was fitted using another independent cohort (same institution) by matching to the radiologist sensitivity. This protocol was further applied to each BI-RADS density. Screening outcomes were measured using the sensitivity, specificity, workload, and false-positive rate. The AI-based screening was tested for noninferiority sensitivity compared with radiologist screening using the Farrington-Manning test. Specificities were compared using the McNemar test. Results The study sample comprised 114 421 screenings for breast cancer in 114 421 women, resulting in 791 screen-detected, 327 interval, and 1473 long-term cancers and 2107 false-positive screenings. The mean age of the women was 59 years ± 6 (SD). The AI-based screening sensitivity was 69.7% (779 of 1118; 95% CI: 66.9, 72.4) and was noninferior (P = .02) to the radiologist screening sensitivity of 70.8% (791 of 1118; 95% CI: 68.0, 73.5). The AI-based screening specificity was 98.6% (111 725 of 113 303; 95% CI: 98.5, 98.7), which was higher (P < .001) than the radiologist specificity of 98.1% (111 196 of 113 303; 95% CI: 98.1, 98.2). The radiologist workload was reduced by 62.6% (71 585 of 114 421), and 25.1% (529 of 2107) of false-positive screenings were avoided. Screening results were consistent across BI-RADS densities, although not significantly so for sensitivity. Conclusion Artificial intelligence (AI)-based screening could detect normal, moderate-risk, and suspicious mammograms in a breast cancer screening program, which may reduce the radiologist workload. AI-based screening performed consistently across breast densities. © RSNA, 2022 Online supplemental material is available for this article.

Can artificial intelligence reduce the interval cancer rate in mammography screening?

ObjectivesTo investigate whether artificial intelligence (AI) can reduce interval cancer in mammography screening.Materials and methodsPreceding screening mammograms of 429 consecutive women diagnosed with interval cancer in Southern Sweden between 2013 and 2017 were analysed with a deep learning–based AI system. The system assigns a risk score from 1 to 10. Two experienced breast radiologists reviewed and classified the cases in consensus as true negative, minimal signs or false negative and assessed whether the AI system correctly localised the cancer. The potential reduction of interval cancer was calculated at different risk score thresholds corresponding to approximately 10%, 4% and 1% recall rates.ResultsA statistically significant correlation between interval cancer classification groups and AI risk score was observed (p < .0001). AI scored one in three (143/429) interval cancer with risk score 10, of which 67% (96/143) were either classified as minimal signs or false negative. Of these, 58% (83/143) were correctly located by AI, and could therefore potentially be detected at screening with the aid of AI, resulting in a 19.3% (95% CI 15.9–23.4) reduction of interval cancer. At 4% and 1% recall thresholds, the reduction of interval cancer was 11.2% (95% CI 8.5–14.5) and 4.7% (95% CI 3.0–7.1). The corresponding reduction of interval cancer with grave outcome (women who died or with stage IV disease) at risk score 10 was 23% (8/35; 95% CI 12–39).ConclusionThe use of AI in screen reading has the potential to reduce the rate of interval cancer without supplementary screening modalities.Key Points• Retrospective study showed that AI detected 19% of interval cancer at the preceding screening exam that in addition showed at least minimal signs of malignancy. Importantly, these were correctly localised by AI, thus obviating supplementary screening modalities.• AI could potentially reduce a proportion of particularly aggressive interval cancers.• There was a correlation between AI risk score and interval cancer classified as true negative, minimal signs or false negative.

Is focusing on unknown items while studying a beneficial long-term strategy?

The amount students learn is influenced by the decisions they make while they study. Prior research shows that when people decide what to study (and what not to), they almost universally prioritise information they do not know over information they do know. We found that focusing on unknown items was beneficial when memory was tested immediately after studying, but not when memory was tested after a delay of one or two weeks. These findings support a model in which studying unknown items causes many items to rise above a recall threshold, which results in a short-term advantage, but also leaves the items vulnerable to forgetting. Studying known items, by contrast, moves items farther above threshold, such that they remain above threshold for longer. These findings have implications for students and teachers who want to maximise long-term learning.

Sleep can reduce the testing effect: it enhances recall of restudied items but can leave recall of retrieved items unaffected.

The testing effect refers to the finding that retrieval practice in comparison to restudy of previously encoded contents can improve memory performance and reduce time-dependent forgetting. Naturally, long retention intervals include both wake and sleep delay, which can influence memory contents differently. In fact, sleep immediately after encoding can induce a mnemonic benefit, stabilizing and strengthening the encoded contents. We investigated in a series of 5 experiments whether sleep influences the testing effect. After initial study of categorized item material (Experiments 1, 2, and 4A), paired associates (Experiment 3), or educational text material (Experiment 4B), subjects were asked to restudy encoded contents or engage in active retrieval practice. A final recall test was conducted after a 12-hr delay that included diurnal wakefulness or nocturnal sleep. The results consistently showed typical testing effects after the wake delay. However, these testing effects were reduced or even eliminated after sleep, because sleep benefited recall of restudied items but left recall of retrieved items unaffected. The findings are consistent with the bifurcation model of the testing effect (Kornell, Bjork, & Garcia, 2011), according to which the distribution of memory strengths across items is shifted differentially by retrieving and restudying, with retrieval strengthening items to a much higher degree than restudy does. On the basis of this model, most of the retrieved items already fall above recall threshold in the absence of sleep, so additional sleep-induced strengthening may not improve recall of retrieved items any further.

Minimum duration of actigraphy-defined nocturnal awakenings necessary for morning recall

Healthy adults awaken between each sleep cycle approximately 5 times each night but generally do not remember all of these awakenings in the morning. A rule of thumb has arisen in the sleep field that approximately 5 min of continuous wakefulness are required to form a memory for an awakening. However, few studies have examined memory for these sleep-wake transitions and none have done so in the home, while participants follow their normal routine. Self-report and actigraphy were used in the participant's home environment to determine the minimum duration of an awakening necessary for morning recall for each of the 39 healthy adults. Recall thresholds ranged from 30 to 600 s with a mean of 259 s (4 min 19 s) and were negatively associated with sleep efficiency but not significantly associated with total sleep time, age, income, or education. There also was a sex by cohabitation interaction, with single men having lower thresholds than single women and cohabiting participants, which was explained by higher sleep efficiency in noncohabitating men. Large individual differences suggest that many factors may influence recall threshold. Our preliminary study is the first to calculate the duration of wakefulness necessary for morning recall of nocturnal awakenings and the first to use a field-based design, allowing for the study of habitual sleep patterns at the participant's home. Further study is needed to explore if recall thresholds calculated using actigraphy can be validated against polysomnography (PSG) or be used to guide potential treatments.

Why tests appear to prevent forgetting: A distribution-based bifurcation model

Retrieving information from memory produces more learning than does being presented with the same information, and the benefits of such retrieval appear to grow as the delay before a final recall test grows longer. Recall tests, however, measure the number of items that are above a recall threshold, not memory strength per se. According to the model proposed in this paper, tests without feedback produce bifurcated item distributions: Retrieved items become stronger, but non-retrieved items remain weak, resulting in a gap between the two classes of items. Restudying items, on the other hand, strengthens all items, though to a lesser degree than does retrieval. These differing outcomes can make tested items appear to be forgotten more slowly than are restudied items—even if all items are forgotten at the same rate—because the test-induced bifurcation leaves items either well above or well below threshold. We review prior evidence and present three new experiments designed to test the bifurcation interpretation.

Formulating MEDLINE queries for article retrieval based on PubMed exemplars

AbstractBibliographic search engines allow endless possibilities for building queries based on specific words or phrases in article titles and abstracts, indexing terms, and other attributes. Unfortunately, deciding which attributes to use in a methodologically sound query is a non-trivial process. In this paper, we describe a system to help with this task, given an example set of PubMed articles to retrieve and a corresponding set of articles to exclude. The system provides the users with unigram and bigram features from the title, abstract, MeSH terms, and MeSH qualifier terms in decreasing order of precision, given a recall threshold. From this information and their knowledge of the domain, users can formulate a query and evaluate its performance. We apply the system to the task of distinguishing original research articles of functional magnetic resonance imaging (fMRI) of sensorimotor function from fMRI studies of higher cognitive functions.

Formulating MEDLINE queries for article retrieval based on PubMed exemplars

AbstractBibliographic search engines allow endless possibilities for building queries based on specific words or phrases in article titles and abstracts, indexing terms, and other attributes. Unfortunately, deciding which attributes to use in a methodologically sound query is a non-trivial process. In this paper, we describe a system to help with this task, given an example set of PubMed articles to retrieve and a corresponding set of articles to exclude. The system provides the users with unigram and bigram features from the title, abstract, MeSH terms, and MeSH qualifier terms in decreasing order of precision, given a recall threshold. From this information and their knowledge of the domain, users can formulate a query and evaluate its performance. We apply the system to the task of distinguishing original research articles of functional magnetic resonance imaging (fMRI) of sensorimotor function from fMRI studies of higher cognitive functions.

PIER: Protein interface recognition for structural proteomics

Recent advances in structural proteomics call for development of fast and reliable automatic methods for prediction of functional surfaces of proteins with known three-dimensional structure, including binding sites for known and unknown protein partners as well as oligomerization interfaces. Despite significant progress the problem is still far from being solved. Most existing methods rely, at least partially, on evolutionary information from multiple sequence alignments projected on protein surface. The common drawback of such methods is their limited applicability to the proteins with a sparse set of sequential homologs, as well as inability to detect interfaces in evolutionary variable regions. In this study, the authors developed an improved method for predicting interfaces from a single protein structure, which is based on local statistical properties of the protein surface derived at the level of atomic groups. The proposed Protein IntErface Recognition (PIER) method achieved the overall precision of 60% at the recall threshold of 50% at the residue level on a diverse benchmark of 490 homodimeric, 62 heterodimeric, and 196 transient interfaces (compared with 25% precision at 50% recall expected from random residue function assignment). For 70% of proteins in the benchmark, the binding patch residues were successfully detected with precision exceeding 50% at 50% recall. The calculation only took seconds for an average 300-residue protein. The authors demonstrated that adding the evolutionary conservation signal only marginally influenced the overall prediction performance on the benchmark; moreover, for certain classes of proteins, using this signal actually resulted in a deteriorated prediction. Thorough benchmarking using other datasets from literature showed that PIER yielded improved performance as compared with several alignment-free or alignment-dependent predictions. The accuracy, efficiency, and dependence on structure alone make PIER a suitable tool for automated high-throughput annotation of protein structures emerging from structural proteomics projects.

Can the NHS Breast Screening Programme Afford not to Double Read Screening Mammograms?

AIM: Rapid expansion of the National Health Service (UK) Breast Screening Programme (NHSBSP) to routinely invite women aged 50–70 years will result in many new readers undertaking screen reading. A timely method for assessing performance and preferably one that facilitates a steep learning curve will be required. MATERIALS AND METHODS: This unit screens a population of 88 000 women aged 50–64 years and double reads >90% films. A record is kept of proven screen-detected cancers not recalled for assessment by either the first or second reader but correctly recalled following third-reader arbitration. Individual readers’ workload and recall rates are obtained by running an annual co-writer report. The results of this 7 year prospective audit are presented. RESULTS: In total 177 167 women were screened between 1/4/95 and 31/3/02 resulting in the detection of 1072 cancers. Eighty-seven cancers (8.1%) were detected after arbitration. Individual readers recall to assessment rates and percentage of cases incorrectly returned to routine recall varied. Prompt feedback of missed/misinterpreted cases allowed both experienced and inexperienced readers to modify their recall thresholds for particular mammographic abnormalities. CONCLUSION: It is recommended this audit method is adopted by all units in the NHSBSP and that the Advisory Committee for Breast Cancer Screening review the policy of single versus double reading.

Assessing performance of screen readers in the National Health Service Breast Screening Programme (NHSBSP)

Rapid expansion of the National Health Service Breast Screening Programme (NHSBSP) will result in many new readers undertaking the task of screen reading. A timely method for assessing performance (sensitivity and specificity), and preferably one that facilitates a steep learning curve, will be required. Since 1995, 90% of films have been double read at this unit. A manual record is kept of cancers detected through double reading and subsequent 3rd reader arbitration. The number of cases read and individuals' recall rates, when acting as the 1st reader, were obtained by running an annual co-writer report. A total of 150,344 women were screened between April 1995 and March 2001, resulting in the detection of 880 cancers. Sixty-six (7.5%) were detected following arbitration. There was variation both in recall to assessment rates and in the number of cases incorrectly returned to routine recall between readers. Prompt feedback of 'missed' cases allowed readers to modify their recall thresholds for particular mammographic abnormalities. If the cases had been single read the reader would have remained unaware of the 'miss' until the woman presented symptomatically with an interval cancer or the cancer was detected at the next screening round. It is recommended that the National Screening Committee review the policy of single versus double reading in the NHSBSP.