Evaluating Computational Approaches for Harmful Content Analysis: Promise, Pitfalls and Tools for Responsible Research
This manuscript develops and demonstrates a practical framework for evaluating automated classifiers used in communication research, using harmful language detection as an illustrative case. We combine (a) a structured review of documentation practices for 27 publicly available classifiers and their associated annotation processes with (b) a cross-dataset evaluation that re-tests each model beyond its original training context. Across 27 datasets, we extract and compare reporting on construct definitions, annotator instructions, and inter-annotator agreement, and we quantify generalization by applying each model to multiple out-of-domain test sets. We also benchmark a contemporary large language model (GPT-5) under a consistent prompting protocol to illustrate how LLM-based classification compares to fine-tuned classifiers. Results show that documentation is uneven and often insufficient for theory-driven measurement, inter-annotator agreement varies widely across datasets, and cross-dataset performance frequently drops substantially relative to within-dataset evaluations. Building on these findings and existing validation guidance, we provide a reusable checklist and decision flow to help researchers select, justify, and report classifier-based measures in ways that support transparency and cumulative science. Recommendations for researchers, reviewers, and journal editors stress aligning model selection with standards of validity, reliability, and transparency.
- Research Article
21
- 10.1037/0735-7044.121.6.1243
- Dec 1, 2007
- Behavioral Neuroscience
The performance of conditioned responses (CRs) is diminished when trained subjects are tested in a novel context. This study tested the hypothesis that the flow of contextual information along the disynaptic "ESA" (entorhinal cortex-ventral subiculum-nucleus accumbens) pathway is responsible for context-related modulation of CRs. Rabbits received electrolytic or sham lesions of the ventral subiculum followed by discriminative avoidance conditioning and counterbalanced extinction sessions in the original training context, a novel context, and the original training context with a novel cue. Neuronal activity was recorded simultaneously in the nucleus accumbens, cingulate cortex, and basolateral amygdala. The lesions did not affect the acquisition of avoidance behavior or prevent the reduction of CRs in response to a novel cue. However, the lesions did reduce CR incidence during extinction, and they did eliminate a further novel-context-induced CR reduction found in controls. In addition, lesions disrupted context-dependent neuronal responses in the nucleus accumbens but not in the cingulate cortex or amygdala. These findings are interpreted as supportive of the hypothesis that the ESA pathway mediates contextual modulation of CRs during extinction.
- Research Article
45
- 10.1038/npp.2010.3
- Feb 3, 2010
- Neuropsychopharmacology
Anxiety disorders are increasingly prevalent in society; hence, there is a need to improve on existing treatments for such disorders. Fibroblast growth factor-2 (FGF2), a mitogen that is involved in brain development and regeneration, has been shown to both facilitate long-term extinction of fear and reduce stress-precipitated relapse in rats. Extinction is the laboratory analog of exposure-based therapies in humans. In this study, we continued to investigate the clinical potential of FGF2 as a pharmacological enhancer of extinction by examining its effect on renewal, a common type of relapse. In all experiments, rats were trained to fear a white noise-conditioned stimulus, and then this learned fear was extinguished the following day. Rats received systemic injections of FGF2 or vehicle immediately after extinction training. At test, on the day after extinction training, levels of freezing elicited by the white noise in either the extinction context or the original training context were measured. FGF2-treated rats showed less renewal of fear when tested in the original training context than did vehicle-treated rats. This pattern occurred even when vehicle rats were given double the amount of extinction training, and when FGF2-treated rats were given equivalent exposure to the extinction context. These results show that FGF2 facilitates long-term extinction and attenuates relapse, and thus highlight its potential as a novel pharmacological adjunct to exposure therapy.
- Research Article
10
- 10.1016/j.apmr.2022.06.010
- Jul 8, 2022
- Archives of Physical Medicine and Rehabilitation
Training in Varying Environmental Contexts Facilitates Transfer of Improved Gait Performance to New Contexts for Individuals With Parkinson Disease: A Randomized Controlled Trial
- Research Article
90
- 10.1016/j.neuroscience.2006.07.035
- Sep 1, 2006
- Neuroscience
Renewal of an extinguished instrumental response: Neural correlates and the role of D1 dopamine receptors
- Research Article
- 10.1371/journal.pone.0281958
- Mar 8, 2023
- PLOS ONE
Regional cerebral blood flow (rCBF) quantification using 123I-N-isopropyl-p-iodoamphetamine (123I-IMP) requires an invasive, one-time-only arterial blood sampling for measuring the 123I-IMP arterial blood radioactivity concentration (Ca10). The purpose of this study was to estimate Ca10 by machine learning (ML) using artificial neural network (ANN) regression analysis and consequently calculating rCBF and cerebral vascular reactivity (CVR) in the dual-table autoradiography (DTARG) method. This retrospective study included 294 patients who underwent rCBF measurements through the 123I-IMP DTARG. In the ML, the objective variable was defined by the measured Ca10, whereas the explanatory variables included 28 numeric parameters, such as patient characteristic values, total injection 123I-IMP radiation dose, cross-calibration factor, and the distribution of 123I-IMP count in the first scan. ML was performed with training (n = 235) and testing (n = 59) sets. Ca10 was estimated in testing set by our proposing model. Alternatively, the estimated Ca10 was also calculated via the conventional method. Subsequently, rCBF and CVR were calculated using estimated Ca10. Pearson's correlation coefficient (r-value) for the goodness of fit and the Bland-Altman analysis for assessing the potential agreement and bias were performed between the measured and estimated values. The r-value of Ca10 estimated by our proposed model was higher compared with the conventional method (0.81 and 0.66, respectively). In the Bland-Altman analysis, mean differences of 4.7 (95% limits of agreement (LoA): -18-27) and 4.1 (95% LoA: -35-43) were observed using proposed model and the conventional method, respectively. The r-values of rCBF at rest, rCBF after the acetazolamide challenge, and CVR calculated using the Ca10 estimated by our proposed model were 0.83, 0.80 and 0.95, respectively. Our proposed ANN-based model could accurately estimate the Ca10, rCBF, and CVR in DTARG. These results would enable non-invasive rCBF quantification in DTARG.
- Research Article
- 10.1371/journal.pone.0281958.r006
- Mar 8, 2023
- PLOS ONE
PurposeRegional cerebral blood flow (rCBF) quantification using 123I-N-isopropyl-p-iodoamphetamine (123I-IMP) requires an invasive, one-time-only arterial blood sampling for measuring the 123I-IMP arterial blood radioactivity concentration (Ca10). The purpose of this study was to estimate Ca10 by machine learning (ML) using artificial neural network (ANN) regression analysis and consequently calculating rCBF and cerebral vascular reactivity (CVR) in the dual-table autoradiography (DTARG) method.Materials and methodsThis retrospective study included 294 patients who underwent rCBF measurements through the 123I-IMP DTARG. In the ML, the objective variable was defined by the measured Ca10, whereas the explanatory variables included 28 numeric parameters, such as patient characteristic values, total injection 123I-IMP radiation dose, cross-calibration factor, and the distribution of 123I-IMP count in the first scan. ML was performed with training (n = 235) and testing (n = 59) sets. Ca10 was estimated in testing set by our proposing model. Alternatively, the estimated Ca10 was also calculated via the conventional method. Subsequently, rCBF and CVR were calculated using estimated Ca10. Pearson’s correlation coefficient (r-value) for the goodness of fit and the Bland–Altman analysis for assessing the potential agreement and bias were performed between the measured and estimated values.ResultsThe r-value of Ca10 estimated by our proposed model was higher compared with the conventional method (0.81 and 0.66, respectively). In the Bland–Altman analysis, mean differences of 4.7 (95% limits of agreement (LoA): −18–27) and 4.1 (95% LoA: −35–43) were observed using proposed model and the conventional method, respectively. The r-values of rCBF at rest, rCBF after the acetazolamide challenge, and CVR calculated using the Ca10 estimated by our proposed model were 0.83, 0.80 and 0.95, respectively.ConclusionOur proposed ANN-based model could accurately estimate the Ca10, rCBF, and CVR in DTARG. These results would enable non-invasive rCBF quantification in DTARG.
- Research Article
17
- 10.1148/ryai.230132
- Nov 15, 2023
- Radiology. Artificial intelligence
Purpose To develop an end-to-end deep learning (DL) pipeline for automated ventricular segmentation of cardiac MRI data from a multicenter registry of patients with Fontan circulation (Fontan Outcomes Registry Using CMR Examinations [FORCE]). Materials and Methods This retrospective study used 250 cardiac MRI examinations (November 2007-December 2022) from 13 institutions for training, validation, and testing. The pipeline contained three DL models: a classifier to identify short-axis cine stacks and two U-Net 3+ models for image cropping and segmentation. The automated segmentations were evaluated on the test set (n = 50) by using the Dice score. Volumetric and functional metrics derived from DL and ground truth manual segmentations were compared using Bland-Altman and intraclass correlation analysis. The pipeline was further qualitatively evaluated on 475 unseen examinations. Results There were acceptable limits of agreement (LOA) and minimal biases between the ground truth and DL end-diastolic volume (EDV) (bias: -0.6 mL/m2, LOA: -20.6 to 19.5 mL/m2) and end-systolic volume (ESV) (bias: -1.1 mL/m2, LOA: -18.1 to 15.9 mL/m2), with high intraclass correlation coefficients (ICCs > 0.97) and Dice scores (EDV, 0.91 and ESV, 0.86). There was moderate agreement for ventricular mass (bias: -1.9 g/m2, LOA: -17.3 to 13.5 g/m2) and an ICC of 0.94. There was also acceptable agreement for stroke volume (bias: 0.6 mL/m2, LOA: -17.2 to 18.3 mL/m2) and ejection fraction (bias: 0.6%, LOA: -12.2% to 13.4%), with high ICCs (>0.81). The pipeline achieved satisfactory segmentation in 68% of the 475 unseen examinations, while 26% needed minor adjustments, 5% needed major adjustments, and in 0.4%, the cropping model failed. Conclusion The DL pipeline can provide fast standardized segmentation for patients with single ventricle physiology across multiple centers. This pipeline can be applied to all cardiac MRI examinations in the FORCE registry. Keywords: Cardiac, Adults and Pediatrics, MR Imaging, Congenital, Volume Analysis, Segmentation, Quantification Supplemental material is available for this article. © RSNA, 2023.
- Research Article
166
- 10.1016/j.neuroscience.2007.11.018
- Nov 28, 2007
- Neuroscience
Renewal of extinguished cocaine-seeking
- Research Article
75
- 10.1016/j.biopsych.2012.08.006
- Sep 12, 2012
- Biological Psychiatry
Cholinergic Blockade Frees Fear Extinction from Its Contextual Dependency
- Research Article
8
- 10.1016/j.bbr.2012.12.030
- Dec 26, 2012
- Behavioural Brain Research
Hippocampal-dependent Pavlovian conditioning in adult rats exposed to binge-like doses of ethanol as neonates
- Research Article
31
- 10.1037/a0024100
- Jan 1, 2011
- Behavioral Neuroscience
We studied the role of context in reacquisition of extinguished reward-seeking. Rats were trained to respond for alcoholic beer, then extinguished and retrained. Reacquisition was faster than acquisition regardless of whether retraining occurred in the original training context, the extinction context, a novel context, or a context with a mixed history of reinforcement. Reacquisition was also rapid after extended extinction training. Nonetheless, context did significantly influence reacquisition via affecting latency to first response: rats took significantly longer to initiate responding when tested in the extinction context. These results suggest that reacquisition of drug and reward seeking is determined by an inhibitory influence caused by the extinction context and a facilitatory influence caused by reintroduction of the reinforcer (Bouton, 1993).
- Research Article
91
- 10.1038/npp.2014.197
- Aug 5, 2014
- Neuropsychopharmacology
After acquisition, hippocampus-dependent memories undergo a systems consolidation process, during which they become independent of the hippocampus and dependent on the anterior cingulate cortex (ACC) for memory expression. However, consolidated remote memories can become transiently hippocampus-dependent again following memory reactivation. How this systems reconsolidation affects the role of the ACC in remote memory expression is not known. Using contextual fear conditioning, we show that the expression of 30-day-old remote memory can transiently be supported by either the ACC or the dorsal hippocampus following memory reactivation, and that the ACC specifically mediates expression of remote generalized contextual fear memory. We found that suppression of neural activity in the ACC with the AMPA/kainate receptor antagonist 6-cyano-7-nitroquinoxaline-2,3-dione (CNQX) impaired the expression of remote, but not recent, contextual fear memory. Fear expression was not affected by this treatment if preceded by memory reactivation 6 h earlier, nor was it affected by suppression of neural activity in the dorsal hippocampus with the GABA-receptor agonist muscimol. However, simultaneous targeting of both the ACC and the dorsal hippocampus 6 h after memory reactivation disrupted contextual fear memory expression. Second, we observed that expression of a 30-day-old generalized contextual fear memory in a novel context was not affected by memory reactivation 6 h earlier. However, intra-ACC CNQX infusion before testing impaired contextual fear expression in the novel context, but not the original training context. Together, these data suggest that although the dorsal hippocampus may be recruited during systems reconsolidation, the ACC remains necessary for the expression of generalized contextual fear memory.
- Research Article
- 10.1101/lm.054104.125
- Jan 1, 2025
- Learning & Memory
Aversive associative learning paradigms such as inhibitory avoidance (IA) are frequently used to examine episodic-like memories in rodents. In IA, rodents learn to associate a context with a footshock, with testing for memory strength in the original training context and memory precision in a similar yet distinct neutral context. The present work assessed how different contextual exposure procedures affect memory strength and precision in IA at both recent and remote time points using male and female Long–Evans rats. An initial experiment found that rats kept in the lit (non-shock) compartment of the IA apparatus for 60 sec during training, as opposed to 10 sec, displayed enhanced memory strength, with discrimination between both chambers at the recent retention test and generalization at the remote retention test. Subsequent experiments investigated the effects of contextual pre-exposure the day before training. The results indicate that pre-exposure to the neutral context promoted generalization without altering memory strength compared to the first experiment, whereas pre-exposure to the aversive chamber promoted discrimination and memory strength. Notably, the different procedures yielded similar effects in both sexes. However, the results also indicate an overall pattern of greater contextual discrimination in females compared to males. These findings provide evidence for how different contextual exposures influence the degree of encoding at the time of training and a behavioral foundation for future studies examining the neurobiological mechanisms underlying memory strength and precision in IA, while highlighting the importance of using both sexes in initial behavioral work.
- Research Article
205
- 10.1016/j.neuroscience.2007.01.063
- Mar 23, 2007
- Neuroscience
The neural correlates and role of D1 dopamine receptors in renewal of extinguished alcohol-seeking
- Research Article
6
- 10.1523/jneurosci.1235-23.2023
- Nov 20, 2023
- The Journal of neuroscience : the official journal of the Society for Neuroscience
The necessity of conscious awareness in human learning has been a long-standing topic in psychology and neuroscience. Previous research on non-conscious associative learning is limited by the low signal-to-noise ratio of the subliminal stimulus, and the evidence remains controversial, including failures to replicate. Using functional MRI decoded neurofeedback, we guided participants from both sexes to generate neural patterns akin to those observed when visually perceiving real-world entities (e.g., dogs). Importantly, participants remained unaware of the actual content represented by these patterns. We utilized an associative DecNef approach to imbue perceptual meaning (e.g., dogs) into Japanese hiragana characters that held no inherent meaning for our participants, bypassing a conscious link between the characters and the dogs concept. Despite their lack of awareness regarding the neurofeedback objective, participants successfully learned to activate the target perceptual representations in the bilateral fusiform. The behavioral significance of our training was evaluated in a visual search task. DecNef and control participants searched for dogs or scissors targets that were pre-cued by the hiragana used during DecNef training or by a control hiragana. The DecNef hiragana did not prime search for its associated target but, strikingly, participants were impaired at searching for the targeted perceptual category. Hence, conscious awareness may function to support higher-order associative learning. Meanwhile, lower-level forms of re-learning, modification, or plasticity in existing neural representations can occur unconsciously, with behavioral consequences outside the original training context. The work also provides an account of DecNef effects in terms of neural representational drift.