Visual Classification Research Articles

Deep learning classification of ex vivo human colon tissues using spectroscopic optical coherence tomography.

Screening for colorectal cancer (CRC) with colonoscopy has improved patient outcomes; however, it remains the third leading cause of cancer-related mortality, novel strategies to improve screening are needed. Here, we propose an optical biopsy technique based on spectroscopic optical coherence tomography (OCT). Depth resolved OCT images are analyzed as a function of wavelength to measure optical tissue properties and used as input to machine learning algorithms. Previously, we used this approach to analyze mouse colon polyps. Here, we extend the approach to examine human biopsied colonic epithelial tissue samples ex vivo. Optical properties are used as input to a novel deep learning architecture, producing accuracy of up to 97.9% in discriminating tissue type. SOCT parameters are used to create false colored en face OCT images and deep learning classifications are used to enable visual classification by tissue type. This study advances SOCT toward clinical utility for analysis of colonic epithelium.

Processing of visual hapaxes in picture naming task: An event-related potential study

Object recognition and visual categorization are typically swift and seemingly effortless tasks that involve numerous underlying processes. In our investigation, we utilized a picture naming task to explore the processing of rarely encountered objects (visual hapaxes) in comparison to common objects. Our aim was to determine the stage at which these rare objects are classified as unnamable. Contrary to our expectations and in contrast to some prior research on event-related potentials (ERPs) with novel and atypical objects, no differences between conditions were observed in the late time windows corresponding to the P300 or N400 components. However, distinctive patterns between hapaxes and common objects surfaced in three early time windows, corresponding to the posterior N1 and P2 waves, as well as a widespread N2 wave. According to the ERP data, the differentiation between hapaxes and common objects occurs within the first 380 ms of the processing line, involving only limited and indirect top-down influence.

SPD Manifold Deep Metric Learning for Image Set Classification.

By characterizing each image set as a nonsingular covariance matrix on the symmetric positive definite (SPD) manifold, the approaches of visual content classification with image sets have made impressive progress. However, the key challenge of unhelpfully large intraclass variability and interclass similarity of representations remains open to date. Although, several recent studies have mitigated the two problems by jointly learning the embedding mapping and the similarity metric on the original SPD manifold, their inherent shallow and linear feature transformation mechanism are not powerful enough to capture useful geometric features, especially in complex scenarios. To this end, this article explores a novel approach, termed SPD manifold deep metric learning (SMDML), for image set classification. Specifically, SMDML first selects a prevailing SPD manifold neural network (SPDNet) as the backbone (encoder) to derive an SPD matrix nonlinear representation. To counteract the degradation of structural information during multistage feature embedding, we construct a Riemannian decoder at the end of the encoder, trained by a reconstruction error term (RT), to induce the generated low-dimensional feature manifold of the hidden layer to capture the pivotal information about the visual data describing the imaged scene. We demonstrate through theory and experiments that it is feasible to replace the Riemannian metric with Euclidean distance in RT. Then, the ReCov layer is introduced into the established Riemannian network to regularize the local statistical information within each input feature matrix, which enhances the effectiveness of the learning process. The theoretical analysis of the activation function used in the ReCov layer in terms of continuity and conditions for generating positive definite matrices is beneficial for network design. Inspired by the fact that the single cross-entropy loss used for training is unable to effectively parse the geometric distribution of the deep representations, we finally endow the suggested model with a novel metric learning regularization term. By explicitly incorporating the encoding and processing of the data variations into the network learning process, this term can not only derive a powerful Riemannian representation but also train an effective classifier. The experimental results show the superiority of the proposed approach on three typical visual classification tasks.

Cortical Face-Selective Responses Emerge Early in Human Infancy.

In human adults, multiple cortical regions respond robustly to faces, including the occipital face area (OFA) and fusiform face area (FFA), implicated in face perception, and the superior temporal sulcus (STS) and medial prefrontal cortex (MPFC), implicated in higher-level social functions. When in development, does face selectivity arise in each of these regions? Here, we combined two awake infant functional magnetic resonance imaging (fMRI) datasets to create a sample size twice the size of previous reports (n = 65 infants; 2.6-9.6 months). Infants watched movies of faces, bodies, objects, and scenes, while fMRI data were collected. Despite variable amounts of data from each infant, individual subject whole-brain activation maps revealed responses to faces compared to nonface visual categories in the approximate location of OFA, FFA, STS, and MPFC. To determine the strength and nature of face selectivity in these regions, we used cross-validated functional region of interest analyses. Across this larger sample size, face responses in OFA, FFA, STS, and MPFC were significantly greater than responses to bodies, objects, and scenes. Even the youngest infants (2-5 months) showed significantly face-selective responses in FFA, STS, and MPFC, but not OFA. These results demonstrate that face selectivity is present in multiple cortical regions within months of birth, providing powerful constraints on theories of cortical development.

Intraindividual Comparison of Different Methods for Automated BPE Assessment at Breast MRI: A Call for Standardization.

Background The level of background parenchymal enhancement (BPE) at breast MRI provides predictive and prognostic information and can have diagnostic implications. However, there is a lack of standardization regarding BPE assessment. Purpose To investigate how well results of quantitative BPE assessment methods correlate among themselves and with assessments made by radiologists experienced in breast MRI. Materials and Methods In this pseudoprospective analysis of 5773 breast MRI examinations from 3207 patients (mean age, 60 years ± 10 [SD]), the level of BPE was prospectively categorized according to the Breast Imaging Reporting and Data System by radiologists experienced in breast MRI. For automated extraction of BPE, fibroglandular tissue (FGT) was segmented in an automated pipeline. Four different published methods for automated quantitative BPE extractions were used: two methods (A and B) based on enhancement intensity and two methods (C and D) based on the volume of enhanced FGT. The results from all methods were correlated, and agreement was investigated in comparison with the respective radiologist-based categorization. For surrogate validation of BPE assessment, how accurately the methods distinguished premenopausal women with (n = 50) versus without (n = 896) antihormonal treatment was determined. Results Intensity-based methods (A and B) exhibited a correlation with radiologist-based categorization of 0.56 ± 0.01 and 0.55 ± 0.01, respectively, and volume-based methods (C and D) had a correlation of 0.52 ± 0.01 and 0.50 ± 0.01 (P < .001). There were notable correlation differences (P < .001) between the BPE determined with the four methods. Among the four quantitation methods, method D offered the highest accuracy for distinguishing women with versus without antihormonal therapy (P = .01). Conclusion Results of different methods for quantitative BPE assessment agree only moderately among themselves or with visual categories reported by experienced radiologists; intensity-based methods correlate more closely with radiologists' ratings than volume-based methods. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Mann in this issue.

Quantification of litter in cities using a smartphone application and citizen science in conjunction with deep learning-based image processing

Cities are a major source of litter pollution. Determination of the abundance and composition of plastic litter in cities is imperative for effective pollution management, environmental protection, and sustainable urban development. Therefore, here, a multidisciplinary approach to quantify and classify the abundance of litter in urban environments is proposed. In the present study, litter data collection was integrated via the Pirika smartphone application and conducted image analysis based on deep learning. Pirika was launched in May 2018 and, to date, has collected approximately one million images. Visual classification revealed that the most common types of litter were cans, plastic bags, plastic bottles, cigarette butts, cigarette boxes, and sanitary masks, in that order. The top six categories accounted for approximately 80 % of the total, whereas the top three categories accounted for more than 60 % of the total imaged litter. A deep-learning image processing algorithm was developed to automatically identify the top six litter categories. Both precision and recall derived from the model were higher than 75 %, enabling proper litter categorization. The quantity of litter derived from automated image processing was also plotted on a map using location data acquired concurrently with the images by the smartphone application. Conclusively, this study demonstrates that citizen science supported by smartphone applications and deep learning-based image processing can enable the visualization, quantification, and characterization of street litter in cities.

Multi-granularity hypergraph-guided transformer learning framework for visual classification

Multi-granularity hypergraph-guided transformer learning framework for visual classification

TransFGVC: transformer-based fine-grained visual classification

TransFGVC: transformer-based fine-grained visual classification

Life-history parameters of adult females of Merluccius capensis (Gadidae) off the south coast of South Africa.

This study explores the life-history parameters of female Merluccius capensis off South Africa (N = 1819) during 2014-2016, including gonadosomatic index (GSI), length-at-maturity, length-weight relationships, and condition indices (relative condition [k] and Fulton's condition factor [K]). We detected weak indications of two peaks of spawning within the year, the first in austral autumn from March to May, whereas the other in austral spring around August. GSI was slightly higher in spring and autumn, though still low at all maturity stages (≤7%), though the opposite was true for the actively spawning stage (≥7%) as well as access to less such data during winter- and summertime. The length (L) at 50% maturity was around 38 cm (L50), though differences occurred between the two applied staging methods, histology and visual (macroscopic) classification, when L approached infinity. The latter method presented underestimated length at maturity values at the 75 and 95 percentiles (48 and 60 cm) compared to the corresponding percentiles given by histology (50 and 65 cm). There were trivial across-method differences in L50. However, we found a clear reduction in L50 in view of published information in prior years when this estimate was 48 (1985), 42 (2008), 53 (2011), and 24.8 (2015) cm. Overall, L explained 90% of the variation in whole body weight (W). As the bootstrapped, grand mean growth coefficient was b = 2.98, indicating a slight allometric growth function, there were no significant variations between years, though an isometric growth existed for 2016 with b = 3.0, whereas for 2014 and 2015 this b was 2.98 and 2.93, respectively. In terms of demography, females <60 cm generally showed isometric growth (b = 3) as opposed to allometric growth (b = 2.95) at >60 cm. The relative condition index (k = 1) exhibited higher values than Fulton's K, which was 0.80. Overall, the maternal stock of M. capensis along the south coast seems to be in good condition and likely spawns throughout the year, but we found that the macroscopic data tend to give biased maturity ogives.

Contrast-enhanced ultrasound evaluation of plaque vulnerability and the relationship between peripheral blood leukocytes.

To evaluate plaque vulnerability by carotid contrast-enhanced ultrasound (CEUS) and to analyze the correlation between plaque vulnerability and peripheral blood leukocyte classification. 135 patients with carotid plaque were examined by contrast-enhanced ultrasound. Plaque vulnerability was assessed by semiquantitative visual classification. Baseline clinical data and peripheral leukocyte classification were collected. Ordered logistic regression was used to analyze the correlation between plaque neovascularization grade and peripheral leukocyte classification count. There were significant differences in leukocyte, monocyte, neutrophil, mean platelet volume, lymphocyte, and neutrophil counts between different neovascular plaque grades and peripheral blood (P < 0.05). Correlation analysis showed that leukocyte, monocyte, and neutrophil counts were significantly positively correlated. The increase in plaque neovascularization is associated with an increase in circulating leukocytes, monocytes, and neutrophils. Therefore, CEUS combined with peripheral blood leukocytes may serve as an early warning of plaque vulnerability and provide a theoretical basis for clinical treatment.

Modelling the impact of single vs. dual presentation on visual discrimination across resolutions.

Visual categorisation relies on our ability to extract useful diagnostic information from complex stimuli. To do this, we can utilise both the "high-level" and "low-level" information in a stimulus; however, the extent to which changes in these properties impact the decision-making process is less clear. We manipulated participants' access to high-level category features via gradated reductions to image resolution while exploring the impact of access to additional category features through a dual-stimulus presentation when compared with single stimulus presentation. Results showed that while increasing image resolution consistently resulted in better choice performance, no benefit was found for dual presentation over single presentation, despite responses for dual presentation being slower compared with single presentation. Applying the diffusion decision model revealed increases in drift rate as a function of resolution, but no change in drift rate for single versus dual presentation. The increase in response time for dual presentation was instead accounted for by an increase in response caution for dual presentations. These findings suggest that while increasing access to high-level features (via increased resolution) can improve participants' categorisation performance, increasing access to both high- and low-level features (via an additional stimulus) does not.

IMG-25. PREDICTING ANATOMICAL TUMOR GROWTH IN PEDIATRIC HIGH-GRADE GLIOMAS VIA DENOISING DIFFUSION MODELS

Abstract BACKGROUND Pediatric diffuse midline glioma (DMG) has a poor prognosis with radiotherapy as the standard of palliative care. Radiation strategies using individual geometric doseshaping have the potential to maximize patient benefit. This study aimed to establish a proof-of-principle to generate clinically relevant anatomical representations of the progression of pediatric DMG magnetic resonance image (MRI) slices through denoising diffusion implicit models (DDIM). METHODS Based on the BraTS23 data, we analyzed the generation of MRIs with enlarged tumor sizes starting from a baseline scan. Models were pre-trained on adult glioblastoma (GBM) prior to transfer learning on pediatric high-grade glioma. Test set performance was evaluated using the Frechet Inception Distance (FID), and Structural Similarity Index (SSIM). Qualitative evaluation involved visual classification of real vs generated images by two independent radiology trainees. The optimized network was tested to forecast anatomical tumor growth in five DMG patients from an independent dataset comprising longitudinal T2-FLAIR images. RESULTS We obtain high-fidelity generated images, supported by an FID of 12.4 and 0.8 SSIM on the adult test set. Despite the smaller pediatric BraTS dataset yielding a 34.8 FID and 0.84 SSIM, human observers failed to distinguish generated from real MRIs (mean recall and precision of 0.51). The generated anatomical predictions also closely resemble the observed longitudinal tumor progression in the pediatric DMG subset. Notably, the direction and extent of tumor growth align while the surrounding brain anatomy is preserved. Quantitatively, the mean DICE score between segmentations of generated and real tumors was 0.8. CONCLUSIONS Our model produces high-quality images that conform to growth patterns of pediatric DMGs. Further quantitative validation on larger longitudinal datasets is required, to ensure the method’s robustness. The proposed method enables a personalized approach for defining target regions in radiation therapy which could translate to improved clinical outcomes.

A deep learning model for FaceSwap and face-reenactment deepfakes detection

Recently, the higher availability of multimedia content on social websites, together with lightweight deep learning (DL) empowered tools like Generative Adversarial Networks (GANs) has caused the generation of realistic deepfakes. Such fabricated data has the potential to spread disinformation, revenge porn, initiate monetary scams, and can result in adverse immoral and illegal societal issues, etc. Hence, the accurate identification of deepfakes is mandatory to discriminate between real and manipulated content. In this work, we have presented a DL-based approach namely a unified network for FaceSwap (FS) and Face-Reenactment (FR) Deepfakes Detection (AUFF-Net). More clearly, both the spatial and temporal information from the video samples are used to detect two types of visual manipulations i.e., FS and FR. For this reason, a novel DL framework namely the Inception-Swish-ResNet-v2 model is introduced as a feature extractor for computing the information at the spatial level. While the Bi-LSTM model is utilized to measure the temporal information. Additionally, 3 dense layers are included at the last of the model structure to suggest a discriminative group of the feature vector We performed extensive experimentation on a challenging dataset namely the FaceForensic++, and attainede average accuracy values of 99.21 %, and 98.32 % for FS, or FR, respectively. Furthermore, we introduced an explainability module to show the reliable keypoints selection capability of our technique. Moreover, we have performed a cross-dataset evaluation to show the generalization power of our approach. Both the qualitative and quantitative results have confirmed the effectiveness of the suggested approach for visual manipulation categorization under the occurrence of various adversarial attacks.

Timber Strength Grading as Necessary Basis for Structural Design in ex-YU Region

The aim of this paper is to contribute to the classification of structural timber in the ex-YU region into strength-class system through the application of experimentally obtained archive data in order to provide a realistic framework for most commonly used II grade (according to JUS) structural coniferous timber. The analysis of archive data was carried out on a sample of 150 specimens of structural size and based on the set of statistical requirements prescribed by EN standards, taking into account the change in disposition of loading in laboratory testing in the past and now. Statistical procedures prescribed by EN standards are given through calculation steps together with necessary adjustment factors that cover the size and number of specimens. The presented procedures given for structural-size specimens are also applicable to small clear specimens, so that a more comprehensive research and additional new examinations could be conducted with the available archive data simultaneously with the harmonization of the visual classification rules applied in the ex-YU region. The paper emphasises the direct dependence of the consistent application of the visual grading rules required by the relevant EN standard on strength-class system.

Visual Ship Image Synthesis and Classification Framework Based on Attention-DCGAN

To improving ship image generation and classification tasks, a deep convolution generative adversarial network based on attention mechanism (ADCGAN) model was constructed. The rectified linear unit (ReLU) activation function was adopted, and three Deconv layers and Conv layers were added to both the generator and discriminator. Subsequently, an attention mechanism was added to the generator, while spectral normalization (SN) was added to the discriminator. Mean squared error (MSE) was used as loss function to stabilize the training process. Furthermore, ship classification tasks were performed using the generated ship images by end-to-end training of the classification network, enabling ship data augmentation and co-learning with other tasks. Experimental results on the Ship700 and Seaship7000 datasets demonstrate that the ADCGAN model can generate clear and robust ship images, with PSNR, LIPIPS, MS-SSIM values of 20.279 and 27.523, 0.596 and 0.096, 0.781 and 0.947, respectively. The effectiveness of the proposed method in ship image classification tasks was also verified, providing a data foundation for other collaborative tasks.

Lv-Adapter: Adapting Vision Transformers for Visual Classification with Linear-layers and Vectors

Large pre-trained models based on Vision Transformers (ViTs) contain nearly billions of parameters, demanding substantial computational resources and storage space. This restricts their transferability across different tasks. Recent approaches try to use adapter fine-tuning to address this drawback. However, there is still potential to improve the number of tunable parameters and the accuracy in these methods. To address this challenge, we propose an adapter fine-tuning module called Lv-Adapter, which consists of a linear layer and vector. This module enables targeted parameter fine-tuning of pretrained models by learning both the prior knowledge of pre-trained task and the information from downstream specific task, to adapt to various downstream tasks in image and video tasks while transfer learning. Compared to full fine-tuning methods, Lv-Adapter has several appealing advantages. Firstly, by adding only about 3% extra parameters to ViT, Lv-Adapter achieves comparable accuracy to full fine-tuning methods and even significantly surpasses them on action recognition benchmarks. Secondly, Lv-Adapter is a lightweight module that can be plug-and-play in different transformer models due to its simplicity. Finally, to validate these claims, extensive experiments were conducted on five image and video datasets in this study, providing evidence for the effectiveness of Lv-Adapter. When only 3.5% of the extra parameters are updated, it respectively achieves a relative boost of about 13% and 24% compared to the fully fine-tuned model on SSv2 and HMDB51.

Systematic analysis of jellyfish galaxy candidates in Fornax, Antlia, and Hydra from the S-PLUS survey: a self-supervised visual identification aid

ABSTRACT We study 51 jellyfish galaxy candidates in the Fornax, Antlia, and Hydra clusters. These candidates are identified using the JClass scheme based on the visual classification of wide-field, twelve-band optical images obtained from the Southern Photometric Local Universe Survey. A comprehensive astrophysical analysis of the jellyfish (JClass &amp;gt; 0), non-jellyfish (JClass = 0), and independently organized control samples is undertaken. We develop a semi-automated pipeline using self-supervised learning and similarity search to detect jellyfish galaxies. The proposed framework is designed to assist visual classifiers by providing more reliable JClasses for galaxies. We find that jellyfish candidates exhibit a lower Gini coefficient, higher entropy, and a lower 2D Sérsic index as the jellyfish features in these galaxies become more pronounced. Jellyfish candidates show elevated star formation rates (including contributions from the main body and tails) by $\sim$1.75 dex, suggesting a significant increase in the SFR caused by the ram-pressure stripping phenomenon. Galaxies in the Antlia and Fornax clusters preferentially fall towards the cluster’s centre, whereas only a mild preference is observed for Hydra galaxies. Our self-supervised pipeline, applied in visually challenging cases, offers two main advantages: it reduces human visual biases and scales effectively for large data sets. This versatile framework promises substantial enhancements in morphology studies for future galaxy image surveys.

Detecting galaxy tidal features using self-supervised representation learning

ABSTRACT Low surface brightness substructures around galaxies, known as tidal features, are a valuable tool in the detection of past or ongoing galaxy mergers, and their properties can answer questions about the progenitor galaxies involved in the interactions. The assembly of current tidal feature samples is primarily achieved using visual classification, making it difficult to construct large samples and draw accurate and statistically robust conclusions about the galaxy evolution process. With upcoming large optical imaging surveys such as the Vera C. Rubin Observatory’s Legacy Survey of Space and Time, predicted to observe billions of galaxies, it is imperative that we refine our methods of detecting and classifying samples of merging galaxies. This paper presents promising results from a self-supervised machine learning model, trained on data from the Ultradeep layer of the Hyper Suprime-Cam Subaru Strategic Program optical imaging survey, designed to automate the detection of tidal features. We find that self-supervised models are capable of detecting tidal features, and that our model outperforms previous automated tidal feature detection methods, including a fully supervised model. An earlier method applied to real galaxy images achieved 76 per cent completeness for 22 per cent contamination, while our model achieves considerably higher (96 per cent) completeness for the same level of contamination. We emphasize a number of advantages of self-supervised models over fully supervised models including maintaining excellent performance when using only 50 labelled examples for training, and the ability to perform similarity searches using a single example of a galaxy with tidal features.

Beyond faces: the contribution of the amygdala to visual processing in the macaque brain.

The amygdala is present in a diverse range of vertebrate species, such as lizards, rodents, and primates; however, its structure and connectivity differs across species. The increased connections to visual sensory areas in primate species suggests that understanding the visual selectivity of the amygdala in detail is critical to revealing the principles underlying its function in primate cognition. Therefore, we designed a high-resolution, contrast-agent enhanced, event-related fMRI experiment, and scanned 3 adult rhesus macaques, while they viewed 96 naturalistic stimuli. Half of these stimuli were social (defined by the presence of a conspecific), the other half were nonsocial. We also nested manipulations of emotional valence (positive, neutral, and negative) and visual category (faces, nonfaces, animate, and inanimate) within the stimulus set. The results reveal widespread effects of emotional valence, with the amygdala responding more on average to inanimate objects and animals than faces, bodies, or social agents in this experimental context. These findings suggest that the amygdala makes a contribution to primate vision that goes beyond an auxiliary role in face or social perception. Furthermore, the results highlight the importance of stimulus selection and experimental design when probing the function of the amygdala and other visually responsive brain regions.

Visual periodicity reveals distinct attentional signatures for face and non-face categories.

Observers can selectively deploy attention to regions of space, moments in time, specific visual features, individual objects, and even specific high-level categories-for example, when keeping an eye out for dogs while jogging. Here, we exploited visual periodicity to examine how category-based attention differentially modulates selective neural processing of face and non-face categories. We combined electroencephalography with a novel frequency-tagging paradigm capable of capturing selective neural responses for multiple visual categories contained within the same rapid image stream (faces/birds in Exp 1; houses/birds in Exp 2). We found that the pattern of attentional enhancement and suppression for face-selective processing is unique compared to other object categories: Where attending to non-face objects strongly enhances their selective neural signals during a later stage of processing (300-500ms), attentional enhancement of face-selective processing is both earlier and comparatively more modest. Moreover, only the selective neural response for faces appears to be actively suppressed by attending towards an alternate visual category. These results underscore the special status that faces hold within the human visual system, and highlight the utility of visual periodicity as a powerful tool for indexing selective neural processing of multiple visual categories contained within the same image sequence.