Visual Classification Research Articles

Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification

Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial transformer, whereby the spiking self-attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this article, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier transform, wavelet transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies (0.4%–1.5%), higher running speed (9%–51% for training and 19%–70% for inference), reduced theoretical energy consumption (20%–25%), and reduced graphic processing unit (GPU) memory usage (4%–26%), compared to the standard spikformer. Our result indicates the continuous refinement of new transformers that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet transform), is promising.

ALMA/SCUBA-2 COSMOS Survey: Properties of X-Ray- and SED-selected Active Galactic Nuclei in Bright Submillimeter Galaxies

Abstract We investigate the properties of active galactic nuclei (AGNs) in the brightest submillimeter galaxies (SMGs) in the COSMOS field. We utilize the bright sample of the ALMA/SCUBA-2 COSMOS Survey (AS2COSMOS), which consists of 260 SMGs with S 870 μm = 0.7–19.2 mJy at z = 0–6. We perform optical to millimeter spectral energy distribution (SED) modeling for the whole sample. We identify 24 AGN-host galaxies from the SEDs. Supplemented by 23 X-ray-detected AGNs (X-ray AGNs), we construct an overall sample of 40 AGN-host galaxies. The X-ray luminosity upper bounds indicate that the X-ray-undetected SED-identified AGNs are likely to be nearly Compton thick or have unusually suppressed X-ray emission. From visual classification, we identify 2 5 − 5 + 6 % of the SMGs without AGNs as major merger candidates. This fraction is almost consistent with the general galaxy population at z ∼ 2, suggesting that major mergers are not necessarily required for the enhanced star formation in SMGs. We also identify 4 7 − 15 + 16 % of the AGN hosts as major merger candidates, which is about twice as high as that in the SMGs without AGNs. This suggests that major mergers play a key role in triggering AGN activity in bright SMGs.

Multi-level navigation network: advancing fine-grained visual classification

Multi-level navigation network: advancing fine-grained visual classification

Flexibility in choosing decision policies in gathering discrete evidence over time.

The brain can remarkably adapt its decision-making process to suit the dynamic environment and diverse aims and demands. The brain's flexibility can be classified into three categories: flexibility in choosing solutions, decision policies, and actions. We employ two experiments to explore flexibility in decision policy: a visual object categorization task and an auditory object categorization task. Both tasks required participants to accumulate discrete evidence over time, with the only difference being the sensory state of the stimuli. We aim to investigate how the brain demonstrates flexibility in selecting decision policies in different sensory contexts when the solution and action remain the same. Our results indicate that the decision policy of the brain in integrating information is independent of inter-pulse interval across these two tasks. However, the decision policy based on how the brain ranks the first and second pulse of evidence changes flexibly. We show that the sequence of pulses does not affect the choice accuracy in the auditory mode. However, in the visual mode, the first pulse had the larger leverage on decisions. Our research underscores the importance of incorporating diverse contexts to improve our understanding of the brain's flexibility in real-world decision-making.

A vision-language foundation model for precision oncology.

Clinical decision-making is driven by multimodal data, including clinical notes and pathological characteristics. Artificial intelligence approaches that can effectively integrate multimodal data hold significant promise in advancing clinical care1,2. However, the scarcity of well-annotated multimodal datasets in clinical settings has hindered the development of useful models. In this study, we developed the Multimodal transformer with Unified maSKed modeling (MUSK), a vision-language foundation model designed to leverage large-scale, unlabelled, unpaired image and text data. MUSK was pretrained on 50 million pathology images from 11,577 patients and one billion pathology-related text tokens using unified masked modelling. It was further pretrained on one million pathology image-text pairs to efficiently align the vision and language features. With minimal or no further training, MUSK was tested in a wide range of applications and demonstrated superior performance across 23 patch-level and slide-level benchmarks, including image-to-text and text-to-image retrieval, visual question answering, image classification and molecular biomarker prediction. Furthermore, MUSK showed strong performance in outcome prediction, including melanoma relapse prediction, pan-cancer prognosis prediction and immunotherapy response prediction in lung and gastro-oesophageal cancers. MUSK effectively combined complementary information from pathology images and clinical reports and could potentially improve diagnosis and precision in cancer therapy.

Item Classification by Difficulty Using Functional Principal Component Clustering and Neural Networks.

Maintaining consistent item difficulty across test forms is crucial for accurately and fairly classifying examinees into pass or fail categories. This article presents a practical procedure for classifying items based on difficulty levels using functional data analysis (FDA). Methodologically, we clustered item characteristic curves (ICCs) into difficulty groups by analyzing their functional principal components (FPCs) and then employed a neural network to predict difficulty for ICCs. Given the degree of similarity between many ICCs, categorizing items by difficulty can be challenging. The strength of this method lies in its ability to provide an empirical and consistent process for item classification, as opposed to relying solely on visual inspection. The findings reveal that most discrepancies between visual classification and FDA results differed by only one adjacent difficulty level. Approximately 67% of these discrepancies involved items in the medium to hard range being categorized into higher difficulty levels by FDA, while the remaining third involved very easy to easy items being classified into lower levels. The neural network, trained on these data, achieved an accuracy of 79.6%, with misclassifications also differing by only one adjacent difficulty level compared to FDA clustering. The method demonstrates an efficient and practical procedure for classifying test items, especially beneficial in testing programs where smaller volumes of examinees tested at various times throughout the year.

Neural correlates of phonetic categorization under auditory (phoneme) and visual (grapheme) modalities

This study assessed the neural mechanisms and relative saliency of categorization for speech sounds and comparable graphemes (i.e., visual letters) of the same phonetic label. Given that linguistic experience shapes categorical processing, and letter-speech sound matching plays a crucial role during early reading acquisition, we hypothesized sound phoneme and visual grapheme tokens representing the same linguistic identity might recruit common neural substrates, despite originating from different sensory modalities. Behavioral and neuroelectric brain responses (ERPs) were acquired as participants categorized stimuli from sound (phoneme) and homologous letter (grapheme) continua each spanning a /da/−/ga/ gradient. Behaviorally, listeners were faster and showed stronger categorization of phoneme compared to graphemes. At the neural level, multidimensional scaling of the EEG revealed responses self-organized in a categorial fashion such that tokens clustered within their respective modality beginning ∼150–250 ms after stimulus onset. Source-resolved ERPs further revealed modality-specific and overlapping brain regions supporting phonetic categorization. Left inferior frontal gyrus and auditory cortex showed stronger responses for sound category members compared to phonetically ambiguous tokens, whereas early visual cortices paralleled this categorical organization for graphemes. Auditory and visual categorization also recruited common visual association areas in extrastriate cortex but in opposite hemispheres (auditory = left; visual = right). Our findings reveal both auditory and visual sensory cortex supports categorical organization for phonetic labels within their respective modalities. However, a partial overlap in phoneme and grapheme processing among occipital brain areas implies the presence of an isomorphic, domain-general mapping for phonetic categories in dorsal visual system

Universal Fine-Grained Visual Categorization by Concept Guided Learning

Universal Fine-Grained Visual Categorization by Concept Guided Learning

An Attention-Locating Algorithm for Eliminating Background Effects in Fine-grained Visual Classification

An Attention-Locating Algorithm for Eliminating Background Effects in Fine-grained Visual Classification

Multiscale feature fusion and enhancement in a transformer for the fine-grained visual classification of tree species

Multiscale feature fusion and enhancement in a transformer for the fine-grained visual classification of tree species

Hierarchical Feature Attention Learning Network for Detecting Object and Discriminative Parts in Fine-Grained Visual Classification

Hierarchical Feature Attention Learning Network for Detecting Object and Discriminative Parts in Fine-Grained Visual Classification

Sequence order resolves ambiguity in a nonlinguistic visual categorization task.

When we encounter an unfamiliar word in a sentence, word order can be used to determine the grammatical category to which that word belongs and clarify ambiguity. However, it is unclear whether a similar categorization effect occurs in nonlinguistic contexts. We created three perceptually distinct categories of shape stimuli-rounded (A); squared (B); pointed (C). Participants were first taught to categorize these shapes, then participants in the experimental condition were trained to select them in a fixed sequence order (A→ B → C) while participants in the control condition were allowed to select them in any order. We then generated ambiguous stimuli by morphing shapes from different categories together. In the experimental condition, when an ambiguous morph was presented in a sequence, its subsequent categorization radically shifted based on which shape it replaced. For example, when an AB morph replaced the A shape in a sequence, it was more likely to be categorized as A. By contrast, participants in the control condition, who selected these stimuli without a fixed sequence order, showed no effect, demonstrating the critical role of sequence order and ruling out alternative explanations. These results demonstrate that, even using nonlinguistic stimuli, the position in which an ambiguous stimulus occurs in a sequence drastically impacts how it is subsequently categorized.

Prediction of Survival After Pediatric Cardiac Arrest Using Quantitative EEG and Machine Learning Techniques.

Early neuroprognostication in children with reduced consciousness after cardiac arrest (CA) is a major clinical challenge. EEG is frequently used for neuroprognostication in adults, but has not been sufficiently validated for this indication in children. Using machine learning techniques, we studied the predictive value of quantitative EEG (qEEG) features for survival 12 months after CA, based on EEG recordings obtained 24 hours after CA in children. The results were confirmed through visual analysis of EEG background patterns. This is a retrospective single-center study including children (0-17 years) with CA, who were subsequently admitted to the pediatric intensive care unit (PICU) of a tertiary care hospital between 2012 and 2021 after return of circulation (ROC) and were monitored using EEG at 24 hours after ROC. Signal features were extracted from a 30-minute EEG segment 24 hours after CA and used to train a random forest model. The background pattern from the same EEG fragment was visually classified. The primary outcome was survival or death 12 months after CA. Analysis of the prognostic accuracy of the model included calculation of receiver-operating characteristic and predictive values. Feature contribution to the model was analyzed using Shapley values. Eighty-six children were included (in-hospital CA 27%, out-of-hospital CA 73%). The median age at CA was 2.6 years; 53 (62%) were male. Mortality at 12 months was 56%; main causes of death on the PICU were withdrawal of life-sustaining therapies because of poor neurologic prognosis (52%) and brain death (31%). The random forest model was able to predict death at 12 months with an accuracy of 0.77 and positive predictive value of 1.0. Continuity and amplitude of the EEG signal were the signal parameters most contributing to the model classification. Visual analysis showed that no patients with a background pattern other than continuous with amplitudes exceeding 20 μV were alive after 12 months. Both qEEG and visual EEG background classification for registrations obtained 24 hours after ROC form a strong predictor of nonsurvival 12 months after CA in children.

The emergence of visual category representations in infants' brains.

Organizing the continuous stream of visual input into categories like places or faces is important for everyday function and social interactions. However, it is unknown when neural representations of these and other visual categories emerge. Here, we used steady-state evoked potential electroencephalography to measure cortical responses in infants at 3-4 months, 4-6 months, 6-8 months, and 12-15 months, when they viewed controlled, gray-level images of faces, limbs, corridors, characters, and cars. We found that distinct responses to these categories emerge at different ages. Reliable brain responses to faces emerge first, at 4-6 months, followed by limbs and places around 6-8 months. Between 6 and 15 months response patterns become more distinct, such that a classifier can decode what an infant is looking at from their brain responses. These findings have important implications for assessing typical and atypical cortical development as they not only suggest that category representations are learned, but also that representations of categories that may have innate substrates emerge at different times during infancy.

Effect of Hemolysis on Routine Blood Gas and Whole Blood Analytes.

Hemolysis is a major pre-analytical concern for many laboratory analytes; however, instruments utilized for whole blood chemistries and blood gas measurements lack the ability to detect and measure the degree of hemolysis. This study evaluated the effect of hemolysis on 13 routine whole blood and blood gas analytes and compared visual assessments of hemolysis to measured hemolysis (H-index). Remnant whole blood samples (n = 85) were split into 2 portions and aspirated through a syringe one or more times. To induce hemolysis, a needle was affixed to the end of the syringe to provide shear stress, and a mock procedure without syringe was used as a control. Samples were analyzed on a Radiometer ABL800 series instrument, centrifuged, and the H-index of the plasma portion was measured. Two medical technologists recorded a visual categorization of the specimens as slightly, moderately, or severely hemolyzed. Hemolysis had a modest effect on metabolites and most cooximetry components, with percent bias within ±5% at all levels of hemolysis. Methemoglobin exhibited a larger overall negative bias, up to 13.3%. The absolute pH bias was fairly consistent (within 0.1 pH units) across all levels of hemolysis. As expected, potassium displayed a significant positive bias with increasing hemolysis. Sodium and ionized calcium displayed overall linear trends with a significant negative bias. Hemolysis can falsely increase or decrease certain blood gas analytes and lead to misinterpretation of results. Therefore, hemolysis detection capabilities are crucial for mitigating this effect and ensuring accurate results for patient care.

Resting-State Network Plasticity Following Category Learning Depends on Sensory Modality.

Learning new categories is fundamental to cognition, occurring in daily life through various sensory modalities. However, it is not well known how acquiring new categories can modulate the brain networks. Resting-state functional connectivity is an effective method for detecting short-term brain alterations induced by various modality-based learning experiences. Using fMRI, our study investigated the intricate link between novel category learning and brain network reorganization. Eighty-four adults participated in an object categorization experiment utilizing visual (n = 41, with 20 females and a mean age of 23.91 ± 3.11 years) or tactile (n = 43, with 21 females and a mean age of 24.57 ± 2.58 years) modalities. Resting-state networks (RSNs) were identified using independent component analysis across the group of participants, and their correlation with individual differences in object category learning across modalities was examined using dual regression. Our results reveal an increased functional connectivity of the frontoparietal network with the left superior frontal gyrus in visual category learning task and with the right superior occipital gyrus and the left middle temporal gyrus after tactile category learning. Moreover, the somatomotor network demonstrated an increased functional connectivity with the left parahippocampus exclusively after tactile category learning. These findings illuminate the neural mechanisms of novel category learning, emphasizing distinct brain networks' roles in diverse modalities. The dynamic nature of RSNs emphasizes the ongoing adaptability of the brain, which is essential for efficient novel object category learning. This research provides valuable insights into the dynamic interplay between sensory learning, brain plasticity, and network reorganization, advancing our understanding of cognitive processes across different modalities.

Mammographic Vascular Microcalcifications as a Surrogate Parameter for Coronary Heart Disease: Correlation to Cardiac Computer Tomography and Proposal of a Classification Score.

This study develops a BI-RADS-like scoring system for vascular microcalcifications in mammographies, correlating breast arterial calcification (BAC) in a mammography with coronary artery calcification (CAC), and specifying differences between microcalcifications caused by BAC and microcalcifications potentially associated with malignant disease. This retrospective single-center cohort study evaluated 124 consecutive female patients (with a median age of 57 years). The presence of CAC was evaluated based on the Agatston score obtained from non-enhanced coronary computed tomography, and the calcifications detected in the mammography were graded on a four-point Likert scale, with the following criteria: (1) no visible or sporadically scattered microcalcifications, (2) suspicious microcalcification not distinguishable from breast arterial calcification, (3) minor breast artery calcifications, and (4) major breast artery calcifications. Inter-rater agreement was assessed in three readers using the Fleiss' kappa, and the correlation between CAC and BAC was evaluated using the Spearman's rank-order and by the calculation of sensitivity/specificity. The reliability of the visual classification of BAC was high, with an overall Fleiss' kappa for inter-rater agreement of 0.76 (ranging between 0.62 and 0.89 depending on the score). In 15.1% of patients, a BAC score of two was assigned indicating calcifications indistinguishable regarding vascular or malignant origin. In 17.7% of patients, minor or major breast artery calcifications were found (BAC 3-4). BAC was more prevalent among the patients with CAC (p < 0.001), and the severity of CAC increased with the BAC score; in the group with a BAC score of one, 15% of patients exhibited mild and severe CAC, in those with a BAC of two, this was 31%, in those with BAC of three, this was 38%, and in those with a BAC of four, this was 44%. The sensitivity for detecting CAC, based on the mammographic BAC score, was 30.3% at a specificity of 96.7%. The standardized visual grading of BAC in mammographies on a four-point scale is feasible with substantial interobserver agreement, potentially improving the treatment of patients with suspicious microcalcifications and CAC.

Validation of the visual body image classification in adolescent idiopathic scoliosis: a retrospective study.

A prospective study. To diagnose scoliosis, a visit to the hospital for radiography is typically necessary. In such cases, children with scoliosis are exposed to radiation, which may place their health at risk. Therefore, we sought to determine whether a classification method based on visual body images obtained through photography can be used to diagnose scoliosis. Scoliosis can be diagnosed and classified into various types using radiographs. However, no studies have attempted to classify scoliosis based on visual body images. From January 1, 2019 to December 31, 2022, 136 patients newly diagnosed with Adolescent idiopathic scoliosis and 124 healthy candidates from our institution were enrolled. This study classified body images into five types based on visual confirmation of the positional relationship of the body. The accuracy of this classification method was identified by calculating its sensitivity, specificity, and reproducibility of this classification method within and between observers according to kappa value. Overall, 136 patients and 124 control subjects who visited Pusan National University Hospital, Busan, Korea were photographed and compared by obtaining back images and X-ray radiographs. The sensitivity and specificity of the classification method showed a satisfactory-to-good degree of accuracy, although the degree varies depending on the visual body image type. The classification methods exhibited good intraobserver reliability (κ=0.855) and moderate interobserver reliability (κ=0.751). Our classification method showed a high degree of sensitivity and specificity (98.1% sensitivity, 98.9% specificity, and 98.4% accuracy) while exhibiting high reproducibility and ease of access. Based on our findings, we believe that our classification method can be used for scoliosis screening.

Novel Multimode Assay Based on Asymmetrically Competitive CRISPR and Raman Barcode Spectra for Multiple Hepatocellular Carcinoma Biomarkers Detection.

Commercial pregnancy test strips (PTS) possess the advantages of lower price, higher stability, and better repeatability and have been popularized to integrate with novel sensing strategies to detect other disease biomarkers, which accelerates the commercialization process of those novel sensing strategies. However, the current integration of novel sensing strategies into commercial PTS still faced the problems of insufficient quantification, low sensitivity, and lack of multiple detection capabilities. Hence, we proposed the concept of "visual classification recognition, spectral signal subdivision" for multiple hepatocellular carcinoma biomarkers (miRNA122 and miRNA233) detection with dual signals based on asymmetric competitive CRISPR (acCRISPR) and surface-enhanced Raman spectroscopy coupling with PTS, named the acCRISPR-PTS-SERS assay. In this assay, acCRISPR was used as a nonamplified cascaded signal amplification method to improve the sensitivity of detection. Two AuNPs-based core-shell Raman tags, each corresponding to different miRNA biomarkers, were used to achieve both visual recognition and spectral segmentation to enhance the quantification of PTS detection and the capability for multiple detection. Under the optimal conditions, the LOD for miRNA122 and miRNA223 were 10.36 and 4.65 fM, respectively. The sensitivity was enhanced by nearly 2 orders of magnitude. In the future, simultaneous hand-held detection for fingerprint barcodes of different cancers can be achieved with the assistance of a microfluidic chip and smartphone.

Classification of Diabetic Retinopathy Based on Efficient Computational Modeling

Convolutional neural networks (CNN) and Vision Transformers (ViT) have long been the main backbone networks for visual classification in the field of deep learning. Although ViT has recently received more attention than CNN due to its excellent fitting ability, their scalability is largely limited by the quadratic complexity of attention computation. For the determination of diabetic retinopathy, the fundus lesions as well as the width, angle, and branching pattern of retinal blood vessels are characterized, inspired by the ability of Mamba and VMamba to efficiently model long sequences, VMamba-m is proposed in this paper. This is a generalized visual skeleton model designed to reduce computational complexity to linear while retaining the advantageous features of ViTs. By modifying the cross-entropy loss function, we enhance the model’s attention to rare categories, especially in large-scale multi-category classification tasks. In order to enhance the adaptability of the VMamba-m model in processing visual data, we introduce the se channel attention mechanism, which enables the model to learn features in the channel dimension and form the importance of each channel. Finally, different weights are assigned to each channel through the incentive part. In addition to this, this paper further improves the implementation details and architectural design by introducing a novel attention mechanism implemented based on the local windowing method, which aims to optimize the model’s ability in processing long sequence data to enhance the performance of VMamba-m and improve its inference speed. Extensive experimental results show that VMamba-m performs well in the retinopathy V classification task, and it has significant advantages in terms of accuracy and computation time over existing benchmark models.