Layers Of Convolutional Neural Network Research Articles

Heart Signal Analysis Using Multistage Classification Denoising Model

Cardiovascular disease is a major cause of death worldwide, and the COVID-19 pandemic has only made the situation worse. The purpose of this work is to explore various time-frequency analysis methods that can be used to classify heart sound signals and identify multiple abnormalities in the heart, such as aortic stenosis, mitral stenosis, and mitral valve prolapse. The signal has been modified using three techniques—tunable quality wavelet transform (TQWT), discrete wavelet transform (DWT), and empirical mode decomposition—to detect heart signal abnormality. The proposed model detects heart signal abnormality at two stages, the user end and the clinical end. At the user end, binary classification of signals is performed, and if signals are abnormal then further classification is done at the clinic. The approach starts with signal preprocessing and uses the discrete wavelet transform (DWT) coefficients to train the hybrid model, which consists of one long short-term memory (LSTM) network layer and three convolutional neural network (CNN) layers. This method produced comparable results, with a 98.9% classification accuracy for signals, through the utilization of the CNN and LSTM model. Combining the CNN’s skill in feature extraction with the LSTM’s capacity to record time-dependent features improves the efficacy of the model. Identifying issues early and initiating appropriate medication can alleviate the burden associated with heart valve diseases.

Machine Learning Models for Real-Time Traffic Prediction: A Case Study in Urban Traffic Management

This study introduces and evaluates the Long-term Traffic Prediction Network (LTPN), a specialized machine learning framework designed for real-time traffic prediction in urban environments. Utilizing a unique combination of convolutional and recurrent neural network layers, the LTPN model consistently outperforms established predictive models across various metrics. It demonstrates significantly lower error rates in both short and long-term traffic forecasts, highlighting its superior accuracy and reliability. The effectiveness of the LTPN model is underscored by its robust performance under diverse traffic conditions, making it a promising tool for enhancing the efficiency and responsiveness of intelligent transportation systems (ITS). This paper details the model's architecture, training processes, and a comprehensive comparison of its predictive capabilities against traditional models, providing clear evidence of its advantages in real-world applications.

Amharic spoken digits recognition using convolutional neural network

Spoken digits recognition (SDR) is a type of supervised automatic speech recognition, which is required in various human–machine interaction applications. It is utilized in phone-based services like dialing systems, certain bank operations, airline reservation systems, and price extraction. However, the design of SDR is a challenging task that requires the development of labeled audio data, the proper choice of feature extraction method, and the development of the best performing model. Even if several works have been done for various languages, such as English, Arabic, Urdu, etc., there is no developed Amharic spoken digits dataset (AmSDD) to build Amharic spoken digits recognition (AmSDR) model for the Amharic language, which is the official working language of the government of Ethiopia. Therefore, in this study, we developed a new AmSDD that contains 12,000 utterances of 0 (Zaero) to 9 (zet’enyi) digits which were recorded from 120 volunteer speakers of different age groups, genders, and dialects who repeated each digit ten times. Mel frequency cepstral coefficients (MFCCs) and Mel-Spectrogram feature extraction methods were used to extract trainable features from the speech signal. We conducted different experiments on the development of the AmSDR model using the AmSDD and classical supervised learning algorithms such as Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Random Forest (RF) as the baseline. To further improve the performance recognition of AmSDR, we propose a three layers Convolutional Neural Network (CNN) architecture with Batch normalization. The results of our experiments show that the proposed CNN model outperforms the baseline algorithms and scores an accuracy of 99% and 98% using MFCCs and Mel-Spectrogram features, respectively.

MRNA-CLA: An interpretable deep learning approach for predicting mRNA subcellular localization

Messenger RNA (mRNA) is vital for post-transcriptional gene regulation, acting as the direct template for protein synthesis. However, the methods available for predicting mRNA subcellular localization need to be improved and enhanced. Notably, few existing algorithms can annotate mRNA sequences with multiple localizations. In this work, we propose the mRNA-CLA, an innovative multi-label subcellular localization prediction framework for mRNA, leveraging a deep learning approach with a multi-head self-attention mechanism. The framework employs a multi-scale convolutional layer to extract sequence features across different regions and uses a self-attention mechanism explicitly designed for each sequence. Paired with Position Weight Matrices (PWMs) derived from the convolutional neural network layers, our model offers interpretability in the analysis. In particular, we perform a base-level analysis of mRNA sequences from diverse subcellular localizations to determine the nucleotide specificity corresponding to each site. Our evaluations demonstrate that the mRNA-CLA model substantially outperforms existing methods and tools.

Translating Gestures: Utilizing CNN to Enhance ASL Communication and Understanding

This research introduces a system for translating American Sign Language (ASL) utilising Convolutional Neural Networks. (CNNs). The system under consideration has the capability to identify manual gestures executed by persons communicating in American Sign Language (ASL) and subsequently convert them into written language, thereby facilitating uninterrupted communication between the deaf and hearing populations. The CNN model was validated and trained using a set of data made up of 78,300 pictures of movements in the ASL Alphabet written in American Sign Language (ASL). In order to improve the performance of the model, the pre-processing of the data included a number of stages, such as converting the photos to grayscale, normalising the pixel values, and enhancing performance of the model. In order to ease classification, fully connected layers were added after a succession of pooling and convolutional layers in CNN's design. After 15 epochs of training, the model attained a validation accuracy of 99.85%. The findings of this research demonstrate the viability of employing Convolutional Neural Networks (CNNs) in the creation of precise and effective American Sign Language (ASL) translation systems, which can serve as a means of communication for people with auditory impairments.

Yoga Posture Recognition Through Computer Vision and Deep Learning

Abstract: India is the place where yoga's science and discipline originated 5,000 years ago. The development of it in ancient India occurred through the Indus-Sarasvati civilization. The Indian subcontinent is where this ancient practice originates,and it has been cherished for its multifaceted benefits that encompass physical, mental, and spiritual. Through asana, meditation, and other breathing techniques, harmony is brought to both body and mind. It also brings peace to the mind. The rise in stress in today's modern lifestyle has led to a global growth in the popularity of yoga. The goal of yoga is to mechanically align the body with effort on the muscles, ligaments, and joints to achieve optimal posture. If the asanas are not performed properly, strain in the joints, ligaments, and backbone can occur, which can have an impact on the hip joints. Therefore, it is vital to maintain correct yoga postures while performing various asanas. The availability of yoga posture prediction and automatic movement analysis is now possible due to the development of computer vision algorithms and sensors. In this work, a framework for recognizing a yoga posture from an image has been built using deep learning techniques such as convolutional neural networks (CNN) and machine learning (ML). The CNN layer is used to extract characteristics from the keypoints, and it is succeeded by a large short-term recollection that recognizes the sequence in which a set of frames occurs in order to make predictions. The stances are then categorized as either correct or inappropriate. The system will provide the user with feedback and display the correctness of posture if the right attitude is detected. This review paper synthesizes the current state-of-the-art in the field of yogic intelligence, focusing on the application of computer vision and deep learning techniques for recognizing yoga postures

AI-algorithm training and validation for identification of endometrial CD138+ cells in infertility-associated conditions; polycystic ovary syndrome (PCOS) and recurrent implantation failure (RIF)

BackgroundEndometrial CD138+ plasma cells serve as a diagnostic biomarker for endometrial inflammation, and their elevated occurrence correlates positively with adverse pregnancy outcomes. Infertility-related conditions like polycystic ovary syndrome (PCOS) and recurrent implantation failure (RIF) are closely associated with systemic and local chronic inflammatory status, wherein endometrial CD138+ plasma cell accumulation could also contribute to endometrial pathology. Current methods for quantifying CD138+ cells typically involve laborious and time-consuming microscopic assessments of only a few random areas from a slide. These methods have limitations in accurately representing the entire slide and are susceptible to significant biases arising from intra- and interobserver variations. Implementing artificial intelligence (AI) for CD138+ cell identification could enhance the accuracy, reproducibility, and reliability of analysis. MethodsHere, an AI algorithm was developed to identify CD138+ plasma cells within endometrial tissue. The AI model comprised two layers of convolutional neural networks (CNNs). CNN1 was trained to segment epithelium and stroma across 28,363 mm2 (2.56 mm2 of epithelium and 24.87 mm2 of stroma), while CNN2 was trained to distinguish stromal cells based on CD138 staining, encompassing 7345 cells in the object layers (6942 CD138− cells and 403 CD138+ cells). The training and performance of the AI model were validated by three experienced pathologists. We collected 193 endometrial tissues from healthy controls (n = 73), women with PCOS (n = 91), and RIF patients (n = 29) and compared the CD138+ cell percentages based on cycle phases, ovulation status, and endometrial receptivity utilizing the AI model. ResultsThe AI algorithm consistently and reliably distinguished CD138− and CD138+ cells, with total error rates of 6.32% and 3.23%, respectively. During the training validation, there was a complete agreement between the decisions made by the pathologists and the AI algorithm, while the performance validation demonstrated excellent accuracy between the AI and human evaluation methods (intraclass correlation; 0.76, 95% confidence intervals; 0.36–0.93, p = 0.002) and a positive correlation (Spearman's rank correlation coefficient: 0.79, p < 0.01). In the AI analysis, the AI model revealed higher CD138+ cell percentages in the proliferative phase (PE) endometrium compared to the secretory phase or anovulatory PCOS endometrium, irrespective of PCOS diagnosis. Interestingly, CD138+ percentages differed according to PCOS phenotype in the PE (p = 0.03). On the other hand, the receptivity status had no impact on the cell percentages in RIF samples. ConclusionOur findings emphasize the potential and accuracy of the AI algorithm in detecting endometrial CD138+ plasma cells, offering distinct advantages over manual inspection, such as rapid analysis of whole slide images, reduction of intra- and interobserver variations, sparing the valuable time of trained specialists, and consistent productivity. This supports the application of AI technology to help clinical decision-making, for example, in understanding endometrial cycle phase-related dynamics, as well as different reproductive disorders.

A hybrid ensemble optimized BiGRU method for short-term photovoltaic generation forecasting

In the context of prominent energy crisis, photovoltaic power (PV) generation has received increasing attention, then accurate PV generation forecasting is crucial for ensuring the smooth operation of power stations. However, existing research is insufficient in comprehensively analyzing the impact of PV generation volatility. To fill the gaps and enhance the prediction accuracy, this paper proposes a new hybrid forecasting method. We first introduce the Locally Weighted Scatterplot Smoothing (LOWESS) method to process the data and enhance the data stability, and use Pearson correlation coefficient (PCC) and Random Forests (RF) for feature selection to improve the quality of input data. Then we use Attention mechanism and Convolutional Neural Network (CNN) layer to optimize Bi-directional Gate Recurrent Unit (BiGRU) model and form a new hybrid model. Finally, based on the Bagging algorithm, we use ensemble learning to further optimize the hybrid BiGRU model to enhance the depth and performance. The proposed method is validated through case analysis results from two different locations, Xuhui District in Shanghai, China and the DKASC area in Alice Springs, Australia. The results demonstrate that, compared with other models, the developed method exhibits exceptional prediction performance and effectively enhances the accuracy of PV generation forecasting. Keywords: photovoltaic power generation; Locally Weighted Scatterplot Smoothing; feature selection; ensemble learning; Bi-directional Gate Recurrent Unit.

Single image dehazing using attention layers in convolutional neural networks

A hazy image is characterized by atmospheric conditions that reduce the image’s clarity and contrast, thereby making it less visible. This degradation in image quality can hinder the performance of advanced computer vision tasks such as object detection and identifying open spaces which need to perform with high accuracy in important real world applications such as security surveillance and autonomous driving. In the recent past, the use of deep learning in image processing tasks have shown a remarkable improvement in performance, in particular, Convolutional Neural Networks (CNNs) perform superior to any other type of neural network in image related tasks. In this paper, we propose the addition of Channel Attention and Pixel Attention layers to four state-of-the-art CNNs, namely, GMAN, U-Net, 123-CEDH and DMPHN, used for the task of image dehazing. We show that the addition of these layers yields a non-trivial improvement on the quality of the dehazed images which we show qualitatively with examples and quantitatively by obtaining PSNR and SSIM scores of 28.63 and 0.959 respectively. Through the experiments, we show that the addition of the mentioned attention layers to the GMAN architecture yields the best results.

On the effects of recursive convolutional layers in convolutional neural networks

The Recursive Convolutional Layer (RCL) is a module that wraps a recursive feedback loop around a convolutional layer (CL). The RCL has been proposed to address some of the shortcomings of Convolutional Neural Networks (CNNs), as its unfolding increases the depth of a network without increasing the number of weights. We investigated the “naïve” substitution of CL with RCL on three base models: a 4-CL model, ResNet, DenseNet and their RCL-ized versions: C-FRPN, R-ResNet, and R-DenseNet using five image classification datasets. We find that this one-to-one replacement significantly improves the performances of the 4-CL model, but not those of ResNet or DenseNet. This led us to investigate the implication of the RCL substitution on the 4-CL model which reveals, among a number of properties, that RCLs are particularly efficient in shallow CNNs. We proceeded to re-visit the first set of experiments by gradually transforming the 4-CL model and the C-FRPN into respectively ResNet and R-ResNet, and find that the performance improvement is largely driven by the training regime whereas any depth increase negatively impacts the RCL-ized version. We conclude that the replacement of CLs by RCLs shows great potential in designing high-performance shallow CNNs.

Research on traffic flow prediction method based on adaptive multi-channel graph convolutional neural networks

In order to address the issues of predefined adjacency matrices inadequately representing information in road networks, insufficiently capturing spatial dependencies of traffic networks, and the potential problem of excessive smoothing or neglecting initial node information as the layers of graph convolutional neural networks increase, thus affecting traffic prediction performance, this paper proposes a prediction model based on Adaptive Multi-channel Graph Convolutional Neural Networks (AMGCN). The model utilizes an adaptive adjacency matrix to automatically learn implicit graph structures from data, introduces a mixed skip propagation graph convolutional neural network model, which retains the original node states and selectively acquires outputs of convolutional layers, thus avoiding the loss of node initial states and comprehensively capturing spatial correlations of traffic flow. Finally, the output is fed into Long Short-Term Memory networks to capture temporal correlations. Comparative experiments on two real datasets validate the effectiveness of the proposed model.

Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks.

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric views (e.g. left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

Non-symmetric over-time pooling using pseudo-grouping functions for convolutional neural networks

Convolutional Neural Networks (CNNs) are a family of networks that have become state-of-the-art in several fields of artificial intelligence due to their ability to extract spatial features. In the context of natural language processing, they can be used to build text classification models based on textual features between words. These networks fuse local features to generate global features in their over-time pooling layers. These layers have been traditionally built using the maximum function or other symmetric functions such as the arithmetic mean. It is important to note that the order of input local features is significant (i.e. the symmetry is not an inherent characteristic of the model). While this characteristic is appropriate for image-oriented CNNs, where symmetry might make the network robust to image rigid transformations, it seems counter-productive for text processing, where the order of the words is certainly important. Our proposal is, hence, to use non-symmetric pooling operators to replace the maximum or average functions. Specifically, we propose to perform over-time pooling using pseudo-grouping functions, a family of non-symmetric aggregation operators that generalize the maximum function. We present a construction method for pseudo-grouping functions and apply different examples of this family to over-time pooling layers in text-oriented CNNs. Our proposal is tested on seven different models and six different datasets in the context of engineering applications, e.g. text classification. The results show an overall improvement of the models when using non-symmetric pseudo-grouping functions over the traditional pooling function.

TriConvUNeXt: A Pure CNN-Based Lightweight Symmetrical Network for Biomedical Image Segmentation.

Biomedical image segmentation is essential in clinical practices, offering critical insights for accurate diagnosis and strategic treatment approaches. Nowadays, self-attention-based networks have achieved competitive performance in both natural language processing and computer vision, but the computational cost has reduced their popularity in practical applications. The recent study of Convolutional Neural Network (CNN) explores linear functions within modified CNN layer demonstrating pure CNN-based networks can still achieve competitive results against Vision Transformer (ViT) in biomedical image segmentation, with fewer parameters. The modified CNN, i.e., Depthwise CNN, however, leaves the features cleaved off in the channel dimension and prevents the extraction of features in the perspective of channel interaction. To effectively further explore the feature learning power of modified CNN with biomedical image segmentation, we design a lightweight multi-convolutional multi-scale convolutional network block (MSConvNeXt) for U-shape symmetrical network. Specifically, a network block consisting of a depthwise CNN, a deformable CNN, and a dilated CNN is proposed to capture semantic feature information effectively while with low computational cost. Furthermore, channel shuffling operation is proposed to dynamically promote an efficient feature fusion among the feature maps. The network block hereby is properly deployed within U-shape symmetrical encoder-decoder style network, named TriConvUNeXt. The proposed network is validated on a public benchmark dataset with a comprehensive evaluation in both computational cost and segmentation performance against 13 baseline methods. Specifically, TriConvUNeXt achieves 1% higher than UNet and TransUNet in Dice-Coefficient while 81% and 97% lower in computational cost. The implementation of TriConvUNeXt is made publicly accessible via https://github.com/ziyangwang007/TriConvUNeXt .

Study of an Optimization Tool Avoided Bias for Brain-Computer Interfaces Using a Hybrid Deep Learning Model

ObjectiveThis study addresses the challenge of user-specific bias in Brain-Computer Interfaces (BCIs) by proposing a novel methodology. The primary objective is to employ a hybrid deep learning model, combining 2D Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) layers, to analyze EEG signals and classify imagined tasks. The overarching goal is to create a generalized model that is applicable to a broader population and mitigates user-specific biases. Materials and MethodsEEG signals from imagined motor tasks in the public dataset Physionet form the basis of the study. This is due to the need to use other databases in addition to the BCI competition. A model of arrays emulating the electrode arrangement in the head is proposed to capture spatial information using CNN, and LSTM algorithms are used to capture temporal information, followed by signal classification. ResultsThe hybrid model is implemented to achieve a high classification rate, reaching up to 90% for specific users and averaging 74.54%. Error detection thresholds are set to eliminate subjects with low task affinity, resulting in a significant improvement in classification accuracy of up to 21.34%. ConclusionThe proposed methodology makes a significant contribution to the BCI field by providing a generalized system trained on diverse user data that effectively captures spatial and temporal EEG signal features. This study emphasizes the value of the hybrid model in advancing BCIs, highlighting its potential for improved reliability and accuracy in human-computer interaction. It also suggests the exploration of additional advanced layers, such as transformers, to further enhance the proposed methodology.

ADS-CNN: Adaptive Dataflow Scheduling for lightweight CNN accelerator on FPGAs

Lightweight convolutional neural networks (CNNs) enable lower inference latency and data traffic, facilitating deployment on resource-constrained edge devices such as field-programmable gate arrays (FPGAs). However, CNNs inference requires access to off-chip synchronous dynamic random-access memory (SDRAM), which significantly degrades inference speed and system power efficiency. In this paper, we propose an adaptive dataflow scheduling method for lightweight CNN accelerator on FPGAs named ADS-CNN. The key idea of ADS-CNN is to efficiently utilize on-chip resources and reduce the amount of SDRAM access. To achieve the reuse of logical resources, we design a time division multiplexing calculation engine to be integrated in ADS-CNN. We implement a configurable module for the convolution controller to adapt to the data reuse of different convolution layers, thus reducing the off-chip access. Furthermore, we exploit on-chip memory blocks as buffers based on the configuration of different layers in lightweight CNNs. On the resource-constrained Intel CycloneV SoC 5CSEBA6 FPGA platform, we evaluated six common lightweight CNN models to demonstrate the performance advantages of ADS-CNN. The evaluation results indicate that, compared with accelerators that use traditional tiling strategy dataflow, our ADS-CNN can achieve up to 1.29× speedup with the overall dataflow scale compression of 23.7%.

Multiclass classification of diseased grape leaf identification using deep convolutional neural network(DCNN) classifier

The cultivation of grapes encounters various challenges, such as the presence of pests and diseases, which have the potential to considerably diminish agricultural productivity. Plant diseases pose a significant impediment, resulting in diminished agricultural productivity and economic setbacks, thereby affecting the quality of crop yields. Hence, the precise and timely identification of plant diseases holds significant importance. This study employs a Convolutional neural network (CNN) with and without data augmentation, in addition to a DCNN Classifier model based on VGG16, to classify grape leaf diseases. A publicly available dataset is utilized for the purpose of investigating diseases affecting grape leaves. The DCNN Classifier Model successfully utilizes the strengths of the VGG16 model and modifies it by incorporating supplementary layers to enhance its performance and ability to generalize. Systematic evaluation of metrics, such as accuracy and F1-score, is performed. With training and test accuracy rates of 99.18 and 99.06%, respectively, the DCNN Classifier model does a better job than the CNN models used in this investigation. The findings demonstrate that the DCNN Classifier model, utilizing the VGG16 architecture and incorporating three supplementary CNN layers, exhibits superior performance. Also, the fact that the DCNN Classifier model works well as a decision support system for farmers is shown by the fact that it can quickly and accurately identify grape diseases, making it easier to take steps to stop them. The results of this study provide support for the reliability of the DCNN classifier model and its potential utility in the field of agriculture.

From explanation to intervention: Interactive knowledge extraction from Convolutional Neural Networks used in radiology.

Deep Learning models such as Convolutional Neural Networks (CNNs) are very effective at extracting complex image features from medical X-rays. However, the limited interpretability of CNNs has hampered their deployment in medical settings as they failed to gain trust among clinicians. In this work, we propose an interactive framework to allow clinicians to ask what-if questions and intervene in the decisions of a CNN, with the aim of increasing trust in the system. The framework translates a layer of a trained CNN into a measurable and compact set of symbolic rules. Expert interactions with visualizations of the rules promote the use of clinically-relevant CNN kernels and attach meaning to the rules. The definition and relevance of the kernels are supported by radiomics analyses and permutation evaluations, respectively. CNN kernels that do not have a clinically-meaningful interpretation are removed without affecting model performance. By allowing clinicians to evaluate the impact of adding or removing kernels from the rule set, our approach produces an interpretable refinement of the data-driven CNN in alignment with medical best practice.

Deep Learning-Based Leaf Disease Detection in Crop Using Images for Agricultural Application

The "Leaf Disease Detection" system addresses the critical challenge of plant diseases in agriculture through the implementation of an automated solution leveraging deep learning techniques. In this comprehensive endeavor, convolutional neural networks (CNNs), specifically DenseNet-121, ResNet-50, VGG-16, and Inception V4, are fine-tuned for efficient and accurate identification of plant diseases. The project utilizes the Plant Village dataset, encompassing 54,305 images across 38 plant disease classes, to conduct a comparative analysis of model performance. DenseNet-121 emerged as the top-performing model, achieving an exceptional 99.81% classification accuracy, surpassing other state-of-the-art models. The system's methodology strategically employs transfer learning to overcome computational challenges associated with training deep CNN layers. This approach, coupled with the multi-class classification strategy, proves robust in handling diverse plant species and diseases within each class. The results highlight the superior efficiency of transfer learning in comparison to building models from scratch, showcasing the potential for real-world applications in agriculture. The system's success is attributed to the careful optimization of hyper parameters and the adoption of advanced deep learning techniques, offering a promising avenue for automated and accurate plant disease detection, with implications for improving agricultural practices, minimizing economic losses, and ensuring global food security.

Visual Recognition Memory of Scenes Is Driven by Categorical, Not Sensory, Visual Representations.

When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene from three similar lures (e.g., three airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as forced-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations.