Advanced Deep Learning Models Research Articles

Comprehensive Survey on Disease Detection in Retinal Imaging through Deep Learning Techniques

Abstract: Millions of people around the world suffer from visual impairment and blindness that could have been prevented or diagnosed earlier. To tackle this issue, we’ve introduced a groundbreaking approach to detect multiple eye diseases using retinal imaging. Our study involves a unique combination of various advanced deep learning models through ensemble learning.We present a detailed examination of deep learning models in the context of multi-disease classification using the Retinal Fundus MultiDisease Image Dataset (RFMiD). The objective is to assess the performance of these models and identify the most effective one for accurate and reliable disease diagnosis.We begin by preprocessing the RFMiD dataset to enhance image quality and extract relevant features, ensuring a robust input for our models. Subsequently, each model undergoes a comprehensive training and finetuning process to optimize its parameters for disease classification. The evaluation metrics include accuracy, precision, recall, and F1 score, providing a comprehensive understanding of model performance.We discuss the strengths and weaknesses of each model, shedding light on factors influencing their performance. The insights gained from this comparative analysis can guide researchers and practitioners in selecting an appropriate model for retinal disease diagnosis based on specific requirements and constraints.In addition, we explore potential avenues for future research, including ensemble methods and hybrid architectures, to further improve classification accuracy. The paper includes discussion on the practical implications of our findings and their significance in advancing the field of medical image analysis, particularly in the context of retinal disease diagnosis.

Advanced deep learning models for phenotypic trait extraction and cultivar classification in lychee using photon-counting micro-CT imaging.

In contemporary agronomic research, the focus has increasingly shifted towards non-destructive imaging and precise phenotypic characterization. A photon-counting micro-CT system has been developed, which is capable of imaging lychee fruit at the micrometer level and capturing a full energy spectrum, thanks to its advanced photon-counting detectors. For automatic measurement of phenotypic traits, seven CNN-based deep learning models including AttentionUNet, DeeplabV3+, SegNet, TransUNet, UNet, UNet++, and UNet3+ were developed. Machine learning techniques tailored for small-sample training were employed to identify key characteristics of various lychee species. These models demonstrate outstanding performance with Dice, Recall, and Precision indices predominantly ranging between 0.90 and 0.99. The Mean Intersection over Union (MIoU) consistently falls between 0.88 and 0.98. This approach served both as a feature selection process and a means of classification, significantly enhancing the study's ability to discern and categorize distinct lychee varieties. This research not only contributes to the advancement of non-destructive plant analysis but also opens new avenues for exploring the intricate phenotypic variations within plant species.

Border Defence Mechanism Using Deep Learning Technique

In response to escalating concerns regarding potential border airstrikes, this paper introduces an advanced deep-learning model that utilizes Convolutional Neural Networks (CNNs) for accurate military aircraft classification. Functioning as a pivotal tool for proactive threat assessment, the system works by distinguishing and categorizing six aircraft types with exceptional accuracy, facilitating real-time analysis, and aiding with the development of responsive counter-attack strategies. The multi-stage process begins with the comprehensive collection of diverse datasets, followed by the extraction of spatial features from images using CNNs. Optimization techniques fine-tune model parameters to ensure optimal performance. This approach significantly strengthens national security by leveraging advanced technology for proactive threat mitigation. Emphasizing this paper’s vital role in enhancing border defence capabilities, it highlights the model’s capacity to navigate evolving security challenges through improved awareness of the environment. The model's ability to adapt to real-world conditions, including variations in lighting, weather, and terrain, demonstrates its practical applicability. For seamless implementation, the deep learning model is deployed using Django and SQLite as a web page providing an efficient and user-friendly interface This sophisticated deep learning model is fundamental in fortifying national security measures against potential threats along border regions.

Advanced Deep Learning Model for Predicting the Academic Performances of Students in Educational Institutions

Educational institutions are increasingly focused on supporting students who may be facing academic challenges, aiming to enhance their educational outcomes through targeted interventions. Within this framework, leveraging advanced deep learning techniques to develop recommendation systems becomes essential. These systems are designed to identify students at risk of underperforming by analyzing patterns in their historical academic data, thereby facilitating personalized support strategies. This research introduces an innovative deep learning model tailored for pinpointing students in need of academic assistance. Utilizing a Gated Recurrent Neural Network (GRU) architecture, the model is rich with features such as a dense layer, max-pooling layer, and the ADAM optimization method used to optimize performance. The effectiveness of this model was tested using a comprehensive dataset containing 15,165 records of student assessments collected across several academic institutions. A comparative analysis with existing educational recommendation models, like Recurrent Neural Network (RNN), AdaBoost, and Artificial Immune Recognition System v2, highlights the superior accuracy of the proposed GRU model, which achieved an impressive overall accuracy of 99.70%. This breakthrough underscores the model’s potential in aiding educational institutions to proactively support students, thereby mitigating the risks of underachievement and dropout.

Advancing glioma diagnosis: Integrating custom U-Net and VGG-16 for improved grading in MR imaging.

In the realm of medical imaging, the precise segmentation and classification of gliomas represent fundamental challenges with profound clinical implications. Leveraging the BraTS 2018 dataset as a standard benchmark, this study delves into the potential of advanced deep learning models for addressing these challenges. We propose a novel approach that integrates a customized U-Net for segmentation and VGG-16 for classification. The U-Net, with its tailored encoder-decoder pathways, accurately identifies glioma regions, thus improving tumor localization. The fine-tuned VGG-16, featuring a customized output layer, precisely differentiates between low-grade and high-grade gliomas. To ensure consistency in data pre-processing, a standardized methodology involving gamma correction, data augmentation, and normalization is introduced. This novel integration surpasses existing methods, offering significantly improved glioma diagnosis, validated by high segmentation dice scores (WT: 0.96, TC: 0.92, ET: 0.89), and a remarkable overall classification accuracy of 97.89%. The experimental findings underscore the potential of integrating deep learning-based methodologies for tumor segmentation and classification in enhancing glioma diagnosis and formulating subsequent treatment strategies.

An integrated deep-learning model for smart waste classification.

Efficient waste management is essential for human well-being and environmental health, as neglecting proper disposal practices can lead to financial losses and the depletion of natural resources. Given the rapid urbanization and population growth, developing an automated, innovative waste classification model becomes imperative. To address this need, our paper introduces a novel and robust solution - a smart waste classification model that leverages a hybrid deep learning model (Optimized DenseNet-121 + SVM) to categorize waste items using the TrashNet datasets. Our proposed approach uses the advanced deep learning model DenseNet-121, optimized for superior performance, to extract meaningful features from an expanded TrashNet dataset. These features are subsequently fed into a support vector machine (SVM) for precise classification. Employing data augmentation techniques further enhances classification accuracy while mitigating the risk of overfitting, especially when working with limited TrashNet data. The results of our experimental evaluation of this hybrid deep learning model are highly promising, with an impressive accuracy rate of 99.84%. This accuracy surpasses similar existing models, affirming the efficacy and potential of our approach to revolutionizing waste classification for a sustainable and cleaner future.

Characterizing Anti-Vaping Posts for Effective Communication on Instagram Using Multimodal Deep Learning.

Instagram is a popular social networking platform for sharing photos with a large proportion of youth and young adult users. We aim to identify key features in anti-vaping Instagram image posts associated with high social media user engagement by artificial intelligence. We collected 8972 anti-vaping Instagram image posts and hand-coded 2200 Instagram images to identify nine image features such as warning signs and person-shown vaping. We utilized a deep-learning model, the OpenAI: contrastive language-image pre-training with ViT-B/32 as the backbone and a 5-fold cross-validation model evaluation, to extract similar features from the Instagram image and further trained logistic regression models for multilabel classification. Latent Dirichlet Allocation model and Valence Aware Dictionary and sEntiment Reasoner were used to extract the topics and sentiment from the captions. Negative binomial regression models were applied to identify features associated with the likes and comments count of posts. Several features identified in anti-vaping Instagram image posts were significantly associated with high social media user engagement (likes or comments), such as educational warnings and warning signs. Instagram posts with captions about health risks associated with vaping received significantly more likes or comments than those about help quitting smoking or vaping. Compared to the model based on 2200 hand-coded Instagram image posts, more significant features have been identified from 8972 AI-labeled Instagram image posts. Features identified from anti-vaping Instagram image posts will provide a potentially effective way to communicate with the public about the health effects of e-cigarette use. Considering the increasing popularity of social media and the current vaping epidemic, especially among youth and young adults, it becomes necessary to understand e-cigarette-related content on social media. Although pro-vaping messages dominate social media, anti-vaping messages are limited and often have low user engagement. Using advanced deep-learning and statistical models, we identified several features in anti-vaping Instagram image posts significantly associated with high user engagement. Our findings provide a potential approach to effectively communicate with the public about the health risks of vaping to protect public health.

Enhancing Day-Ahead Cooling Load Prediction in Tropical Commercial Buildings Using Advanced Deep Learning Models: A Case Study in Singapore

Commercial buildings in hot and humid tropical climates rely significantly on cooling systems to maintain optimal occupant comfort. A well-accurate day-ahead load profile prediction plays a pivotal role in planning the energy requirements of cooling systems. Despite the pressing need for effective day-ahead cooling load predictions, current methodologies have not fully harnessed the potential of advanced deep-learning techniques. This paper aims to address this gap by investigating the application of innovative deep-learning models in day-ahead hourly cooling load prediction for commercial buildings in tropical climates. A range of multi-output deep learning techniques, including Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory networks (LSTMs), are employed to enhance prediction accuracy. Furthermore, these individual deep learning techniques are synergistically integrated to create hybrid models, such as CNN-LSTM and Sequence-to-Sequence models. Experiments are conducted to choose the time horizons from the past that can serve as input to the models. In addition, the influence of various categories of input parameters on prediction performance has been assessed. Historical cooling load, calendar features, and outdoor weather parameters are found in decreasing order of influence on prediction accuracy. This research focuses on buildings located in Singapore and presents a comprehensive case study to validate the proposed models and methodologies. The sequence-to-sequence model provided better performance than all the other models. It offered a CV-RMSE of 7.4%, 10%, and 6% for SIT@Dover, SIT@NYP, and the simulated datasets, which were 2.3%, 3%, and 1% less, respectively, than the base Deep Neural Network model.

Layer-wise multi-defect detection for laser powder bed fusion using deep learning algorithm with visual explanation

In Laser Powder Bed Fusion (LPBF), it is a major challenge to obtain detailed spatial information on different powder bed defects in real-time and simultaneously. Deep Learning (DL) algorithms under the field of Machine Learning (ML) have promoted the intelligent development of the powder bed defect detection method. However, they still need to be further evaluated in terms of detection accuracy and time delay, training data overhead, and model robustness under complex environments. Also, the DL model usually treated as a black-box demands further explanation. Herein, three advanced DL models are constructed using bounding boxes to locate multi-defects quickly and accurately for the powder spreading of LPBF. High detection accuracy is achieved with limited training samples through both data augmentation aiming at expanding image samples and model-based Transfer Learning (TL) used for transferring from the source domain to the target domain by reusing trained models. Further, the data augmentation method is employed to generate low-resolution images with interference to test the robustness of the detection model in harsh environments. Besides, visual feature maps and saliency maps were generated with the Detector Randomized Input Sampling for Explanation (D-RISE) method to help understand the validity of the defect detection process of the DL model. Overall, this work shows that the proposed multi-defect detection algorithms can provide comprehensive information on powder bed defects accurately and quickly.

DeepPRMS: advanced deep learning model to predict protein arginine methylation sites.

Protein methylation is a form of post-translational modifications of protein, which is crucial for various cellular processes, including transcription activity and DNA repair. Correctly predicting protein methylation sites is fundamental for research and drug discovery. Some experimental techniques, such as methyl-specific antibodies, chromatin immune precipitation and mass spectrometry, exist for predicting protein methylation sites, but these techniques are time-consuming and costly. The ability to predict methylation sites using in silico techniques may help researchers identify potential candidate sites for future examination and make it easier to carry out site-specific investigations and downstream characterizations. In this research, we proposed a novel deep learning-based predictor, named DeepPRMS, to identify protein methylation sites in primary sequences. The DeepPRMS utilizes the gated recurrent unit (GRU) and convolutional neural network (CNN) algorithms to extract the sequential and spatial information from the primary sequences. GRU is used to extract sequential information, while CNN is used for spatial information. We combined the latent representation of GRU and CNN models to have a better interaction among them. Based on the independent test data set, DeepPRMS obtained an accuracy of 85.32%, a specificity of 84.94%, Matthew's correlation coefficient of 0.71 and a sensitivity of 85.80%. The results indicate that DeepPRMS can predict protein methylation sites with high accuracy and outperform the state-of-the-art models. The DeepPRMS is expected to effectively guide future research experiments for identifying potential methylated protein sites. The web server is available at http://deepprms.nitsri.ac.in/.

Automated quality inspection of baby corn using image processing and deep learning

The food industry typically relies heavily on manual operations with high proficiency and skills. According to the quality inspection process, a baby corn with black marks or blemishes is considered a defect or unqualified class which should be discarded. Quality inspection and sorting of agricultural products like baby corn are labor-intensive and time-consuming. The main goal of this work is to develop an automated quality inspection framework to differentiate between ‘pass’ and ‘fail’ categories based on baby corn images. A traditional image processing method using a threshold principle is compared with relatively more advanced deep learning models. Particularly, Convolutional neural networks, specific sub-types of deep learning models, were implemented. Thorough experiments on choices of network architectures and their hyperparameters were conducted and compared. A Shapley additive explanations (SHAP) framework was further utilized for network interpretation purposes. The EfficientNetB5 networks with relatively larger input sizes yielded up to 99.06% accuracy as the best performance against 95.28% obtained from traditional image processing. Incorporating a region of interest identification, several model experiments, data application on baby corn images, and the SHAP framework are our main contributions. Our proposed quality inspection system to automatically differentiate baby corn images provides a potential pipeline to further support the agricultural production process.

TBACkp: HER2 expression status classification network focusing on intrinsic subenvironmental characteristics of breast cancer liver metastases

The HER2 expression status in breast cancer liver metastases is a crucial indicator for the diagnosis, treatment, and prognosis assessment of patients. And typical diagnosis involves assessing the HER2 expression status through invasive procedures like biopsy. However, this method has certain drawbacks, such as being difficult in obtaining tissue samples and requiring long examination periods. To address these limitations, we propose an AI-aided diagnostic model. This model enables rapid diagnosis. It diagnoses a patient's HER2 expression status on the basis of preprocessed images, which is the region of the lesion extracted from a CT image rather than from an actual tissue sample. The algorithm of the model adopts a parallel structure, including a Branch Block and a Trunk Block. The Branch Block is responsible for extracting the gradient characteristics between the tumor sub-environments, and the Trunk Block is for fusing the characteristics extracted by the Branch Block. The Branch Block contains CNN with self-attention, which combines the advantages of CNN and self-attention to extract more meticulous and comprehensive image features. And the Trunk Block is so designed that it fuses the extracted image feature information without affecting the transmission of the original image features. The Conv-Attention is used to calculate the attention in the Trunk Block, which uses kernel dot product and is responsible for providing the weight for the self-attention in the process of using convolution induced deviation calculation. Combined with the structure of the model and the method used, we refer to this model as TBACkp. The dataset comprises the enhanced abdominal CT images of 151 patients with liver metastases from breast cancer, together with the corresponding HER2 expression levels for each patient. The experimental results are as follows: (AUC: 0.915, ACC: 0.854, specificity: 0.809, precision: 0.863, recall: 0.881, F1-score: 0.872). The results demonstrate that this method can accurately assess the HER2 expression status in patients when compared with other advanced deep learning model.

Fall Recognition Based on Time-Level Decision Fusion Classification

We propose a vision-based fall detection algorithm using advanced deep learning models and fusion methods for smart safety management systems. By detecting falls through visual cues, it is possible to leverage existing surveillance cameras, thus minimizing the need for extensive additional equipment. Consequently, we developed a cost-effective fall detection system. The proposed system consists of four modules: object detection, pose estimation, action recognition, and result fusion. Constructing the fall detection system involved the utilization of state-of-the-art (SOTA) models. In the fusion module, we experimented with various approaches, including voting, maximum, averaging, and probabilistic fusion. Notably, we observed a significant performance improvement with the use of probabilistic fusion. We employed the HAR-UP dataset to demonstrate this enhancement, achieving an average 0.84% increase in accuracy compared to the baseline, which did not incorporate fusion methods. By applying our proposed time-level ensemble and skeleton-based fall detection approach, coupled with the use of enhanced object detection and pose estimation modules, we substantially improved the robustness and accuracy of the system, particularly for fall detection in challenging scenarios.

Comprehensive exploration of multi-modal and multi-branch imaging markers for autism diagnosis and interpretation: insights from an advanced deep learning model.

Autism spectrum disorder is a complex neurodevelopmental condition with diverse genetic and brain involvement. Despite magnetic resonance imaging advances, autism spectrum disorder diagnosis and understanding its neurogenetic factors remain challenging. We propose a dual-branch graph neural network that effectively extracts and fuses features from bimodalities, achieving 73.9% diagnostic accuracy. To explain the mechanism distinguishing autism spectrum disorder from healthy controls, we establish a perturbation model for brain imaging markers and perform a neuro-transcriptomic joint analysis using partial least squares regression and enrichment to identify potential genetic biomarkers. The perturbation model identifies brain imaging markers related to structural magnetic resonance imaging in the frontal, temporal, parietal, and occipital lobes, while functional magnetic resonance imaging markers primarily reside in the frontal, temporal, occipital lobes, and cerebellum. The neuro-transcriptomic joint analysis highlights genes associated with biological processes, such as "presynapse," "behavior," and "modulation of chemical synaptic transmission" in autism spectrum disorder's brain development. Different magnetic resonance imaging modalities offer complementary information for autism spectrum disorder diagnosis. Our dual-branch graph neural network achieves high accuracy and identifies abnormal brain regions and the neuro-transcriptomic analysis uncovers important genetic biomarkers. Overall, our study presents an effective approach for assisting in autism spectrum disorder diagnosis and identifying genetic biomarkers, showing potential for enhancing the diagnosis and treatment of this condition.

Graph autoencoder with mirror temporal convolutional networks for traffic anomaly detection

Traffic time series anomaly detection has been intensively studied for years because of its potential applications in intelligent transportation. However, classical traffic anomaly detection methods often overlook the evolving dynamic associations between road network nodes, which leads to challenges in capturing the long-term temporal correlations, spatial characteristics, and abnormal node behaviors in datasets with high periodicity and trends, such as morning peak travel periods. In this paper, we propose a mirror temporal graph autoencoder (MTGAE) framework to explore anomalies and capture unseen nodes and the spatiotemporal correlation between nodes in the traffic network. Specifically, we propose the mirror temporal convolutional module to enhance feature extraction capabilities and capture hidden node-to-node features in the traffic network. Morever, we propose the graph convolutional gate recurrent unit cell (GCGRU CELL) module. This module uses Gaussian kernel functions to map data into a high-dimensional space, and enables the identification of anomalous information and potential anomalies within the complex interdependencies of the traffic network, based on prior knowledge and input data. We compared our work with several other advanced deep-learning anomaly detection models. Experimental results on the NYC dataset illustrate that our model works best compared to other models for traffic anomaly detection.

Enhanced deep learning model enables accurate alignment measurement across diverse institutional imaging protocols

BackgroundAchieving consistent accuracy in radiographic measurements across different equipment and protocols is challenging. This study evaluates an advanced deep learning (DL) model, building upon a precursor, for its proficiency in generating uniform and precise alignment measurements in full-leg radiographs irrespective of institutional imaging differences.MethodsThe enhanced DL model was trained on over 10,000 radiographs. Utilizing a segmented approach, it separately identified and evaluated regions of interest (ROIs) for the hip, knee, and ankle, subsequently integrating these regions. For external validation, 300 datasets from three distinct institutes with varied imaging protocols and equipment were employed. The study measured seven radiologic parameters: hip-knee-ankle angle, lateral distal femoral angle, medial proximal tibial angle, joint line convergence angle, weight-bearing line ratio, joint line obliquity angle, and lateral distal tibial angle. Measurements by the model were compared with an orthopedic specialist's evaluations using inter-observer and intra-observer intraclass correlation coefficients (ICCs). Additionally, the absolute error percentage in alignment measurements was assessed, and the processing duration for radiograph evaluation was recorded.ResultsThe DL model exhibited excellent performance, achieving an inter-observer ICC between 0.936 and 0.997, on par with an orthopedic specialist, and an intra-observer ICC of 1.000. The model's consistency was robust across different institutional imaging protocols. Its accuracy was particularly notable in measuring the hip-knee-ankle angle, with no instances of absolute error exceeding 1.5 degrees. The enhanced model significantly improved processing speed, reducing the time by 30-fold from an initial 10–11 s to 300 ms.ConclusionsThe enhanced DL model demonstrated its ability for accurate, rapid alignment measurements in full-leg radiographs, regardless of protocol variations, signifying its potential for broad clinical and research applicability.

Can we trust our eyes? Interpreting the misperception of road safety from street view images and deep learning

Road safety is a critical concern that impacts both human lives and urban development, drawing significant attention from city managers and researchers. The perception of road safety has gained increasing research interest due to its close connection with the behavior of road users. However, safety isn't always as it appears, and there is a scarcity of studies examining the association and mismatch between road traffic safety and road safety perceptions at the city scale, primarily due to the time-consuming nature of data acquisition. In this study, we applied an advanced deep learning model and street view images to predict and map human perception scores of road safety in Manhattan. We then explored the association and mismatch between these perception scores and traffic crash rates, while also interpreting the influence of the built environment on this disparity. The results showed that there was heterogeneity in the distribution of road safety perception scores. Furthermore, the study found a positive correlation between perception scores and crash rates, indicating that higher perception scores were associated with higher crash rates. In this study, we also concluded four perception patterns: “Safer than it looks”, “Safe as it looks”, “More dangerous than it looks”, and “Dangerous as it looks”. Wall view index, tree view index, building view index, distance to the nearest traffic signals, and street width were found to significantly influence these perception patterns. Notably, our findings underscored the crucial role of traffic lights in the “More dangerous than it looks” pattern. While traffic lights may enhance people's perception of safety, areas in close proximity to traffic lights were identified as potentially accident-prone regions.

Deep learning-based chatbot by natural language processing for supportive risk management in river dredging projects

Dredging projects play a vital role in water conservation, especially for flood control, but they often encounter natural and human-induced risks. This study addresses these challenges by creating a comprehensive knowledge base for dredging risks. This knowledge base serves several essential functions: it aids in immediate and thorough risk prevention, provides practical risk management recommendations, and assists dredging personnel in making informed decisions when dealing with risks. Furthermore, the study introduces an advanced deep-learning model that enhances access to critical information. Users can quickly find relevant risk prevention strategies by inputting keywords. The model's distinctive feature is its ability to analyze unforeseen risks through natural language processing, predict the potential impact and frequency of unexpected events, and propose appropriate risk response measures. This empowers dredging personnel to conduct preliminary risk assessments confidently. The deep learning model seamlessly integrates into LINE's communication platform, creating a river dredging project risk knowledge chatbot. This user-friendly chatbot is accessible to personnel at all project stages, enabling real-time interaction. It offers a practical way to develop and implement risk prevention plans and response measures. In summary, this research presents an innovative approach that enhances the efficiency and safety of dredging projects. By merging a knowledge base with cutting-edge technology and real-time communication, it equips dredging personnel to manage known and unforeseen risks proficiently, ultimately contributing to the success of water conservancy and flood control projects.

Mathematical Modelling and Deep Learning: Innovations in E-Commerce Sentiment Analysis

This research explores e-commerce dynamics, focusing on the challenge of predicting customer churn using deep learning [65]. It integrates and analyses both textual and transactional data, including social media posts and customer feedback [59]. The approach uses an advanced deep learning model, involving data collection, pre-processing, and feature extraction [40]. Novel methods fuse data to create a detailed customer profile combining sentiment analysis with behavioural insights derived from transaction data [25]. The deep learning architecture is designed to analyse and predict customer sentiments and purchasing behaviours, informed by the latest research [65]. This study is significant as it provides an innovative solution for predicting customer churn in e-commerce, aiding sustainability [45]. It also enables targeted retention strategies and personalized customer engagement [59]. Additionally, it contributes insights to big data analytics and customer relationship management in e-commerce, showcasing deep learning's potential in transforming business practices and enhancing customer experience [40].

CATNet: Cross-modal fusion for audio–visual speech recognition

Automatic speech recognition (ASR) is a typical pattern recognition technology that converts human speeches into texts. With the aid of advanced deep learning models, the performance of speech recognition is significantly improved. Especially, the emerging Audio–Visual Speech Recognition (AVSR) methods achieve satisfactory performance by combining audio-modal and visual-modal information. However, various complex environments, especially noises, limit the effectiveness of existing methods. In response to the noisy problem, in this paper, we propose a novel cross-modal audio–visual speech recognition model, named CATNet. First, we devise a cross-modal bidirectional fusion model to analyze the close relationship between audio and visual modalities. Second, we propose an audio–visual dual-modal network to preprocess audio and visual information, extract significant features and filter redundant noises. The experimental results demonstrate the effectiveness of CATNet, which achieves excellent WER, CER and converges speeds, outperforms other benchmark models and overcomes the challenge posed by noisy environments.