Deep Architecture Research Articles

Double-Condensing Attention Condenser: Leveraging Attention in Deep Learning to Detect Skin Cancer from Skin Lesion Images

Skin cancer is the most common type of cancer in the United States and is estimated to affect one in five Americans. Recent advances have demonstrated strong performance on skin cancer detection, as exemplified by state of the art performance in the SIIM-ISIC Melanoma Classification Challenge; however, these solutions leverage ensembles of complex deep neural architectures requiring immense storage and computation costs, and therefore may not be tractable. A recent movement for TinyML applications is integrating Double-Condensing Attention Condensers (DC-AC) into a self-attention neural network backbone architecture to allow for faster and more efficient computation. This paper explores leveraging an efficient self-attention structure to detect skin cancer in skin lesion images and introduces a deep neural network design with DC-AC customized for skin cancer detection from skin lesion images. We demonstrate that our approach with only 1.6 million parameters and 0.32 GFLOPs achieves better performance compared to traditional architecture designs as it obtains an area under the ROC curve of 0.90 on the public ISIC 2020 test set and 0.89 on the private ISIC test set, over 0.13 above the best Cancer-Net SCa network architecture design. The final model is publicly available as a part of a global open-source initiative dedicated to accelerating advancement in machine learning to aid clinicians in the fight against cancer. Future work of this research includes iterating on the design of the selected network architecture and refining the approach to generalize to other forms of cancer.

Optimizing deep neural network architectures for renewable energy forecasting

Optimizing deep neural network architectures for renewable energy forecasting

A Review of an Effective Deep Learning Network for Estimating COVID-19 severity from chest X-ray Imagery: Co Severity-Net

The COVID-19 pandemic has become an integral part of our existence and shows no signs of abating. In 2020, the World Health Organization declared it a pandemic, and its effects have impacted our lives in myriad ways. To detect COVID-19 from Chest X-Ray images, researchers have developed deep learning techniques. Given the severity of the disease, scoring the level of COVID-19 infection can aid in determining the optimal course of treatment and care for patients, as not all COVID-19 positive patients require specialized medical attention. However, the limited availability of a large-scale dataset has resulted in few studies being conducted to estimate the severity of the disease from Chest X-Ray images. Consequently, we propose CoV Severity-Net, a deep learning-based architecture trained on a public COVID-19 dataset curated by experienced radiologists for severity estimation. Our proposal is based on a review of 18 papers from reputable sources such as Google Scholar, PubMed, and Science Direct.

Probabilistic oil price forecasting with a variational mode decomposition-gated recurrent unit model incorporating pinball loss

Prediction methods have garnered significant attention in intelligent decision-making. Most existing approaches to predicting crude oil prices prioritize accuracy and stability while providing precise prediction intervals (PIs) that can offer valuable insights. Thus far, we introduce a novel hybrid model to forecast future crude oil prices. Our approach leverages the variational mode decomposition (VMD) to simplify the complexity of the original time series, yielding a set of subseries. These subseries are then modeled using a deep neural network architecture called a gated recurrent unit (GRU). To address the prediction uncertainty, we employed the pinball loss function rather than the mean square error (MSE) to guide the proposed VMD-GRU. This adaptation extends the traditional GRU-based point forecasting to probabilistic forecasting by estimating quantiles. We evaluated our proposed model on a well-established crude oil price series by conducting both single- and multi-step-ahead forecasting analyses. Our findings underscore the efficacy of the combined model, demonstrating its superior predictive performance compared to benchmark models.

Multi-point temperature or humidity prediction for office building indoor environment based on CGC-BiLSTM deep neural network

The aim of this study is to predict the temperature or humidity changes at multiple relevant points in a building using a deep neural network architecture with multi-task learning to provide more reference information for the design and optimal operation of heating and ventilation systems. For this purpose, traditional multi-task prediction algorithm architecture is combined with Customized Gate Control and other neural networks to build deep neural network architectures for indoor environments with multi-point temperature or humidity prediction tasks. To test the prediction effectiveness of the architecture, a task of predicting temperature or humidity 24h in advance was designed on a real office building indoor environment dataset, and the prediction results were compared with other single-task and multi-task prediction models. Two experimental conditions were designed for this study, one using the complete training set and the other reducing the training set at a certain point. Through the final prediction results, it is found that the multi-task prediction architecture used in this paper shows better or nearly optimal results compared to other prediction models under both working conditions. This study provides some reference value for the application of multi-task prediction algorithms to the task of predicting indoor environments in buildings.

CGLight: An effective indoor illumination estimation method based on improved convmixer and GauGAN

Illumination consistency is a key factor for seamlessly integrating virtual objects with real scenes in augmented reality (AR) systems. High dynamic range (HDR) panoramic images are widely used to estimate scene lighting accurately. However, generating environment maps requires complex deep network architectures, which cannot operate on devices with limited memory space. To address this issue, this paper proposes CGLight, an effective illumination estimation method that predicts HDR panoramic environment maps from a single limited field-of-view (LFOV) image. We first design a CMAtten encoder to extract features from input images, which learns the spherical harmonic (SH) lighting representation with fewer model parameters. Guided by the lighting parameters, we train a generative adversarial network (GAN) to generate HDR environment maps. In addition, to enrich lighting details and reduce training time, we specifically introduce the color consistency loss and independent discriminator, considering the impact of color properties on the lighting estimation task while improving computational efficiency. Furthermore, the effectiveness of CGLight is verified by relighting virtual objects using the predicted environment maps, and the root mean square error and angular error are 0.0494 and 4.0607 in the gray diffuse sphere, respectively. Extensive experiments and analyses demonstrate that CGLight achieves a balance between indoor illumination estimation accuracy and resource efficiency, attaining higher accuracy with nearly 4 times fewer model parameters than the ViT-B16 model.

Deep learning-enhanced remote sensing-integrated crop modeling for rice yield prediction

This study introduces a novel crop modeling approach based on cutting-edge computational tools to advance crop production monitoring methodologies, and, thereby, tackle global food security issues. Our approach pioneers integrating deep learning and remote sensing with process-based crop models to enhance rice yield predictions while leveraging the strengths and weaknesses of each model. We developed and evaluated four models based on distinct deep neural network architectures: feed-forward neural network, long short-term memory (LSTM), gated recurrent units, and bidirectional LSTM. All the models demonstrated high predictive accuracies, with percent biases of 0.74–2.62 and Nash–Sutcliffe model efficiencies of 0.954–0.996; however, the LSTM performed best among the four models. Notably, the models' performances varied when applied to regional datasets that were not included in the training phase; this highlighted the critical need for diverse training data to enhance model robustness. This research marks a significant advancement in agricultural modeling by combining state-of-the-art computational techniques with established methodologies, setting a new standard for crop yield prediction.

SAR target augmentation and recognition via cross-domain reconstruction

The deep learning-based target recognition methods have achieved great performance in the preceding works. Large amounts of training data with label were collected to train a deep architecture, by which the inference can be obtained. For radar sensors, the data could be collected easily, yet the prior knowledge on label was difficult to be accessed. To solve the problem, a cross-domain re-imaging target augmentation method was proposed in this paper. The original image was first recast into the frequency domain. The frequency were then randomly filtered by a randomly generated mask. The size and the shape of mask was randomly determined. The filtering results were finally used for re-imaging. The original target can be then reconstructed accordingly. A series of new samples can be generated freely. The amounts and the diversity of dataset can be therefore improved. The proposed augmentation method can be implemented on-line or off-line, making it adaptable to various downstream tasks. Multiple comparative studies throw the light on the superiority of proposed method over the standard and recent techniques. It served to generate the images that would aid the downstream tasks.

ValveVision AI - a multimodal language model for qualitative reporting of the aortic valve in echocardiography

Abstract Background The precise assessment of the aortic valve via echocardiography is critical for early detection and management of aortic valve diseases. Until recently, previous studies have examined machine learning models to estimate individual measurements and severity of aortic stenosis (AS) from echocardiographic images. These image processing algorithms, while precise in their narrow focus, fall short in mirroring the holistic and interconnected clinical judgment typical of human echocardiographers in producing a qualitative report. Large language models (LLMs), particularly image-to-text multi-modal LLMs, are a fundamental advance in the field of deep learning with implications for a host of applications in medical imaging. They promise to encapsulate not just discrete data points typical in traditional machine learning, but also the complex contextual interrelations in clinical diagnosis. Methods In this study, a large-scale heterogeneous database of echocardiographic images containing over 90,681 studies with textual descriptors of the aortic valve was used to train a single, image-to-text multimodal LLM called ValveVision AI. The ground truth textual summaries were drafted by level III echocardiographers in a clinical setting between 2015-2020. BLEU and ROUGE score was calculated. The models were retrospectively assessed on a holdout dataset. Reviewing physicians compared the generated summary to the ground truth and binarily agreed or rejected it. Receiver Operating Characteristics (ROC) for distinct pathologies were also assessed (Figure I). Results ValveVision AI performed with a BLEU score of 0.45 and a ROUGE score of 0.49. The performance of the model in reporting on classification of moderate/severe vs none/mild AS in concordance with the validation protocol described above achieved a specificity of 91.98% and a sensitivity of 83.89%, along with more precise qualitative description. Qualitatively, the model exhibited the capability of zero-shot learning in certain instances, however, this result remains an area of exploration. Conclusion This study represents to our knowledge, the first attempt at an image-representation to text-tokenizer deep learning model architecture to mimic the thought and subtlety of echocardiographic qualitative analysis of the aortic valve. The results suggest that this multimodal LLM has sufficient accuracy to create a preliminary textual summary of the aortic valve that, if paired with a point of care ultrasound (POCUS) device in a primary care setting, may facilitate case triage, increase efficiency, and determine a more precise care pathway for patients.

Hybrid Feature Extraction for Breast Cancer Classification Using the Ensemble Residual VGG16 Deep Learning Model

Introduction: Breast Cancer (BC) is a significant cause of high mortality amongst women globally and probably will remain a disease posing challenges about its detectability. Advancements in medical imaging technology have improved the accuracy and efficiency of breast cancer classification. However, tumor features' complexity and imaging data variability still pose challenges. Method: This study proposes the Ensemble Residual-VGG-16 model as a novel combination of the Deep Residual Network (DRN) and VGG-16 architecture. This model is purposely engineered with maximal precision for the task of breast cancer diagnosis based on mammography images. We assessed its performance by accuracy, recall, precision, and the F1-Score. All these metrics indicated the high performance of this Residual-VGG-16 model. The diagnostic residual-VGG16 performed exceptionally well with an accuracy of 99.6%, precision of 99.4%, recall of 99.7%, F1 score of 98.6%, and Mean Intersection over Union (MIoU) of 99.8% with MIAS datasets. Result: Similarly, the INBreast dataset achieved an accuracy of 93.8%, a precision of 94.2%, a recall of 94.5%, and an F1-score of 93.4% Conclusion: The proposed model is a significant advancement in breast cancer diagnosis, with high accuracy and potential as an automated grading.

Federated Deep Learning Model for False Data Injection Attack Detection in Cyber Physical Power Systems

Cyber-physical power systems (CPPS) integrate information and communication technology into conventional electric power systems to facilitate bidirectional communication of information and electric power between users and power grids. Despite its benefits, the open communication environment of CPPS is vulnerable to various security attacks. This paper proposes a federated deep learning-based architecture to detect false data injection attacks (FDIAs) in CPPS. The proposed work offers a strong, decentralized alternative with the ability to boost detection accuracy while maintaining data privacy, presenting a significant opportunity for real-world applications in the smart grid. This framework combines state-of-the-art machine learning and deep learning models, which are used in both centralized and federated learning configurations, to boost the detection of false data injection attacks in cyber-physical power systems. In particular, the research uses a multi-stage detection framework that combines several models, including classic machine learning classifiers like Random Forest and ExtraTrees Classifiers, and deep learning architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). The results demonstrate that Bidirectional GRU and LSTM models with attention layers in a federated learning setup achieve superior performance, with accuracy approaching 99.8%. This approach enhances both detection accuracy and data privacy, offering a robust solution for FDIA detection in real-world smart grid applications.

Human risk recognition and prediction in manned submersible diving tasks driven by deep learning models

The complexity of human cognition is increased by the multiple and interactive information clusters brought about by the application of advanced intelligent information technologies. Especially in systems operating in extreme regions far from society, human errors are more pronounced than ever before. Prolonged social isolation, extreme weightless or overweight environments, stressful atmospheres, and lack of situational awareness are all added potential elements contributing to human risk. Although the development of human reliability analysis methods and their variants continues to mature, accurately predicting the potential risk of dynamic human behavior from sparse and discrete events remains a great challenge. We focus on deep learning computational architectures that are similar to the cognitive processes and mechanisms of the brain, and build neural networks that match the perceptual activation and memory cycling of the cognitive features of the brain. This study focuses on investigating the ability of the joint SNN-ITLSTM network to predict human error behavior and the clusters of performance shaping factors that effectively characterize the far-social nature. Combining the bionic properties of SNN and the temporal update mechanism of LSTM in the form of hierarchical events constitutes a computationally efficient network architecture. Our results show that the joint model proposed in this study has the performance to strengthen temporal influences and characterize cognitive features of the brain.

Enhanced MRI-based brain tumour classification with a novel Pix2pix generative adversarial network augmentation framework.

The scarcity of medical imaging datasets and privacy concerns pose significant challenges in artificial intelligence-based disease prediction. This poses major concerns to patient confidentiality as there are now tools capable of extracting patient information by merely analysing patient's imaging data. To address this, we propose the use of synthetic data generated by generative adversarial networks as a solution. Our study pioneers the utilisation of a novel Pix2Pix generative adversarial network model, specifically the 'image-to-image translation with conditional adversarial networks,' to generate synthetic datasets for brain tumour classification. We focus on classifying four tumour types: glioma, meningioma, pituitary and healthy. We introduce a novel conditional deep convolutional neural network architecture, developed from convolutional neural network architectures, to process the pre-processed generated synthetic datasets and the original datasets obtained from the Kaggle repository. Our evaluation metrics demonstrate the conditional deep convolutional neural network model's high performance with synthetic images, achieving an accuracy of 86%. Comparative analysis with state-of-the-art models such as Residual Network50, Visual Geometry Group 16, Visual Geometry Group 19 and InceptionV3 highlights the superior performance of our conditional deep convolutional neural network model in brain tumour detection, diagnosis and classification. Our findings underscore the efficacy of our novel Pix2Pix generative adversarial network augmentation technique in creating synthetic datasets for accurate brain tumour classification, offering a promising avenue for improved disease prediction and treatment planning.

Perturbation defense ultra high-speed weak target recognition

Ultra high-speed target recognition in complex electromagnetic environments is a critical and fundamental machine perception issue. It is difficult to ensure privacy protection and intelligent confrontation with centralized training in many practical situations. In this paper, an efficient ultra high-speed target recognition approach, named InVision, for robot systems is proposed using deep trustworthy federated learning (DTFL). InVision is accurate, safe, fast, and robust. Particularly, geometric component transformer (GCT) is presented to significantly promote neural element's complex representation description ability. And an ambiguity-aware cooperative learning (AACL) scheme is developed to relieve the noisy label problem. Moreover, decentralized federated training (DFT) is designed to mitigate the intractable privacy protection problem, to efficiently search for similarities and reduce the representation redundancy. Furthermore, to promote the running speed of the system in real-world environments, a lightweight deep architecture, called Mobile-XB, is developed. Extensive quantitative and qualitative experiments are carried out, and the results demonstrate that InVision greatly outperforms the outstanding comparison methods, establishing efficient connections and extraction, and providing security guarantees.

Optimized deep learning networks for accurate identification of cancer cells in bone marrow

Radiologists utilize pictures from X-rays, magnetic resonance imaging, or computed tomography scans to diagnose bone cancer. Manual methods are labor-intensive and may need specialized knowledge. As a result, creating an automated process for distinguishing between malignant and healthy bone is essential. Bones that have cancer have a different texture than bones in unaffected areas. Diagnosing hematological illnesses relies on correct labeling and categorizing nucleated cells in the bone marrow. However, timely diagnosis and treatment are hampered by pathologists' need to identify specimens, which can be sensitive and time-consuming manually. Humanity's ability to evaluate and identify these more complicated illnesses has significantly been bolstered by the development of artificial intelligence, particularly machine, and deep learning. Conversely, much research and development is needed to enhance cancer cell identification—and lower false alarm rates. We built a deep learning model for morphological analysis to solve this problem. This paper introduces a novel deep convolutional neural network architecture in which hybrid multi-objective and category-based optimization algorithms are used to optimize the hyperparameters adaptively. Using the processed cell pictures as input, the proposed model is then trained with an optimized attention-based multi-scale convolutional neural network to identify the kind of cancer cells in the bone marrow. Extensive experiments are run on publicly available datasets, with the results being measured and evaluated using a wide range of performance indicators. In contrast to deep learning models that have already been trained, the total accuracy of 99.7% was determined to be superior.

A Novel Approach to Predict the Asian Exchange Stock Market Index Using Artificial Intelligence

This study uses real-world illustrations to explore the application of deep learning approaches to predict economic information. In this, we investigate the effect of deep learning model architecture and time-series data properties on prediction accuracy. We aim to evaluate the predictive power of several neural network models using a financial time-series dataset. These models include Convolutional RNNs, Convolutional LSTMs, Convolutional GRUs, Convolutional Bi-directional RNNs, Convolutional Bi-directional LSTMs, and Convolutional Bi-directional GRUs. Our main objective is to utilize deep learning techniques for simultaneous predictions on multivariable time-series datasets. We utilize the daily fluctuations of six Asian stock market indices from 1 April 2020 to 31 March 2024. This study’s overarching goal is to evaluate deep learning models constructed using training data gathered during the early stages of the COVID-19 pandemic when the economy was hit hard. We find that the limitations prove that no single deep learning algorithm can reliably forecast financial data for every state. In addition, predictions obtained from solitary deep learning models are more precise when dealing with consistent time-series data. Nevertheless, the hybrid model performs better when analyzing time-series data with significant chaos.

HEFS-MLDR: A novel hybrid ensemble feature selection framework for improved deep neural network architecture in the diagnosis of Parkinson’s disease

HEFS-MLDR: A novel hybrid ensemble feature selection framework for improved deep neural network architecture in the diagnosis of Parkinson’s disease

HiCDiffusion - diffusion-enhanced, transformer-based prediction of chromatin interactions from DNA sequences

Prediction of chromatin interactions from DNA sequence has been a significant research challenge in the last couple of years. Several solutions have been proposed, most of which are based on encoder-decoder architecture, where 1D sequence is convoluted, encoded into the latent representation, and then decoded using 2D convolutions into the Hi-C pairwise chromatin spatial proximity matrix. Those methods, while obtaining high correlation scores and improved metrics, produce Hi-C matrices that are artificial - they are blurred due to the deep learning model architecture. In our study, we propose the HiCDiffusion, sequence-only model that addresses this problem. We first train the encoder-decoder neural network and then use it as a component of the diffusion model - where we guide the diffusion using a latent representation of the sequence, as well as the final output from the encoder-decoder. That way, we obtain the high-resolution Hi-C matrices that not only better resemble the experimental results - improving the Fréchet inception distance by an average of 11 times, with the highest improvement of 56 times - but also obtain similar classic metrics to current state-of-the-art encoder-decoder architectures used for the task.

Kilometer-Scale Precipitation Forecasting Utilizing Convolutional Neural Networks: A Case Study of Jiangsu’s Coastal Regions

High-resolution precipitation forecasts play a pivotal role in formulating comprehensive disaster prevention and mitigation plans. As spatial resolution enhances, striking a balance between computation, storage, and simulation accuracy becomes imperative to ensure optimal cost-effectiveness. Convolutional neural networks (CNNs), a cornerstone of deep learning, are examined in this study for their downscaling capabilities in precipitation simulation. During a precipitation event on 23 June 2022, in Jiangsu Province, China, distinct rain belts emerged in both southern and northern Jiangsu, precisely captured by a numerical model (the Weather Research and Forecasting, WRF) with a 3 km spatial resolution. Specifically, precipitation was prevalent in northern Jiangsu from 00:00 to 11:00 Beijing Time (BJT), transitioning to southern Jiangsu from 12:00 to 23:00 BJT on the same day. Upon dynamic downscaling, the model reproduced precipitation in these periods with an average error of 12.35 mm at 3 km and 12.48 mm at 1 km spatial resolutions. Employing CNN technology for statistical downscaling to a 1 km spatial resolution, samples from the initial period were utilized for training, while those from the subsequent period served for validation. Following dynamic downscaling, CNNs with four, five, six, and seven layers exhibited average errors of 8.86 mm, 8.93 mm, 9.71 mm, and 9.70 mm, respectively, accompanied by correlation coefficients of 0.550, 0.570, 0.574, and 0.578, respectively. This analysis indicates that for this precipitation event, a shallower CNN depth yields a lower average error and correlation coefficient, whereas a deeper architecture enhances the correlation coefficient. By employing deep network architectures, CNNs are capable of capturing nonlinear patterns and subtle local features from complex meteorological data, thereby providing more accurate predictions during the downscaling process. Leveraging faster computation and reduced storage requirements, machine learning has demonstrated immense potential in high-resolution forecasting research. There is significant scope for advancing technologies that integrate numerical models with machine learning to achieve higher-resolution numerical forecasts.

A Neural Architecture Search CNN for Alzheimer’s Disease Classification

The evolution of automated machine learning (AutoML) is gradually reengineering the design of deep learning architectures for various imaging tasks. AutoML effectively develops model architectures and tunes hyperparameters through neural architecture search (NAS). Deep learning model architecture design is generally considered a tedious and time-consuming task that requires mastery skills to develop robust and better-performing models for imaging tasks. Again, the model's hyperparameters must be well-tuned to ensure optimal performances, which can be tedious and time-consuming if the hyperparameters are manually selected; using existing hyperparameter optimization algorithms can be expensive regarding resources. This study addresses these challenges in developing an optimal convolutional neural network (CNN) for classifying Alzheimer's (AD). The study, therefore, adopted a NAS approach to generate a CNN model architecture using a customized search space comprising only CNN patterns implemented with a NAS framework. The search was done for ten (10) trials, yielding a CNN architecture with an accuracy of 95.85% and a loss of 0.22. Training the model with a 10-fold cross-validation approach using a 0.0009 learning rate for 150 epochs improved the model's performance. The model recorded 97.17% accuracy, 97.21% precision, 97.14% recall, and a 0.99 area under the curve (AUC) in classifying AD as one of AD, mild cognitive impairment (MCI), and normal control (NC). The model obtained 98.06%, 98.66%, and 98.62% accuracy on binary classes of AD/NC, AD/MCI, and NC/MCI, respectively. The model generally showed robustness and better performance than existing CNN architectures.