Deep Architecture Research Articles

Study of Neural Network Architectures for Determining Gas Concentrations by Spectra

A study of a number of neural networks of different architectures for determining gas concentrations from spectra obtained using an optical emission gas analyzer, which measures the spectrum of electromagnetic radiation emitted by gases when excited by an electric discharge, is presented. The neural network is trained on data from the optical spectroscopy laboratory and is able to predict gas concentrations from spectra at high speed. The research concerned the deep neural network architectures with convolutional and recurrent layers. Convolutional layers highlight the features of the spectra, while recurrent layers take into account the consistent structure of the data. The quality of the neural network is evaluated by the R2 coefficient of determination, and the comparison between networks by the RMSE indicator between the predicted and real gas concentrations.

A Deep Learning-based Architecture for Diabetes Detection, Prediction, and Classification

A Deep Learning-based Architecture for Diabetes Detection, Prediction, and Classification

Learned liquid crystal microlens array for joint optimized deep optical architecture in identifying metameric materials.

Multispectral imaging holds great promise for the detection of metameric materials. However, traditional multispectral imaging systems are characterized by their large volume, complex structure, and high computational requirements, limiting their practical application. We propose a jointly optimized deep optical architecture that combines the liquid crystal (LC) microlens array (MLA) characteristics and a multi-level perceptual spectral reconstruction network (MLP-SRN). The core of the architecture is to integrate the physical properties of the LC-MLA into the MLP-SRN using point spread function (PSF) optical convolution kernels, decoupling the light-field characteristic information collected by the LC-MLA at different voltages. Experimental results demonstrate that the incorporation of the physical properties of the LC-MLA not only reduces the system size and computational complexity but demonstrates excellent performance in identifying a metameric material.

Modified scaled exponential linear unit

<p>Activation functions assume a crucial role in elucidating the intricacies of training dynamics and the overall performance of neural networks. Despite its simplicity and effectiveness, the ubiquitously embraced ReLU activation function harbors certain drawbacks, notably the predicament recognized as the “Dying ReLU” issue. To address such challenges, we propose the introduction of a pioneering activation function, the modified scaled exponential linear unit (M-SELU). Drawing from an array of experiments conducted across diverse computer vision tasks employing cutting-edge architectures, it becomes apparent that M-SELU exhibits superior performance compared to ReLU (used as the baseline) and various other activation functions. The simplicity of the proposed activation function (M-SELU) makes this solution particularly suitable for multi-layered deep neural architecture, including applications in CNN, CIFAR-10, and the broader field of deep learning.</p>

Rapid-Learning Collaborative Pushing and Grasping via Deep Reinforcement Learning and Image Masking

When multiple objects are positioned close together or stacked, pre-grasp operations such as pushing objects can be used to create space for the grasp, thereby improving the grasping success rate. This study develops a model based on a deep Q-learning network architecture and introduces a fully convolutional network to accurately identify pixels in the workspace image that correspond to target locations for exploration. In addition, this study incorporates image masking to limit the exploration area of the robotic arm, ensuring that the agent consistently explores regions containing objects. This approach effectively addresses the sparse reward problem and improves the convergence rate of the model. Experimental results from both simulated and real-world environments show that the proposed method accelerates the learning of effective grasping strategies. When image masking is applied, the success rate in the grasping task reaches 80% after 600 iterations. The time required to reach 80% success rate is 25% shorter when image masking is used compared to when it is not used. The main finding of this study is the direct integration of image masking technique with a deep reinforcement learning (DRL) algorithm, which offers significant advancement in robotic arm control. Furthermore, this study shows that image masking technique can substantially reduce training time and improve the object grasping success rate. This innovation enables the robotic arm to better adapt to scenarios that conventional DRL methods cannot handle, thereby improving training efficiency and performance in complex and dynamic industrial applications.

Synergistic application of digital outcrop characterization techniques and deep learning algorithms in geological exploration

In order to meet the needs of geologists for the analysis of data characterizing field outcrops (rock sections or formations exposed on the ground surface), this study developed a field digital outcrop visualization platform based on Cesium (a 3D geospatial visualization technology) digital outcrop characterization technology. The platform was developed based on WebGL (a protocol for rendering interactions on web pages), which overcame the shortcomings of traditional software in terms of visualization, cross-device, cross-platform, and ease of use. Firstly, UAV inclined photography is used for data collection, which transforms a large amount of geological data into an intuitive 3D geological model, while the visualization platform provides rich measurement and mapping tools for the identified features, which more intuitively displays the outcrop information, helps geological explorers to understand the geological conditions in the field more quickly and comprehensively, and improves the analysis efficiency and ease-of-use of outcrop characterization data. Combined with the improved VGG19 (a deep convolutional neural network architecture) algorithm model, it has excellent performance in dealing with the fine texture and complex structure of rocks, which significantly improves the accuracy of lithology identification. The synergistic application of this technology provides geologists with a faster and more comprehensive means to understand the geological conditions in the field. The reliability of combining the Cesium digital outcrop characterization technology with the VGG19 lithology identification algorithm in geological exploration is verified through case studies. The synergistic application of this technology will greatly enhance the efficiency and ease of analysis of outcrop characterization in the field, and provide new perspectives for future research in geosciences.

Deep transformer-based architecture for the recognition of mathematical equations from real-world math problems

Identifying mathematical equations from real-world math problems presents a unique and challenging task within the field of Natural Language Processing (NLP). It has a wide range of applications in various areas, such as academics, digital content design, and the development of automatic or interactive learning systems. However, the accurate understanding of these equations still needs to be enhanced due to the intrinsic complexity of mathematical symbols and various structural formats. Additionally, the unique syntax, diverse symbols, and complex structure of mathematical equations present significant obstacles that traditional NLP methods and Optical Character Recognition (OCR) systems need help to overcome. In this research, we utilize deep transformer architecture to recognize mathematical equations, and we have utilized our novel dataset with 3433 distinct observations. This dataset, which we have collected to include a diverse range of mathematical equations, is used to predict six (z=x+y, z=x−y, z=xy, z=x/y, y=x! and y=x) basic mathematical equations. We applied different transformer-based architectures, such as BERT, ELECTRA, XLNet, RoBERTa, and DistilBERT, and BERT performs best with 99.80% accuracy. To the best of our knowledge, this is the first NLP work in any language where we recognize the equation from the mathematical text.

Deep hybrid architecture with stacked ensemble learning for binary classification of retinal disease

PurposeEarly retinal disease identification is vital since symptoms are passive at initial stages but lead to irreversible vision loss at advanced stages. Globally, a substantial population is at risk of vision impairment, prompting researchers to investigate methods for efficient classification. MethodsThis work experimented one hundred and forty-four different hybrid architectures amalgamating each of the eight convolutional neural architectures (VGG, EfficientNet, Inception, ResNet, NasNet, DenseNet, InceptionResNet, Xception) with seven classifiers (Logistic regression, K-Nearest Neighbours, Support Vector Classifier, Decision Tree, Bagging classifier, Random Forest, Adaptive Boosting, Light Gradient Boost and Extra tree classifier). The top performing n (n=3,4,5) classifiers were ensembled with meta-learner using stacking strategy. The performance of the pipeline is evaluated with two distinct meta-learners, three different image sizes, and feature counts on the RFMiD dataset. ResultsThe architectures were assessed using (1) the performance metrics – accuracy, precision, recall, and F1-score, (2) statistical graphics to understand the prevalence of classifiers, Borda count voting method to identify the best CNN model, (3) Tukey's honestly significance difference test to identify best-performing architecture. Two architectures achieved a high accuracy of 92.34 percent and F1 scores of 95.11 and 95.19. ConclusionThis is the first work to experiment with 144 combinations to identify suitable deep architecture for binary retinal disease classification. The study recommends Xception for feature extraction ensembled with ExtraTreeClassifier, Light gradient boosting machine, Random Forest, AdaBoost classifiers, and meta-learner as Logistic Regression. Another best-performing architecture is similar with a variation in DenseNet121 for feature extraction and support vector as a meta-learner.

Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration.

The deep unfolding approach has attracted significant attention in computer vision tasks, which well connects conventional image processing modeling manners with more recent deep learning techniques. Specifically, by establishing a direct correspondence between algorithm operators at each implementation step and network modules within each layer, one can rationally construct an almost "white box" network architecture with high interpretability. In this architecture, only the predefined component of the proximal operator, known as a proximal network, needs manual configuration, enabling the network to automatically extract intrinsic image priors in a data-driven manner. In current deep unfolding methods, such a proximal network is generally designed as a CNN architecture, whose necessity has been proven by a recent theory. That is, CNN structure substantially delivers the translational symmetry image prior, which is the most universally possessed structural prior across various types of images. However, standard CNN-based proximal networks have essential limitations in capturing the rotation symmetry prior, another universal structural prior underlying general images. This leaves a large room for further performance improvement in deep unfolding approaches. To address this issue, this study makes efforts to suggest a high-accuracy rotation equivariant proximal network that effectively embeds rotation symmetry priors into the deep unfolding framework. Especially, we deduce, for the first time, the theoretical equivariant error for such a designed proximal network with arbitrary layers under arbitrary rotation degrees. This analysis should be the most refined theoretical conclusion for such error evaluation to date and is also indispensable for supporting the rationale behind such networks with intrinsic interpretability requirements. Through experimental validation on different vision tasks, including blind image super-resolution, medical image reconstruction, and image de-raining, the proposed method is validated to be capable of directly replacing the proximal network in current deep unfolding architecture and readily enhancing their state-of-the-art performance. This indicates its potential usability in general vision tasks.

Performance comparison of state-of-the-art deep learning model architectures in Indonesian food image classification

Performance comparison of state-of-the-art deep learning model architectures in Indonesian food image classification

Few-Shot Relation Extraction With Dual Graph Neural Network Interaction.

Recent advances in relation extraction with deep neural architectures have achieved excellent performance. However, current models still suffer from two main drawbacks: 1) they require enormous volumes of training data to avoid model overfitting and 2) there is a sharp decrease in performance when the data distribution during training and testing shift from one domain to the other. It is thus vital to reduce the data requirement in training and explicitly model the distribution difference when transferring knowledge from one domain to another. In this work, we concentrate on few-shot relation extraction under domain adaptation settings. Specifically, we propose DUAL GRAPH, a novel graph neural network (GNN) based approach for few-shot relation extraction. DUAL GRAPH leverages an edge-labeling dual graph (i.e., an instance graph and a distribution graph) to explicitly model the intraclass similarity and interclass dissimilarity in each individual graph, as well as the instance-level and distribution-level relations across graphs. A dual graph interaction mechanism is proposed to adequately fuse the information between the two graphs in a cyclic flow manner. We extensively evaluate DUAL GRAPH on FewRel1.0 and FewRel2.0 benchmarks under four few-shot configurations. The experimental results demonstrate that DUAL GRAPH can match or outperform previously published approaches. We also perform experiments to further investigate the parameter settings and architectural choices, and we offer a qualitative analysis.

Alignable kernel network

To enhance the adaptability and performance of Convolutional Neural Networks (CNN), we present an adaptable mechanism called Alignable Kernel (AliK) unit, which dynamically adjusts the receptive field (RF) dimensions of a model in response to varying stimuli. The branches of AliK unit are integrated through a novel align transformation softmax attention, incorporating prior knowledge through rank ordering constraints. The attention weightings across the branches establish the effective RF scales, leveraged by neurons in the fusion layer. This mechanism is inspired by neuroscientific observations indicating that the RF dimensions of neurons in the visual cortex vary with the stimulus, a feature often overlooked in CNN architectures. By aggregating successive AliK ensembles, we develop a deep network architecture named the Alignable Kernel Network (AliKNet). AliKNet with interdisciplinary design improves the network’s performance and interpretability by taking direct inspiration from the structure and function of human neural systems, especially the visual cortex. Empirical evaluations in the domains of image classification and semantic segmentation have demonstrated that AliKNet excels over numerous state-of-the-art architectures, achieving this without increasing model complexity. Furthermore, we demonstrate that AliKNet can identify target objects across various scales, confirming their ability to dynamically adapt their RF sizes in response to the input data.

Application of a deep-learning marker for morbidity and mortality prediction derived from retinal photographs: a cohort development and validation study

Biological ageing markers are useful to risk stratify morbidity and mortality more precisely than chronological age. In this study, we aimed to develop a novel deep-learning-based biological ageing marker (referred to as RetiPhenoAge hereafter) using retinal images and PhenoAge, a composite biomarker of phenotypic age. We used retinal photographs from the UK Biobank dataset to train a deep-learning algorithm to predict the composite score of PhenoAge. We used a deep convolutional neural network architecture with multiple layers to develop our deep-learning-based biological ageing marker, as RetiPhenoAge, with the aim of identifying patterns and features in the retina associated with variations of blood biomarkers related to renal, immune, liver functions, inflammation, and energy metabolism, and chronological age. We determined the performance of this biological ageing marker for the prediction of morbidity (cardiovascular disease and cancer events) and mortality (all-cause, cardiovascular disease, and cancer) in three independent cohorts (UK Biobank, the Singapore Epidemiology of Eye Diseases [SEED], and the Age-Related Eye Disease Study [AREDS] from the USA). We also compared the performance of RetiPhenoAge with two other known ageing biomarkers (hand grip strength and adjusted leukocyte telomere length) and one lifestyle factor (physical activity) for risk stratification of mortality and morbidity. We explored the underlying biology of RetiPhenoAge by assessing its associations with different systemic characteristics (eg, diabetes or hypertension) and blood metabolite levels. We also did a genome-wide association study to identify genetic variants associated with RetiPhenoAge, followed by expression quantitative trait loci mapping, a gene-based analysis, and a gene-set analysis. Cox proportional hazards models were used to estimate the hazard ratios (HRs) and corresponding 95% CIs for the associations between RetiPhenoAge and the different morbidity and mortality outcomes. Retinal photographs for 34 061UK Biobank participants were used to train the model, and data for 9429participants from the SEED cohort and for 3986participants from the AREDS cohort were included in the study. RetiPhenoAge was associated with all-cause mortality (HR 1·92 [95% CI 1·42-2·61]), cardiovascular disease mortality (1·97 [1·02-3·82]), cancer mortality (2·07 [1·29-3·33]), and cardiovascular disease events (1·70[1·17-2·47]), independent of PhenoAge and other possible confounders. Similar findings were found in the two independent cohorts (HR 1·67 [1·21-2·31] for cardiovascular disease mortality in SEED and 2·07 [1·10-3·92] in AREDS). RetiPhenoAge had stronger associations with mortality and morbidity than did hand grip strength, telomere length, and physical activity. We identified two genetic variants that were significantly associated with RetiPhenoAge (single nucleotide polymorphisms rs3791224 and rs8001273), and were linked to expression quantitative trait locis in various tissues, including the heart, kidneys, and the brain. Our new deep-learning-derived biological ageing marker is a robust predictor of mortality and morbidity outcomes and could be used as a novel non-invasive method to measure ageing. Singapore National Medical Research Council and Agency for Science, Technology and Research, Singapore.

ALGORITHM FOR IDENTIFYING ABNORMAL ACTIONS

The study is devoted to the problem of recognition of human activity recognition and the definition of normal and abnormal behavior (activity) depending on the action scene. Automated detection of abnormal activity using computer vision technologies and rapid response makes it possible to improve the work of rapid response services, thereby saving human lives or stopping offenses. The paper presents a comprehensive review of methods for recognizing human activity and detecting abnormal human activity based on deep learning. Various classifications of abnormal activity are investigated, and then deep learning methods and neural network architectures used to detect abnormal activity are discussed and analyzed. Based on the comparative analysis of various approaches, an algorithm for recognizing human activity has been proposed and a neural network has been developed that determines violent and nonviolent actions with an accuracy of 92,22% in 150 epochs.

Recognition of Real-Time Video Activities Using Stacked Bi-GRU with Fusion-based Deep Architecture

Recognizing and understanding human activities in real-time videos is a challenging task due to the complex nature of video data and the need for efficient and accurate analysis. This research pioneers a breakthrough in video activity recognition by introducing a robust framework leveraging the power of a stacked Bidirectional Long Short-Term Memory (Bi-LSTM) and Gated Recurrent Unit (GRU) architecture, harmonized within a fusion-based deep model. The stacked Bi-LSTM-GRU model capitalizes on its dual recurrent architecture, capturing nuanced temporal dependencies within video sequences. The fusion-based deep architecture synergizes spatial and temporal features, enabling the model to discern intricate patterns in human activities. To further enhance the discriminative power of the model, we introduce a fusion module in the proposed deep architecture. The fusion module integrates multi-modal features extracted from different levels of the network hierarchy, allowing for a more comprehensive representation of video activities. We demonstrate the efficacy of our approach through rigorous experimentation on UCF50, UCF101, and HMDB51 datasets. In experiments on the UCF50 dataset, our model achieves an accuracy of 97.01% and 95.86% on training and validation sets respectively, showcasing its proficiency in discerning activities across a diverse range of scenarios. The evaluation extends to the UCF101 dataset, where the proposed approach achieves a competitive accuracy of 97.62% and 96.93% on training and validation sets, surpassing previous benchmarks by a margin of approx 1%. Further-more, on the challenging HMDB51 dataset, the model demonstrates a robust accuracy of 89.71%and 88.88% on training and validation sets, solidifying its efficacy in intricate action recognition tasks.

Deep learning for automated encrustation detection in sewer inspection

Rapid urbanization and population growth in recent decades have placed significant pressure on urban cities to rely heavily on underground infrastructure, such as sewers and tunnels, to maintain the provision of essential services. These sewers, typically having a limited lifespan of 50 to 100 years, are prone to various forms of defects. While prior research has primarily addressed common sewer defect like crack, root intrusion, and infiltration among others, the challenge of encrustation—the formation of hard deposits within sewer systems—has received less attention. This study presents a pioneering deep-learning approach to detect encrustation in sewers by leveraging survey videos from 14 different sewers in the United Kingdom. Our work marks the first effort to develop models specifically for detecting encrustation using deep learning techniques, as previous studies have focused on other types of deposits such as settled and attached deposits. By converting the videos into sequential image frames, we subjected them to thorough analysis and several image pre-processing techniques. Our contributions include the development and comparison of different classification models using backbone CNN networks such as AlexNet, VGG16, EfficientNet, and VGG19 to classify encrustation. Notably, this study provides the first metric-based comparison of these backbone networks to identify the most effective model for encrustation detection. The results demonstrate an impressive 96 % accuracy using the deep architecture of VGG19. Beyond accuracy, this research explores the impact of data augmentation and network dropout on reducing overfitting and enhancing model performance. Additionally, we analyze the time complexities associated with training models with and without data augmentation, providing valuable insights into the efficiency of our approach.

Identifying defects and varieties of Malting Barley Kernels

This study introduces a comprehensive approach for classifying individual malting barley kernels, involving dual-sided kernel imaging, a specifically designed image processing algorithm, an optimized deep neural network architecture, and a mechanical sorting system. The proposed method achieves precise classification into multiple classes, aligning with quality standards for malting material assessment. Throughout the study, various image analysis techniques were assessed, including traditional feature engineering, established transfer learning deep neural network architectures, and our custom-designed convolutional neural network tailored for barley kernel image analysis. Comparative analysis underscores the superior performance of our network model. The study reveals that our proposed deep learning network achieves a 94% accuracy in classifying barley kernel defects and varieties, outperforming well-established transfer learning models to complex architectures that attain 93% accuracy. Additionally, it surpasses the traditional machine learning approach involving feature extraction and support vector machine classifiers, which achieve accuracy below 90% in detecting defective kernels and below 70% in varietal classification. However, we also noted the traditional approach’s advantage in morphological feature recognition. This observation guides new research toward integrating morphological feature extraction techniques with modern convolutional networks. This paper presents a deep neural network designed specifically for the analysis of cereal kernel images in two applications: defect and variety classification. It emphasizes the importance of standardizing kernel orientation and merging images from both sides of the kernel, and introduces a device for image acquisition that fulfills this need.

Accurate crowd counting for intelligent video surveillance systems

The paper presents a novel deep learning approach for crowd counting in intelligent video surveillance systems, addressing the growing need for accurate monitoring of public spaces in urban environments. The demand for precise crowd estimation arises from challenges related to security, public safety, and efficiency in urban areas, particularly during large public events. Existing crowd counting techniques, including feature-based object detection and regression-based methods, face limitations in high-density environments due to occlusions, lighting variations, and diverse human figures. To overcome these challenges, the authors propose a new deep encoder-decoder architecture based on VGG16, which incorporates hierarchical feature extraction with spatial and channel attention mechanisms. This architecture enhances the model’s ability to manage variations in crowd density, leveraging adaptive pooling and dilated convolutions to extract meaningful features from dense crowds. The model’s decoder is further refined to handle sparse and crowded scenes through separate density maps, improving its adaptability and accuracy. Evaluations of the proposed model on benchmark datasets, including Shanghai Tech and UCF CC 50, demonstrate superior performance over state-of-the-art methods, with significant improvements in mean absolute error and mean squared error metrics. The paper emphasizes the importance of addressing environmental variability and scale differences in crowded environments and shows that the proposed model is effective in both sparse and dense crowd conditions. This research contributes to the advancement of intelligent video surveillance systems by providing a more accurate and efficient method for crowd counting, with potential applications in public safety, transportation management, and urban planning.

Θ-Net: A Deep Neural Network Architecture for the Resolution Enhancement of Phase-Modulated Optical Micrographs In Silico.

Optical microscopy is widely regarded to be an indispensable tool in healthcare and manufacturing quality control processes, although its inability to resolve structures separated by a lateral distance under ~200 nm has culminated in the emergence of a new field named fluorescence nanoscopy, while this too is prone to several caveats (namely phototoxicity, interference caused by exogenous probes and cost). In this regard, we present a triplet string of concatenated O-Net ('bead') architectures (termed 'Θ-Net' in the present study) as a cost-efficient and non-invasive approach to enhancing the resolution of non-fluorescent phase-modulated optical microscopical images in silico. The quality of the afore-mentioned enhanced resolution (ER) images was compared with that obtained via other popular frameworks (such as ANNA-PALM, BSRGAN and 3D RCAN), with the Θ-Net-generated ER images depicting an increased level of detail (unlike previous DNNs). In addition, the use of cross-domain (transfer) learning to enhance the capabilities of models trained on differential interference contrast (DIC) datasets [where phasic variations are not as prominently manifested as amplitude/intensity differences in the individual pixels unlike phase-contrast microscopy (PCM)] has resulted in the Θ-Net-generated images closely approximating that of the expected (ground truth) images for both the DIC and PCM datasets. This thus demonstrates the viability of our current Θ-Net architecture in attaining highly resolved images under poor signal-to-noise ratios while eliminating the need for a priori PSF and OTF information, thereby potentially impacting several engineering fronts (particularly biomedical imaging and sensing, precision engineering and optical metrology).

Large-scale deep learning identifies the antiviral potential of PKI-179 and MTI-31 against coronaviruses

Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has led to the global pandemic of Coronavirus Disease (2019) (COVID-19), underscoring the urgency for effective antiviral drugs. Despite the development of different vaccination strategies, the search for specific antiviral compounds remains crucial. Here, we combine machine learning (ML) techniques with in vitro validation to efficiently identify potential antiviral compounds. We overcome the limited amount of SARS-CoV-2 data available for ML using various techniques, supplemented with data from diverse biomedical assays, which enables end-to-end training of a deep neural network architecture. We use its predictions to identify and prioritize compounds for in vitro testing. Two top-hit compounds, PKI-179 and MTI-31, originally identified as Pi3K-mTORC1/2 pathway inhibitors, exhibit significant antiviral activity against SARS-CoV-2 at low micromolar doses. Notably, both compounds outperform the well-known mTOR inhibitor rapamycin. Furthermore, PKI-179 and MTI-31 demonstrate broad-spectrum antiviral activity against SARS-CoV-2 variants of concern and other coronaviruses. In a physiologically relevant model, both compounds show antiviral effects in primary human airway epithelial (HAE) cultures derived from healthy donors cultured in an air-liquid interface (ALI). This study highlights the potential of ML combined with in vitro testing to expedite drug discovery, emphasizing the adaptability of AI-driven approaches across different viruses, thereby contributing to pandemic preparedness.