Shallow Network Research Articles

Improved deep convolutional dictionary learning with no noise parameter for low-dose CT image processing.

Low-Dose computed tomography (LDCT) reduces radiation damage to patients, however, the reconstructed images contain severe noise, which affects doctors' diagnosis of the disease. The convolutional dictionary learning has the advantage of the shift-invariant property. The deep convolutional dictionary learning algorithm (DCDicL) combines deep learning and convolutional dictionary learning, which has great suppression effects on Gaussian noise. However, applying DCDicL to LDCT images cannot get satisfactory results. To address this challenge, this study proposes and tests an improved deep convolutional dictionary learning algorithm for LDCT image processing and denoising. First, we use a modified DCDicL algorithm to improve the input network and make it do not need to input noise intensity parameter. Second, we use DenseNet121 to replace the shallow convolutional network to learn the prior on the convolutional dictionary, which can obtain more accurate convolutional dictionary. Last, in the loss function, we add MSSIM to enhance the detail retention ability of the model. The experimental results on the Mayo dataset show that the proposed model obtained an average value of 35.2975 dB in PSNR, which is 0.2954 -1.0573 dB higher than the mainstream LDCT algorithm, indicating the excellent denoising performance. The study demonstrates that the proposed new algorithm can effectively improve the quality of LDCT images acquired in the clinical practice.

The short-term prediction of daily traffic volume for rural roads using shallow and deep learning networks: ANN and LSTM

The short-term prediction of daily traffic volume for rural roads using shallow and deep learning networks: ANN and LSTM

MOD3NN: A Framework for Automatic Signal Modulation Detection Using 3D CNN

In this work, we present an application of a three-dimensional convolutional neural network for the task of automatic modulation recognition from raw I/Q signal data. Raw I/Q signal data exhibits a special “helical” structure that can be exploited with three-dimensional convolutions (3D convolutions) to learn spatio-temporal features from the signal for the problem of modulation recognition. By tweaking the convolutional filters to learn the helical symmetry of the data, we can design a shallow network for automatic modulation recognition (AMR). We present the results of our experiments with raw I/Q signal data collected in an uncalibrated radio frequency (RF) environment using several different modulation schemes. We show that with our methods and implementation, we can achieve around 99 % accuracy for automatic modulation recognition, for a variety of practical modulation techniques without the need for explicit feature engineering.

Semi-supervised learning improves regulatory sequence prediction with unlabeled sequences

MotivationGenome-wide association studies have systematically identified thousands of single nucleotide polymorphisms (SNPs) associated with complex genetic diseases. However, the majority of those SNPs were found in non-coding genomic regions, preventing the understanding of the underlying causal mechanism. Predicting molecular processes based on the DNA sequence represents a promising approach to understand the role of those non-coding SNPs. Over the past years, deep learning was successfully applied to regulatory sequence prediction using supervised learning. Supervised learning required DNA sequences associated with functional data for training, whose amount is strongly limited by the finite size of the human genome. Conversely, the amount of mammalian DNA sequences is exponentially increasing due to ongoing large sequencing projects, but without functional data in most cases.ResultsTo alleviate the limitations of supervised learning, we propose a paradigm shift with semi-supervised learning, which does not only exploit labeled sequences (e.g. human genome with ChIP-seq experiment), but also unlabeled sequences available in much larger amounts (e.g. from other species without ChIP-seq experiment, such as chimpanzee). Our approach is flexible and can be plugged into any neural architecture including shallow and deep networks, and shows strong predictive performance improvements compared to supervised learning in most cases (up to 70%).Availability and implementationhttps://forgemia.inra.fr/raphael.mourad/deepgnn.

The universal approximation theorem for complex-valued neural networks

We generalize the classical universal approximation theorem for neural networks to the case of complex-valued neural networks. Precisely, we consider feedforward networks with a complex activation function σ:C→C in which each neuron performs the operation CN→C,z↦σ(b+wTz) with weights w∈CN and a bias b∈C, and with σ applied componentwise. We completely characterize those activation functions σ for which the associated complex networks have the universal approximation property, meaning that they can uniformly approximate any continuous function on any compact subset of Cd arbitrarily well. Unlike the classical case of real networks, the set of “good activation functions”—which give rise to networks with the universal approximation property—differs significantly depending on whether one considers deep networks or shallow networks: For deep networks with at least two hidden layers, the universal approximation property holds as long as σ is neither a polynomial, a holomorphic function, or an antiholomorphic function. Shallow networks, on the other hand, are universal if and only if the real part or the imaginary part of σ is not a polyharmonic function.

A Multi-Scale Attention Mechanism Based Domain Adversarial Neural Network Strategy for Bearing Fault Diagnosis

There are a large number of bearings in aircraft engines that are subjected to extreme operating conditions, such as high temperature, high speed, and heavy load, and their fatigue, wear, and other failure problems seriously affect the reliability of the engine. The complex and variable bearing operating conditions can lead to differences in the distribution of data between the source and target operating conditions, as well as insufficient labels. To solve the above challenges, a multi-scale attention mechanism-based domain adversarial neural network strategy for bearing fault diagnosis (MADANN) is proposed and verified using Case Western Reserve University bearing data and PT500mini mechanical bearing data in this paper. First, a multi-scale feature extractor with an attention mechanism is proposed to extract more discriminative multi-scale features of the input signal. Subsequently, the maximum mean discrepancy (MMD) is introduced to measure the difference between the distribution of the target domain and the source domain. Finally, the fault diagnosis process of the rolling is realized by minimizing the loss of the feature classifier, the loss of the MMD distance, and maximizing the loss of the domain discriminator. The verification results indicate that the proposed strategy has stronger learning ability and better diagnosis performance than shallow network, deep network, and commonly used domain adaptive models.

SSMDA: Self-Supervised Cherry Maturity Detection Algorithm Based on Multi-Feature Contrastive Learning

Due to the high cost of annotating dense fruit images, annotated target images are limited in some ripeness detection applications, which significantly restricts the generalization ability of small object detection networks in complex environments. To address this issue, this study proposes a self-supervised cherry ripeness detection algorithm based on multi-feature contrastive learning, consisting of a multi-feature contrastive self-supervised module and an object detection module. The self-supervised module enhances features of unlabeled fruit images through random contrastive augmentation, reducing interference from complex backgrounds. The object detection module establishes a connection with the self-supervised module and designs a shallow feature fusion network based on the input target scale to improve the detection performance of small-sample fruits. Finally, extensive experiments were conducted on a self-made cherry dataset. The proposed algorithm showed improved generalization ability compared to supervised baseline algorithms, with better accuracy in terms of mAP, particularly in detecting distant small cherries.

Continuous Non-Invasive Blood Pressure Measurement Using 60 GHz-Radar-A Feasibility Study.

Blood pressure monitoring is of paramount importance in the assessment of a human's cardiovascular health. The state-of-the-art method remains the usage of an upper-arm cuff sphygmomanometer. However, this device suffers from severe limitations-it only provides a static blood pressure value pair, is incapable of capturing blood pressure variations over time, is inaccurate, and causes discomfort upon use. This work presents a radar-based approach that utilizes the movement of the skin due to artery pulsation to extract pressure waves. From those waves, a set of 21 features was collected and used-together with the calibration parameters of age, gender, height, and weight-as input for a neural network-based regression model. After collecting data from 55 subjects from radar and a blood pressure reference device, we trained 126 networks to analyze the developed approach's predictive power. As a result, a very shallow network with just two hidden layers produced a systolic error of 9.2±8.3 mmHg (mean error ± standard deviation) and a diastolic error of 7.7±5.7 mmHg. While the trained model did not reach the requirements of the AAMI and BHS blood pressure measuring standards, optimizing network performance was not the goal of the proposed work. Still, the approach has displayed great potential in capturing blood pressure variation with the proposed features. The presented approach therefore shows great potential to be incorporated into wearable devices for continuous blood pressure monitoring for home use or screening applications, after improving this approach even further.

Rice grains and grain impurity segmentation method based on a deep learning algorithm-NAM-EfficientNetv2

An appropriate image segmentation algorithm is required for discriminating between full grains and grain impurities. In this study, a lightweight fully convolutional segmentation algorithm based on NAM EfficientNetV2 was proposed to solve the problem of slow processing speed in mobile terminal equipment owing to limited computing resources and a large number of model parameters and improve the detection accuracy. First, a standardized NAM attention mechanism was introduced to replace the SE attention mechanism used in EfficientNetV2 and the improved NAM-EfficientNetV2 network was used as a feature extraction structure. Then, in the up-sampling process, the multi-scale features output by the shallow network are fused to effectively use low-level semantics to encode spatial details and fully convolutional pixel segmentation technology is used to achieve rice grain and impurity segmentation. Finally, compared with the baseline model, the detection accuracy of the model was further improved on a self-made dataset. The comprehensive evaluation index F1 of rice grain and its impurities were 95.26% and 93.27%, respectively, and the model parameter amount was 20.6 M. Combined with post-processing, detecting an image on the GPU device took an average of 0.103 s and 0.301 s on the CPU device. The experimental results showed that the improved algorithm is more lightweight, which provides a reference for the model to be deployed in mobile terminal equipment to realize the function of real-time detection of grain impurity in the combine harvester.

Automatic Analysis of Transverse Musculoskeletal Ultrasound Images Based on the Multi-Task Learning Model

Musculoskeletal ultrasound imaging is an important basis for the early screening and accurate treatment of muscle disorders. It allows the observation of muscle status to screen for underlying neuromuscular diseases including myasthenia gravis, myotonic dystrophy, and ankylosing muscular dystrophy. Due to the complexity of skeletal muscle ultrasound image noise, it is a tedious and time-consuming process to analyze. Therefore, we proposed a multi-task learning-based approach to automatically segment and initially diagnose transverse musculoskeletal ultrasound images. The method implements muscle cross-sectional area (CSA) segmentation and abnormal muscle classification by constructing a multi-task model based on multi-scale fusion and attention mechanisms (MMA-Net). The model exploits the correlation between tasks by sharing a part of the shallow network and adding connections to exchange information in the deep network. The multi-scale feature fusion module and attention mechanism were added to MMA-Net to increase the receptive field and enhance the feature extraction ability. Experiments were conducted using a total of 1827 medial gastrocnemius ultrasound images from multiple subjects. Ten percent of the samples were randomly selected for testing, 10% as the validation set, and the remaining 80% as the training set. The results show that the proposed network structure and the added modules are effective. Compared with advanced single-task models and existing analysis methods, our method has a better performance at classification and segmentation. The mean Dice coefficients and IoU of muscle cross-sectional area segmentation were 96.74% and 94.10%, respectively. The accuracy and recall of abnormal muscle classification were 95.60% and 94.96%. The proposed method achieves convenient and accurate analysis of transverse musculoskeletal ultrasound images, which can assist physicians in the diagnosis and treatment of muscle diseases from multiple perspectives.

Transfer learning-assisted 3D deep learning models for knee osteoarthritis detection: Data from the osteoarthritis initiative.

Knee osteoarthritis is one of the most common musculoskeletal diseases and is usually diagnosed with medical imaging techniques. Conventionally, case identification using plain radiography is practiced. However, we acknowledge that knee osteoarthritis is a 3D complexity; hence, magnetic resonance imaging will be the ideal modality to reveal the hidden osteoarthritis features from a three-dimensional view. In this work, the feasibility of well-known convolutional neural network (CNN) structures (ResNet, DenseNet, VGG, and AlexNet) to distinguish knees with and without osteoarthritis (OA) is investigated. Using 3D convolutional layers, we demonstrated the potential of 3D convolutional neural networks of 13 different architectures in knee osteoarthritis diagnosis. We used transfer learning by transforming 2D pre-trained weights into 3D as initial weights for the training of the 3D models. The performance of the models was compared and evaluated based on the performance metrics [balanced accuracy, precision, F1 score, and area under receiver operating characteristic (AUC) curve]. This study suggested that transfer learning indeed enhanced the performance of the models, especially for ResNet and DenseNet models. Transfer learning-based models presented promising results, with ResNet34 achieving the best overall accuracy of 0.875 and an F1 score of 0.871. The results also showed that shallow networks yielded better performance than deeper neural networks, demonstrated by ResNet18, DenseNet121, and VGG11 with AUC values of 0.945, 0.914, and 0.928, respectively. This encourages the application of clinical diagnostic aid for knee osteoarthritis using 3DCNN even in limited hardware conditions.

Face-based age estimation using improved Swin Transformer with attention-based convolution.

Recently Transformer models is new direction in the computer vision field, which is based on self multihead attention mechanism. Compared with the convolutional neural network, this Transformer uses the self-attention mechanism to capture global contextual information and extract more strong features by learning the association relationship between different features, which has achieved good results in many vision tasks. In face-based age estimation, some facial patches that contain rich age-specific information are critical in the age estimation task. The present study proposed an attention-based convolution (ABC) age estimation framework, called improved Swin Transformer with ABC, in which two separate regions were implemented, namely ABC and Swin Transformer. ABC extracted facial patches containing rich age-specific information using a shallow convolutional network and a multiheaded attention mechanism. Subsequently, the features obtained by ABC were spliced with the flattened image in the Swin Transformer, which were then input to the Swin Transformer to predict the age of the image. The ABC framework spliced the important regions that contained rich age-specific information into the original image, which could fully mobilize the long-dependency of the Swin Transformer, that is, extracting stronger features by learning the dependency relationship between different features. ABC also introduced loss of diversity to guide the training of self-attention mechanism, reducing overlap between patches so that the diverse and important patches were discovered. Through extensive experiments, this study showed that the proposed framework outperformed several state-of-the-art methods on age estimation benchmark datasets.

Knowledge Distillation in Histology Landscape by Multi-Layer Features Supervision.

Automatic tissue classification is a fundamental task in computational pathology for profiling tumor micro-environments. Deep learning has advanced tissue classification performance at the cost of significant computational power. Shallow networks have also been end-to-end trained using direct supervision however their performance degrades because of the lack of capturing robust tissue heterogeneity. Knowledge distillation has recently been employed to improve the performance of the shallow networks used as student networks by using additional supervision from deep neural networks used as teacher networks. In the current work, we propose a novel knowledge distillation algorithm to improve the performance of shallow networks for tissue phenotyping in histology images. For this purpose, we propose multi-layer feature distillation such that a single layer in the student network gets supervision from multiple teacher layers. In the proposed algorithm, the size of the feature map of two layers is matched by using a learnable multi-layer perceptron. The distance between the feature maps of the two layers is then minimized during the training of the student network. The overall objective function is computed by summation of the loss over multiple layers combination weighted with a learnable attention-based parameter. The proposed algorithm is named as Knowledge Distillation for Tissue Phenotyping (KDTP). Experiments are performed on five different publicly available histology image classification datasets using several teacher-student network combinations within the KDTP algorithm. Our results demonstrate a significant performance increase in the student networks by using the proposed KDTP algorithm compared to direct supervision-based training methods.

Causality-Inspired Single-Source Domain Generalization for Medical Image Segmentation.

Deep learning models usually suffer from the domain shift issue, where models trained on one source domain do not generalize well to other unseen domains. In this work, we investigate the single-source domain generalization problem: training a deep network that is robust to unseen domains, under the condition that training data are only available from one source domain, which is common in medical imaging applications. We tackle this problem in the context of cross-domain medical image segmentation. In this scenario, domain shifts are mainly caused by different acquisition processes. We propose a simple causality-inspired data augmentation approach to expose a segmentation model to synthesized domain-shifted training examples. Specifically, 1) to make the deep model robust to discrepancies in image intensities and textures, we employ a family of randomly-weighted shallow networks. They augment training images using diverse appearance transformations. 2) Further we show that spurious correlations among objects in an image are detrimental to domain robustness. These correlations might be taken by the network as domain-specific clues for making predictions, and they may break on unseen domains. We remove these spurious correlations via causal intervention. This is achieved by resampling the appearances of potentially correlated objects independently. The proposed approach is validated on three cross-domain segmentation scenarios: cross-modality (CT-MRI) abdominal image segmentation, cross-sequence (bSSFP-LGE) cardiac MRI segmentation, and cross-site prostate MRI segmentation. The proposed approach yields consistent performance gains compared with competitive methods when tested on unseen domains.

Modeling Multiple Temporal Scales of Full-Body Movements for Emotion Classification

This work investigates classification of emotions from full-body movements by using a novel Convolutional Neural Network-based architecture. The model is composed of two shallow networks processing in parallel where the 8-bit RGB images obtained from time intervals of 3D-positional data are the inputs. One network performs a coarse-grained modelling in the time domain while the other one applies a fine-grained modelling. We show that combining different temporal scales into one architecture improves the classification results of a dataset composed of short excerpts of the performances of professional dancers who interpreted four affective states: anger, happiness, sadness, and insecurity. Additionally, we investigate the effect of data chunk duration, overlapping, the size of the input images and the contribution of several data augmentation strategies for our proposed method. Better recognition results were obtained when the duration of a data chunk was longer, and this was further improved by applying balanced data augmentation. Moreover, we test our method on other existing motion capture datasets and compare the results with prior art. In all of the experiments, our results surpassed the state-of-the-art approaches, showing that this method generalizes across diverse settings and contexts.

FAS-Res2net: An Improved Res2net-Based Script Identification Method for Natural Scenes

Problems such as complex image backgrounds, low image quality, diverse text forms, and similar or common character layouts in different script categories in natural scenes pose great challenges to scene script identification. This paper proposes a new Res2Net-based improved script identification method, namely FAS-Res2Net. In the feature extraction part, the feature pyramid network (FPN) module is introduced, which is beneficial to aggregate the geometric feature information extracted by the shallow network and the semantic feature information extracted by the deep network. Integrating the Adaptive Spatial Feature Fusion (ASFF) module is beneficial to obtain local feature information for optimal weight fusion. In addition, the global feature information of the image is extracted by introducing the swin transformer coding block, which makes the extracted feature information more abundant. In the classification part, the convolutional classifier is used to replace the traditional Linear classification, and the classification confidence of each category is output, which improves the identification efficiency. The improved algorithm achieved identification rates of 94.7% and 96.0% on public script identification datasets SIW-13 and CVSI-2015, respectively, which verified the superiority of the method.

Floor plan recommendation system using graph neural network with spatial relationship dataset

The purpose of this study was to develop a recommendation system that, in the pre-design phase, quickly and easily search adequate floor plans satisfying the client requirements about the spatial relationship type using artificial intelligence (AI) technology. In this study using a graph dataset representing the spatial relationship between entities, we propose a deep neural network approach using SimGNN and shallow networks with teacher–student learning to compute graph similarity, measured by graph edit distance, fast and accurately during the search operation in the recommendation system. The prediction errors between the GED score (ground truth) and the predicted score were small enough to employ the neural networks for the recommendation system instead of using GED, which takes a long calculation time. The proposed recommendation systems based deep networks also suggested floor plans satisfying given conditions on the spatial relationship with high accuracy.

Greedy training algorithms for neural networks and applications to PDEs

Recently, neural networks have been widely applied for solving partial differential equations (PDEs). Although such methods have been proven remarkably successful on practical engineering problems, they have not been shown, theoretically or empirically, to converge to the underlying PDE solution with arbitrarily high accuracy. The primary difficulty lies in solving the highly non-convex optimization problems resulting from the neural network discretization, which are difficult to treat both theoretically and practically. It is our goal in this work to take a step toward remedying this. For this purpose, we develop a novel greedy training algorithm for shallow neural networks. Our method is applicable to both the variational formulation of the PDE and also to the residual minimization formulation pioneered by physics informed neural networks (PINNs). We analyze the method and obtain a priori error bounds when solving PDEs from the function class defined by shallow networks, which rigorously establishes the convergence of the method as the network size increases. Finally, we test the algorithm on several benchmark examples, including high dimensional PDEs, to confirm the theoretical convergence rate. Although the method is expensive relative to traditional approaches such as finite element methods, we view this work as a proof of concept for neural network-based methods, which shows that numerical methods based upon neural networks can be shown to rigorously converge.

Lava Flow Hazard and Its Implication in Geopark Development for the Active Harrat Khaybar Intracontinental Monogenetic Volcanic Field, Saudi Arabia

Harrat Khaybar is an active monogenetic volcanic field in western Saudi Arabia that hosts spectacular monogenetic volcanoes and a Holocene volcanic cone with extensive lava fields. The volcanic region is a subject of intensive land use development, especially along tourism ventures, where the volcanic features are the key elements to utilize for increasing visitation rates to the region. The youngest eruption is suspected to be Holocene and occurred fewer than 5000 years ago based on the cross-cutting relationship between the youngest lava flows and archaeological sites. Lava flows are typical, from pāhoehoe to ‘a‘ā types with great diversity of transitional textural forms. Here, we recorded typical transitional lava flow surface textures from the youngest flows identified by digital-elevation-model-based terrain analysis, satellite imagery, and direct field observations. We performed lava flow simulations using the Q-LavHA plug-in within the QGIS environment. Lava flow simulations yielded satisfactory results if we applied eruptions along fissures, long simulation distances, and ~5 m lava flow thickness. In these simulations, the upper flow regimes were reconstructed well, but long individual lava flows were not possible to simulate, suggesting that morphological steps likely promoted lava ponding, inflation, and sudden deflation by releasing melts further along shallow syneruptive valley networks.

Object tracking in surveillance video based on Siamese network considering geographical spatiotemporal constraints

AbstractIdentifying and tracking objects in surveillance videos is an important means of mining information during surveillance. Currently, most object‐tracking methods rely only on image features, which cannot accurately express the motion of the tracked object in real geographical scenes and are easily influenced by occlusion and surrounding objects having similar features. However, tracked objects, such as pedestrians and vehicles, usually move in geographical space with fixed patterns of motion, and the motion in a short time is constrained by geographical space and time, the motion trajectory is predictable, and the range of motion is limited. Therefore, based on the SiamFC object tracking framework, this study introduces geographical spatiotemporal constraints into the tracking framework and proposes the GeoSiamFC method. The objective of this is to: (1) construct the mapping relationship between geographical space and image space to solve the problem that the pixel movement within the image after perspective imaging cannot accurately express the motion of the tracked object in a real geographical scene; (2) add candidate search areas according to the predicted trajectory location to correct the tracking errors caused by the occlusion of the object; and (3) focus on the search for the range of motion of the mapped pixel within the image space according to the limited geographical range of motion of the tracked objects in a certain time to reduce the interference of similar objects within the search area. In this study, separate experiments were conducted on a common test dataset using multiple methods to deal with challenges such as occlusion and illumination changes. In addition, a robust test dataset with noise addition and luminance adjustment based on the common test dataset was used. The results show that GeoSiamFC outperforms other object‐tracking methods in the common test dataset with a precision score of 0.995 and a success score of 0.756 compared with most other object‐tracking algorithms under the base condition of using only shallow networks. Moreover, GeoSiamFC maintained the highest precision score (0.970) and high success score (0.734) in the more challenging robust test dataset as well. The tracking speed of 59 frames per second far exceeds the real‐time requirement of 25 FPS. Geographical spatiotemporal constraints were considered to improve tracker performance while providing real‐time feedback on the motion trajectory of the target in geographical space. Thus, the proposed method is suitable for real‐time tracking of the motion trajectory of a target in real geographical scenes in various surveillance videos.