Dual Stream Research Articles

Semi-supervised progressive dehazing network using unlabeled contrastive guidance

Image dehazing aims to restore the missing high-quality content from its original hazy observation. Most of the existing learning-based methods achieve promising achievements by designing various networks. However, these approaches cannot generalize well on real-world scenes, since they fail to exploit natural haze priors. Towards this end, we propose a novel Semi-supervised Progressive Dehazing Network (Semi-PDNet), which leverages both synthetic and real-world images in training process. The overall network follows a progressive architecture, and can be divided into three core stages: image encode stage (IES), feature enhance stage (FES) and hierarchical reconstruction stage (HRS). Specifically, IES is responsible for encoding shallow features from the corrupted hazy image. Then, our FES tries to distill finer local and global features via the well-designed dual stream attentive block (DSAB). The HRS is to estimate semantic and contextual information based on a hierarchical structure, and accurately reconstructs the final clear image. This stage-by-stage paradigm can make full use of informative features from shallow to deep, thus facilitating network for better haze removal. Furthermore, we utilize an unlabeled contrastive guidance (UCG) to bridge the domain gap between synthetic and real-world images. Extensive experimental comparisons show that our Semi-PDNet can obtain comparable results with other state-of-the-art dehazing algorithms.

Spatial attention-based dual stream transformer for concrete defect identification

This paper addresses the problem of automatic crack identification within building structures. Cracks can cause structural damage to a building, necessitating prompt repairs to preserve lives. Detecting structural defects is a difficult process, especially in tall towers. Machine learning and deep learning approaches automate damage identification and crack detection. Machine learning-based methods train models with manually produced features that may contain errors. Deep convolutional neural networks (DCNN), on the other hand, enhance performance by extracting high-dimensional features. However, DCNN algorithms for crack identification lack a global focus on features that can compromise performance. This study inserts a dual-stream transformer module into a single 2D convolution neural network (CNN) layer. The dual-stream block contains a semantic and pixels path. Precise views of local and global features at the pixel level can be obtained through two paths. This increases self-attention information distributed parallel along the semantic and pixel paths to train the model. Further, the DSTNet (Dual Stream Transformer Network) superiority is validated on the three datasets and achieved notable performance.

Low-light aware framework for human activity recognition via optimized dual stream parallel network

Human Activity Recognition (HAR) plays a crucial role in communication and the Internet of Things (IoT), by enabling vision sensors to understand and respond to human behavior more intelligently and efficiently. Existing deep learning models are complex to deal with the low illumination, diverse viewpoints, and cluttered backgrounds, which require substantial computing resources and are not appropriate for edge devices. Furthermore, without an effective video analysis technique it processes entire frames, resulting inadequate performance. To address these key challenges, a cloud-assisted IoT computing framework is proposed for HAR in uncertain low-lighting environments, which is mainly composed of two tiers: edge and cloud computing. Initially, a lightweight Convolutional Neural Network (CNN) model is developed which is responsible to enhance the low-light frames, followed by the human detection algorithm to process the selective frames, thus enabling efficient resource utilization. Next, these refined frames are then transmitted to the cloud for accurate HAR, where dual stream CNN and transformer fusion network extract both short-and long-range spatiotemporal discriminative features followed by proposed Optimized Parallel Sequential Temporal Network (OPSTN) with squeeze and excitation attention to efficiently learn HAR in complex scenarios. Finally, extensive experiments are conducted over three challenging HAR datasets to deeply examine the proposed framework from various perspectives such as complex activity recognition, lowlighting, etc., where the results are outperformed compared with the state-of-art methods.

A Dual Stream Generative Adversarial Network with Phase Awareness for Speech Enhancement

Implementing single-channel speech enhancement under unknown noise conditions is a challenging problem. Most existing time-frequency domain methods are based on the amplitude spectrogram, and these methods often ignore the phase mismatch between noisy speech and clean speech, which largely limits the performance of speech enhancement. To solve the phase mismatch problem and further improve enhancement performance, this paper proposes a dual-stream Generative Adversarial Network (GAN) with phase awareness, named DPGAN. Our generator uses a dual-stream structure to predict amplitude and phase separately and adds an information communication module between the two streams to fully apply the phase information. To make the prediction more efficient, we apply Transformer to build the generator, which can learn the sound’s structural properties more easily. Finally, we designed a perceptually guided discriminator that quantitatively evaluates the quality of speech, optimising the generator for specific evaluation metrics. We conducted experiments on the most widely used Voicebank-DEMAND dataset and DPGAN achieved state-of-the-art on most metrics.

Dual stream network with attention mechanism for photovoltaic power forecasting

The operations of renewable power generation systems highly depend on precise Photovoltaic (PV) power forecasting, providing significant economic, and environmental advantages for energy efficient buildings and urban energy systems. However, precise PV power forecasting, particularly, solar power is more challenging due to solar energy intermittence, instability, and randomness. These challenges hinder the integration of PV into smart grids, where accurate power forecasting is a promising solution in this direction, providing effective planning and management services. Therefore, in this work, we introduce a dual-stream network for accurate PV forecasting. The proposed network parallelly learns spatial patterns using convolutional network and temporal representations via sequential learning algorithm. These features are then integrated together to form a single, yet representative feature vector used as an input to self-attention mechanism to further select the optimal features for PV power forecasting. To the best of our knowledge, the proposed dual stream network with advanced features selection mechanism is a pioneering approach for time series analysis, narrowed towards PV power forecasting. We derive our network after a series of experimentations involving solo and hybrid models, resulting in higher forecasting accuracy against state-of-the-art models.

Semi-Supervised Dual Stream Segmentation Network for Fundus Lesion Segmentation.

Accurate segmentation of retinal images can assist ophthalmologists to determine the degree of retinopathy and diagnose other systemic diseases. However, the structure of the retina is complex, and different anatomical structures often affect the segmentation of fundus lesions. In this paper, a new segmentation strategy called a dual stream segmentation network embedded into a conditional generative adversarial network is proposed to improve the accuracy of retinal lesion segmentation. First, a dual stream encoder is proposed to utilize the capabilities of two different networks and extract more feature information. Second, a multiple level fuse block is proposed to decode the richer and more effective features from the two different parallel encoders. Third, the proposed network is further trained in a semi-supervised adversarial manner to leverage from labeled images and unlabeled images with high confident pseudo labels, which are selected by the dual stream Bayesian segmentation network. An annotation discriminator is further proposed to reduce the negativity that prediction tends to become increasingly similar to the inaccurate predictions of unlabeled images. The proposed method is cross-validated in 384 clinical fundus fluorescein angiography images and 1040 optical coherence tomography images. Compared to state-of-the-art methods, the proposed method can achieve better segmentation of retinal capillary non-perfusion region and choroidal neovascularization.

A Key-Based Multi-Mode Clock-Controlled Stream Cipher for Real-Time Secure Communications of IoT

With the rapid development of the Internet and wireless communications, as well as the popularization of personal communication systems, the security of real-time communications is demanded. The efficient technology of stream ciphers can satisfy this requirement of security. In this paper, to enhance the security strength of stream ciphers, we design a key-based multi-mode clock-controlled stream cipher for real-time secure communications of the Internet of things (IoT). The proposed stream cipher is equipped with a multi-mode depending on the key. The different working modes are shipped with different encrypting circuits depending on the user’s key. We analyze the period, the linear complexity, and use known attacks to verify the security strength of the proposed cipher. Compared with existing dual mode clock-controlled stream ciphers, the merits of our proposed cipher are its long period, high linear complexity, low hardware complex, low initialization clock, and simplicity in mode switching. Furthermore, the proposed cipher passes the FIPS PUB 140-1 and SP800-22 tests, obtaining at least 97.00%.

ADS-detector: An attention-based dual stream adversarial example detection method

Adversarial attacks seriously threaten the security of machine learning models. Thus, detecting adversarial examples has become an important and interesting research topic facing various adversarial attacks. However, the majority of existing adversarial example detection algorithms cannot perform well in detecting adversarial examples with slight perturbations. In this paper, we propose a novel attention-based dual stream detector (ADS-Detector) that can address the detection of adversarial examples with both slight and large perturbations. Specifically, we first design a data process module to generate pixel and prediction confidence stream data from the raw image. Then, we propose an N-layer attention module to extract the channel and spatial feature weights between the pixel and prediction confidence stream data. Eventually, we feed the dual-stream data into the same subdetection model with a convolutional block attention module; then, the output results are combined to determine whether the input image is an adversarial example or not. To validate the performance, we conduct extensive experiments on three public datasets: CIFAR10, Dogs vs. Cats and ImageNet. After sufficient analysis of the simulation results, we find that our proposed method outperforms the others for the detection of adversarial attacks generated by the considered attack methods.

Bi-Unet: A Dual Stream Network for Real-Time Highway Surface Segmentation

Highway surface segmentation consists of extracting road surface at pixel-level from the surveillance camera view. Since the intelligent traffic event detection task does not require the detection of off-road scene, the segmentation of highway surface is of great demand. However, it is challenging to accurately extract road surface in real time scenarios. To cope with the above issues, Bi-Unet, a dual stream lightweight network is proposed. Firstly, the dual stream structure enhances segmentation performance on the narrow remote end of the highway and preserves the detailed border information. Then, to perform real-time segmentation, a novel lightweight module (LSM) is introduced to lighten the model and provide higher segmentation accuracy. Moreover, to ensure road segmentation for complex scenes, a Road Attention Network (RAN) module is proposed. Lastly, due to the lack of a suitable benchmark dataset serve for the highway segmentation problem, a new large and high-quality segmentation dataset named Highway-Surface-Free (HSF) is proposed in this paper, which is collected from the perspective of highway surveillance cameras under all-day and all-weather conditions. Compared with the state of arts, the extensive experimental results verify that our proposed Bi-Unet achieves the best overall performance on our proposed HSF dataset.

Dual Stream Conditional Generative Adversarial Network Fusion for Video Abnormal Behavior Detection

Deep learning has been successfully applied to video anomaly detection. However, the way that deep network learn spatio-temporal features autonomously will ignore the specificity of different pattern features. Therefore, this paper focuses on how to efficiently learn deep appearance feature, introduces the idea of learning appearance information by predicting future frame, and proposes dual stream conditional generative adversarial network fusion for video abnormal behavior detection. The video frame and its corresponding optical flow image are transferred to the conditional generative adversarial network to learn the motion feature representation. In addition, inputting the video frame and its corresponding future frame to the network to generate the appearance representation complementary to motion feature. The model is only trained with normal events, therefore it is not able to generate abnormal events accurately. During the test, for the foreground moving targets, the images generated by the model are compared with the ground truth to obtain a two-stream anomaly probability distribution model based on the mean square error used to achieve the purpose of region anomaly detection. Experiments on the public datasets show that the proposed method can effectively detect and locate abnormal behaviors in the video.

Solar Power Prediction Using Dual Stream CNN-LSTM Architecture.

The integration of solar energy with a power system brings great economic and environmental benefits. However, the high penetration of solar power is challenging due to the operation and planning of the existing power system owing to the intermittence and randomicity of solar power generation. Achieving accurate predictions for power generation is important to provide high-quality electric energy for end-users. Therefore, in this paper, we introduce a deep learning-based dual-stream convolutional neural network (CNN) and long short-term nemory (LSTM) network followed by a self-attention mechanism network (DSCLANet). Here, CNN is used to learn spatial patterns and LSTM is incorporated for temporal feature extraction. The output spatial and temporal feature vectors are then fused, followed by a self-attention mechanism to select optimal features for further processing. Finally, fully connected layers are incorporated for short-term solar power prediction. The performance of DSCLANet is evaluated on DKASC Alice Spring solar datasets, and it reduces the error rate up to 0.0136 MSE, 0.0304 MAE, and 0.0458 RMSE compared to recent state-of-the-art methods.

TnTViT-G: Transformer in Transformer Network for Guidance Super Resolution

Image Super Resolution is a potential approach that can improve the image quality of low-resolution optical sensors, leading to improved performance in various industrial applications. It is important to emphasize that most state-of-the-art super resolution algorithms often use a single channel of input data for training and inference. However, this practice ignores the fact that the cost of acquiring high-resolution images in various spectral domains can differ a lot from one another. In this paper, we attempt to exploit complementary information from a low-cost channel (visible image) to increase the image quality of an expensive channel (infrared image). We propose a dual stream Transformer-based super resolution approach that uses the visible image as a guide to super-resolve another spectral band image. To this end, we introduce Transformer in Transformer network for Guidance super resolution, named TnTViT-G, an efficient and effective method that extracts the features of input images via different streams and fuses them together at various stages. In addition, unlike other guidance super resolution approaches, TnTViT-G is not limited to a fixed upsample size and it can generate super-resolved images of any size. Extensive experiments on various datasets show that the proposed model outperforms other state-of-the-art super resolution approaches. TnTViT-G surpasses state-of-the-art methods by up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.19 \sim 2.3dB $ </tex-math></inline-formula> , while it is memory efficient.

Limb Apraxias: The Influence of Higher Order Perceptual and Semantic Deficits in Motor Recovery After Stroke.

Stroke is a leading cause of disability worldwide. Limb apraxia is a group of higher order motor disorders associated with greater disability and dependence after stroke. Original neuropsychology studies distinguished separate brain pathways involved in perception and action, known as the dual stream hypothesis. This framework has allowed a better understanding of the deficits identified in Limb Apraxia. In this review, we propose a hierarchical organization of this disorder, in which a distinction can be made between several visuomotor pathways that lead to purposeful actions. Based on this, executive apraxias (such as limb kinetic apraxia) cause deficits in executing fine motor hand skills, and intermediate apraxias (such as optic ataxia and tactile apraxia) cause deficits in reaching to grasp and manipulating objects in space. These disorders usually affect the contralesional limb. A further set of disorders collectively known as limb apraxias include deficits in gesture imitation, pantomime, gesture recognition, and object use. These deficits are due to deficits in integrating perceptual and semantic information to generate complex movements. Limb apraxias are usually caused by left-hemisphere lesions in right-handed stroke patients, affecting both limbs. The anterior- to posterior-axis of brain areas are disrupted depending on the increasing involvement of perceptual and semantic processes with each condition. Lower-level executive apraxias are linked to lesions in the frontal lobe and the basal ganglia, while intermediate apraxias are linked to lesions in dorso-dorsal subdivisions of the dorsal fronto-parietal networks. Limb apraxias can be caused by lesions in both dorsal and ventral subdivisions including the ventro-dorsal stream and a third visuomotor pathway, involved in body schema and social cognition. Rehabilitation of these disorders with behavioral therapies has aimed to either restore perceptuo-semantic deficits or compensate to overcome these deficits. Further studies are required to better stratify patients, using modern neurophysiology and neuroimaging techniques, to provide targeted and personalized therapies for these disorders in the future.

DS-UNet: Dual-Stream U-Net for Oil Spill Detection of SAR Image

DS-UNet: Dual-Stream U-Net for Oil Spill Detection of SAR Image

A dual stream hierarchical transformer for starvation grading of golden pomfret in marine aquaculture

IntroductionFish starvation grading can provide feeding information for aquaculture, reducing the cost of lures and helping to promote the unmanned and intelligent process of offshore aquaculture.MethodsIn this study, we used golden pompano as the experimental object to address the fish starvation grading problem in the marine culture vessel environment, and proposed the dual stream hierarchical transformer to provide additional temporal information for the starvation grading task, which improved the grading accuracy. We first built a dual stream dataset with both spatial and temporal channel, and divided the fish school starvation status into five levels (very bloated, a little bloated, modest, a little starving, very starving) according to the feeding time and experience. Based on the marine image characteristics, we proposed a dual stream hierarchical transformer with hierarchical convolutional network, composite fusion convolution and transformer.Results and discussionWe finally evaluated the efficacy of the model based on qualitative and quantitative analyses, revealing that the proposed dual stream hierarchical transformer achieved the state-of-the-art starvation grading performance with a test accuracy of 98.05%. Our model outperformed other mainstream models, including VGG, ResNet, attentionbased model and other fish status grading related model. Field tests on the vessel further suggested that the model can be applied to the mariculture environment of golden pomfret.

Collaborative Learning Attention Network Based on RGB Image and Depth Image for Surface Defect Inspection of No-Service Rail

Surface defect inspection of no-service rail is important for safety of railway transportation. However, there are several challenges of irregular defect boundary, similar foreground and background for no-service rail surface defect inspection. To deal with the above challenges, depth image is used to provide complementary spatial information to RGB image. In recent years, with the development of deep learning and computer vision technology, intelligent inspection of defect has made great progress. We propose a neural network named collaborative learning attention network (CLANet) for no-service rail surface defect inspection. Our method can inspect the defect object of rail surface and segment the accurate region of that defect. The proposed method consists of three main stages: feature extraction, cross-modal information fusion, and defect location and segmentation. A multimodal attention block is proposed to highlight complex defect object with a new cross-modal fusion strategy. Furthermore, dual stream decoder enriches the representation of advanced features and avoids the dilution of information in the decoding stage. Suffering from the scarcity of defective data, an industrial RGB-D dataset NEU RSDDS-AUG is built. Finally, ablation studies verify the effectiveness of our proposed method. Compared with the existing nine state-of-the-art methods, CLANet has achieved improvements in all five parameters. Our method is also competitive on four public benchmark datasets.

Dual geometric perception for cross-domain road segmentation

Road segmentation plays an important role in navigation systems and autonomous driving. However, many methods in road segmentation are based on supervised learning and suffer from performance degradation in the real world. There is a certain domain gap (distribution shift problem) between the source domain (training data) and the target domain (testing data). In this paper, we propose a Dual-Geometric Perception (DGP) approach for cross-domain road segmentation, which jointly uses semantic and dual-geometric information to learn the domain-invariant feature for road segmentation. First, we propose an RGB-N dual stream network structure, which effectively fuses normal vector information and RGB information to reduce domain gap. Moreover, a dual geometric adversarial learning strategy is proposed to utilize depth-aware and normal vector features to perform better domain alignment. Furthermore, a self-training learning strategy is used to further improve the model’s generalizability in the target domain. Extensive experiments demonstrate that our proposed DGP achieves superior performance on lane-to-lane and lane-to-sidewalk road domain adaptation tasks.

Nested Named Entity Recognition Based on Dual Stream Feature Complementation

Named entity recognition is a basic task in natural language processing, and there is a large number of nested structures in named entities. Nested named entities become the basis for solving many tasks in NLP. A nested named entity recognition model based on dual-flow features complementary is proposed for obtaining efficient feature information after text coding. Firstly, sentences are embedded at both the word level and the character level of the words, then sentence context information is obtained separately via the neural network Bi-LSTM; Afterward, two vectors perform low-level feature complementary to reinforce low-level semantic information; Sentence-local information is captured with the multi-head attention mechanism, then the feature vector is sent to the high-level feature complementary module to obtain deep semantic information; Finally, the entity word recognition module and the fine-grained division module are entered to obtain the internal entity. The experimental results show that the model has a great improvement in feature extraction compared to the classical model.

Brain Functional Imaging Anatomy.

Human brain function is an increasingly complex framework that has important implications in clinical medicine. In this review, the anatomy of the most commonly assessed brain functions in clinical neuroradiology, including motor, language, and vision, is discussed. The anatomy and function of the primary and secondary sensorimotor areas are discussed with clinical case examples. Next, the dual stream of language processing is reviewed, as well as its implications in clinical medicine and surgical planning. Last, the authors discuss the striate and extrastriate visual cortex and review the dual stream model of visual processing.

Human Motion Pattern Recognition and Feature Extraction: An Approach Using Multi-Information Fusion.

An exoskeleton is a kind of intelligent wearable device with bioelectronics and biomechanics. To realize its effective assistance to the human body, an exoskeleton needs to recognize the real time movement pattern of the human body in order to make corresponding movements at the right time. However, it is of great difficulty for an exoskeleton to fully identify human motion patterns, which are mainly manifested as incomplete acquisition of lower limb motion information, poor feature extraction ability, and complicated steps. Aiming at the above consideration, the motion mechanisms of human lower limbs have been analyzed in this paper, and a set of wearable bioelectronics devices are introduced based on an electromyography (EMG) sensor and inertial measurement unit (IMU), which help to obtain biological and kinematic information of the lower limb. Then, the Dual Stream convolutional neural network (CNN)-ReliefF was presented to extract features from the fusion sensors’ data, which were input into four different classifiers to obtain the recognition accuracy of human motion patterns. Compared with a single sensor (EMG or IMU) and single stream CNN or manual designed feature extraction methods, the feature extraction based on Dual Stream CNN-ReliefF shows better performance in terms of visualization performance and recognition accuracy. This method was used to extract features from EMG and IMU data of six subjects and input these features into four different classifiers. The motion pattern recognition accuracy of each subject under the four classifiers is above 97%, with the highest average recognition accuracy reaching 99.12%. It can be concluded that the wearable bioelectronics device and Dual Stream CNN-ReliefF feature extraction method proposed in this paper enhanced an exoskeleton’s ability to capture human movement patterns, thus providing optimal assistance to the human body at the appropriate time. Therefore, it can provide a novel approach for improving the human-machine interaction of exoskeletons.