Dual-stream Model Research Articles

Dual-Stream Enhanced Deep Network for Transmission Near-Infrared Dorsal Hand Vein Age Estimation with Attention Mechanisms

Dorsal hand vein recognition, with unique stable and reliable advantages, has attracted considerable attention from numerous researchers. In this case, the dorsal hand vein images captured by the means of transmission infrared imaging are clearer than those collected by other infrared methods, enabling it to be more suitable for the biometric applications. However, less attention is paid to individual age estimation based on dorsal hand veins. To this end, this paper proposes an efficient dorsal hand vein age estimation model using a deep neural network with attention mechanisms. Specifically, a convolutional neural network (CNN) is developed to extract the expressive features for age estimation. Simultaneously, another deep residual network is leveraged to strengthen the representation ability on subtle dorsal vein textures. Moreover, variable activation functions and pooling layers are integrated into the respective streams to enhance the nonlinearity modeling of the dual-stream model. Finally, a dynamic attention mechanism module is embedded into the dual-stream network to achieve multi-modal collaborative enhancement, guiding the model to concentrate on salient age-specific features. To evaluate the performance of dorsal hand vein age estimation, this work collects dorsal hand vein images using the transmission near-infrared spectrum from 300 individuals across various age groups. The experimental results show that the dual-stream enhanced network with the attention mechanism significantly improves the accuracy of dorsal hand vein age estimation in comparison with other deep learning approaches, indicating the potential of using near-infrared dorsal hand vein imaging and deep learning technology for efficient human age estimation.

How Can Graph Theory Inform the Dual-stream Model of Speech Processing? A Resting-state Functional Magnetic Resonance Imaging Study of Stroke and Aphasia Symptomology.

The dual-stream model of speech processing describes a cortical network involved in speech processing. However, it is not yet known if the dual-stream model represents actual intrinsic functional brain networks. Furthermore, it is unclear how disruptions after a stroke to the functional connectivity of the dual-stream model's regions are related to speech production and comprehension impairments seen in aphasia. To address these questions, in the present study, we examined two independent resting-state fMRI data sets: (1) 28 neurotypical matched controls and (2) 28 chronic left-hemisphere stroke survivors collected at another site. We successfully identified an intrinsic functional network among the dual-stream model's regions in the control group using functional connectivity. We then used both standard functional connectivity analyses and graph theory approaches to determine how this connectivity may predict performance on clinical aphasia assessments. Our findings provide evidence that the dual-stream model of speech processing is an intrinsic network as measured via resting-state MRI and that functional connectivity of the hub nodes of the dual-stream network defined by graph theory methods, but not overall average network connectivity, is weaker in the stroke group than in the control participants. In addition, the functional connectivity of the hub nodes predicted linguistic impairments on clinical assessments. In particular, the relative strength of connectivity of the right hemisphere's homologues of the left dorsal stream hubs to the left dorsal hubs, versus to the right ventral stream hubs, is a particularly strong predictor of poststroke aphasia severity and symptomology.

DSTANet: learning a dual-stream model for anomaly driving action detection using spatio-temporal and appearance features

DSTANet: learning a dual-stream model for anomaly driving action detection using spatio-temporal and appearance features

What Is a Visual Stream?

The dual stream model of the human and non-human primate visual systems remains Leslie Ungerleider's (1946-2020) most indelible contribution to visual neuroscience. In this model, a dorsal "where" stream specialized for visuospatial representation extends through occipitoparietal cortex, whereas a ventral "what" stream specialized for representing object qualities extends through occipito-temporal cortex. Over time, this model underwent a number of revisions and expansions. In one of her last scientific contributions, Leslie proposed a third visual stream specialized for representing dynamic signals related to social perception. This alteration invites the question: What is a visual stream, and how are different visual streams individuated? In this article, we first consider and reject a simple answer to this question based on a common idealizing visualization of the model, which conflicts with the complexities of the visual system that the model was intended to capture. Next, we propose a taxonomic answer that takes inspiration from the philosophy of science and Leslie's body of work, which distinguishes between neural mechanisms, pathways, and streams. In this taxonomy, visual streams are superordinate to pathways and mechanisms and provide individuation conditions for determining whether collections of cortical connections delineate different visual streams. Given this characterization, we suggest that the proposed third visual stream does not yet meet these conditions, although the tripartite model still suggests important revisions to how we think about the organization of the human and non-human primate visual systems.

Musical Sophistication and Multilingualism: Effects on Arcuate Fasciculus Characteristics.

The processing of auditory stimuli which are structured in time is thought to involve the arcuate fasciculus, the white matter tract which connects the temporal cortex and the inferior frontal gyrus. Research has indicated effects of both musical and language experience on the structural characteristics of the arcuate fasciculus. Here, we investigated in a sample of n = 84 young adults whether continuous conceptualizations of musical and multilingual experience related to structural characteristics of the arcuate fasciculus, measured using diffusion tensor imaging. Probabilistic tractography was used to identify the dorsal and ventral parts of the white matter tract. Linear regressions indicated that different aspects of musical sophistication related to the arcuate fasciculus' volume (emotional engagement with music), volumetric asymmetry (musical training and music perceptual abilities), and fractional anisotropy (music perceptual abilities). Our conceptualization of multilingual experience, accounting for participants' proficiency in reading, writing, understanding, and speaking different languages, was not related to the structural characteristics of the arcuate fasciculus. We discuss our results in the context of other research on hemispheric specializations and a dual-stream model of auditory processing.

E-FNet: A EEG-fNIRS dual-stream model for Brain–Computer Interfaces

With the rapid advancements in Brain–Computer Interfaces (BCI) and multimodal technology, researchers are actively exploring the intricate domain of multimodal BCI. The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) in BCI systems has garnered significant attention due to the combination of strengths from both modalities and the overcoming of limitations inherent in standalone systems. In this study, the E-FNet architecture is introduced, a (2+1)D dual-stream model explicitly designed to process multimodal EEG and fNIRS data for brain–machine interface applications. By leveraging the (2+1)D architecture and adopting a bias-free design, E-FNet efficiently processes multimodal data. A unified 4D tensor representation enables the achievement of both temporal and spatial alignment of EEG and fNIRS signals. For Motor Imagery (MI) and Mental Arithmetic (MA) tasks, the early integration of EEG and fNIRS signals yields remarkable accuracy rates of 95.86% and 95.80% respectively, outperforming the accuracy obtained with EEG signals alone. Moreover, the minimal variability in results across subjects underscores the complementary effects achieved through the integration of different signals.

Video-based AI module with raw-scale and ROI-scale information for thyroid nodule diagnosis

ObjectivesUltrasound examination is a primary method for detecting thyroid lesions in clinical practice. Incorrect ultrasound diagnosis may lead to delayed treatment or unnecessary biopsy punctures. Therefore, our objective is to propose an artificial intelligence model to increase the precision of thyroid ultrasound diagnosis and reduce puncture rates. MethodsWe consecutively collected ultrasound recordings from 672 patients with 845 nodules across two Chinese hospitals. This dataset was divided into training, validation, and internal test sets in a ratio of 7:1:2. We constructed and tested six different model variants based on different video feature distillation strategies and whether additional information from ROI (Region of Interest) scales was used. The models' performances were evaluated using the internal test set and an additional external test set containing 126 nodules from a third hospital. ResultsThe dual-stream model, which contains both raw-scale and ROI-scale streams with the time dimensional convolution layer, achieved the best performance on both internal and external test sets. On the internal test set, it achieved an AUROC (Area Under Receiver Operating Characteristic Curve) of 0.969 (95 % confidence interval, CI: 0.944–0.993) and an accuracy of 92.6 %, outperforming other variants (AUROC: 0.936–0.955, accuracy: 80.2%–88.3 %) and experienced radiologists (accuracy: 91.9 %). The AUROC of the best model in the external test was 0.931 (95 % CI: 0.890–0.972). ConclusionIntegrating a dual-stream model with additional ROI scale information and the time dimensional convolution layer can improve performance in diagnosing thyroid ultrasound videos.

Deep learning based fine-grained recognition technology for basketball movements

The technical essentials of basketball are very complex, in order to accomplish the identification of fine-grained basketball movements and improve the quality of basketball training, the study innovatively proposes a region screening strategy to obtain the local region information of image frames, and draws on the SlowFast network structure to design the static and dynamic detail models, and finally utilizes the 3D attention feature fusion module to complete the multi-feature relationship processing and recognition. Performance testing experiments have confirmed that the dual stream model based on region filtering has achieved a maximum improvement in average accuracy of 16.35 % and 60.36 % in the recognition of RGB images, preserving relatively complete local effective region information of RGB images. The model based on channel domain attention mechanism improves its average accuracy to over 60 %. The model recognition accuracy calculation time after dual stream fusion is reduced by about 50 %. The model based on 3D attention feature fusion module exhibits stronger robustness when the feature dimension changes. And the loss value of this model rapidly decreases and converges to the minimum loss value, with its accuracy and recall reaching 90.03 % and 88.47 % respectively, and its AUC value reaching a maximum of 0.761. The design of the study contributes to the digital development of teaching and training in basketball and enriches the theoretical body of knowledge on deep learning and action recognition.

The Second Wave of Attention Economics. Attention as a Universal Symbolic Currency on Social Media and beyond

Abstract Since the advent of social media, capturing and holding the attention of people has become paramount for the success of products, political messages and even research. The economics of attention is often seen as part of the market economy. We argue that a larger societal transformation is underway, which will see attention become the defining currency that moves individuals, exchanges, and many other elements of society. This paper connects the attention economy to the institutional foundations of modernity. It then discusses how attention can be accumulated and exchanged like a currency and proposes a dual-stream model distinguishing between calcified and flow attention. Based on this model, we investigate recent developments facilitating the use of attention as a currency, and their potential impact on our daily lives more generally. We conclude by providing an outlook and concrete questions for future research to understand where the economics of attention economy is heading.

Neural correlates of musical timbre: an ALE meta-analysis of neuroimaging data.

Timbre is a central aspect of music that allows listeners to identify musical sounds and conveys musical emotion, but also allows for the recognition of actions and is an important structuring property of music. The former functions are known to be implemented in a ventral auditory stream in processing musical timbre. While the latter functions are commonly attributed to areas in a dorsal auditory processing stream in other musical domains, its involvement in musical timbre processing is so far unknown. To investigate if musical timbre processing involves both dorsal and ventral auditory pathways, we carried out an activation likelihood estimation (ALE) meta-analysis of 18 experiments from 17 published neuroimaging studies on musical timbre perception. We identified consistent activations in Brodmann areas (BA) 41, 42, and 22 in the bilateral transverse temporal gyri, the posterior superior temporal gyri and planum temporale, in BA 40 of the bilateral inferior parietal lobe, in BA 13 in the bilateral posterior Insula, and in BA 13 and 22 in the right anterior insula and superior temporal gyrus. The vast majority of the identified regions are associated with the dorsal and ventral auditory processing streams. We therefore propose to frame the processing of musical timbre in a dual-stream model. Moreover, the regions activated in processing timbre show similarities to the brain regions involved in processing several other fundamental aspects of music, indicating possible shared neural bases of musical timbre and other musical domains.

Brain age estimation with a greedy dual-stream model for limited datasets

Brain age estimation involves predicting an individual’s biological age from their brain images. This process offers valuable insights into the aging process and the progression of neurodegenerative diseases. Conducting large-scale datasets for medical image analysis is a challenging and time-consuming task. Existing approaches mostly depend on large datasets, which are hard to obtain and expensive. These approaches also require sophisticated, resource-intensive models with a large number of parameters, necessitating a significant amount of processing power. As a result, there is a vital need to develop innovative methods that can achieve robust performance with limited datasets and efficient use of computational resources. This paper proposes a novel slice-based dual-stream method called GDSM (Greedy Dual-Stream Model) for brain age estimation. This method addresses the limitations of large dataset requirements and computational resource intensiveness. The proposed method incorporates local and global aspects of the brain, thereby refining the focus on specific target regions. The approach employs four backbones to predict ages based on local and global features, complemented by a final model for age correction. Our method demonstrates a Mean Absolute Error (MAE) of 3.25 years on the IBID’s test set, which contains only 289 subjects. To demonstrate the robustness of our approach for any small dataset, we analyzed the proposed method with the IXI dataset and achieved an MAE of 4.18 years on the IXI’s test set. By leveraging dual-stream and greedy strategies, this approach achieves efficiency and robust performance, making it comparable with other state-of-the-art methods. The code for the GDSM model is available at https://github.com/iman2693/GDSM.

Temperature Structure Inversion of Mesoscale Eddies in the South China Sea Based on Deep Learning

Mesoscale eddies are common in global oceans, playing crucial roles in ocean dynamics, ocean circulation, and heat transport, and their vertical structures can affect the water layers from tens to thousands of meters. In this study, we integrated sea surface height and sea surface temperature data into deep learning methods to study the mesoscale eddy subsurface temperature structure and to explore the relationship between sea surface data and eddy subsurface layers. In this study, we introduce Dual_EddyNet, a deep learning algorithm designed to invert the subsurface temperature structure of mesoscale eddies. Using this algorithm, we explore the impact of the sea surface height and sea surface temperature on the subsurface temperature structure inversion of mesoscale eddies. Furthermore, we compare different data fusion strategies, namely single-stream neural networks and dual-stream neural networks, to validate the effectiveness of the dual-stream model. To capture the interrelations among surface data and integrate feature information across various dimensions, we introduce the Triplet Attention Mechanism. The experimental results demonstrate that the proposed Dual_EddyNet performs well in reconstructing the three-dimensional structure of mesoscale eddies in the South China Sea (within a depth of 1000 m), with an inversion accuracy of 91.44% for cyclonic eddies and 95.25% for anticyclonic eddies. This algorithm provides a new method for inverting the subsurface temperatures of mesoscale eddies, and can not only be directly deployed in systems, embedded in ship moving platforms, etc., but can also provide a data reference for assimilations and numerical simulations, demonstrating its rich application potential.

Actionable code smell identification with fusion learning of metrics and semantics

Code smell detection is one of the essential tasks in the field of software engineering. Identifying whether a code snippet has a code smell is subjective and varies by programming language, developer, and development method. Moreover, developers tend to focus on code smells that have a real impact on development and ignore insignificant ones. However, existing static code analysis tools and code smell detection approaches exhibit a high false positive rate in detecting code smells, which makes insignificant smells drown out those smells that developers value. Therefore, accurately reporting those actionable code smells that developers tend to spend energy on refactoring can prevent developers from getting lost in the sea of smells and improve refactoring efficiency. In this paper, we aim to detect actionable code smells that developers tend to refactor. Specifically, we first collect actionable and non-actionable code smells from projects with numerous historical versions to construct our datasets. Then, we propose a dual-stream model for fusion learning of code metrics and code semantics to detect actionable code smells. On the one hand, code metrics quantify the code's structure and even some rules or patterns, providing fundamental information for detecting code smells. On the other hand, code semantics encompass information about developers' refactoring tendencies, which prove valuable in detecting actionable code smells. Extensive experiments show that our approach can detect actionable code smells more accurately compared to existing approaches.

SBIT-Fuse: Infrared and visible image fusion based on Symmetrical Bilateral interaction and Transformer

Infrared and visible image fusion, as a typical image enhancement task, aims to generate an image containing complementary information. Existing deep learning-based methods usually use a dual-stream model to extract features, which suffers from insufficient feature interaction in the feature extraction process. In order to effectively utilize information, this work proposes a Symmetrical Bilateral Interaction and Transformer fusion method (SBIT-Fuse) that is simple and efficient for constructing a dual-stream interaction network. Specifically, inspired by the problem of dead ReLU, we propose a Symmetrical Bilateral Interaction (SBI) module which consists of a certain number of cascaded cross domain activation interaction (CDAI) layers, where the deactivation information of the ReLU rectifier is transferred from one flow to another instead of being discarded. The proposed module exploits the drawbacks of ReLU to efficiently build the interaction of two streams and combine spatial and channel attention mechanisms, achieving the enhancement of feature extraction. Besides, our network is end-to-end and a transformer module is employed for extracting long-range semantic information. Moreover, combining a novel designed loss function, the proposed method can generate higher-quality fusion images. The comparative experiments with existing advanced methods and ablation studies on publicly datasets demonstrate the effectiveness of the proposed approach. Our code can be found at https://github.com/ljx111790/SBIT-Fuse.

Transfer learning–based PET/CT three-dimensional convolutional neural network fusion of image and clinical information for prediction of EGFR mutation in lung adenocarcinoma

BackgroundTo introduce a three-dimensional convolutional neural network (3D CNN) leveraging transfer learning for fusing PET/CT images and clinical data to predict EGFR mutation status in lung adenocarcinoma (LADC).MethodsRetrospective data from 516 LADC patients, encompassing preoperative PET/CT images, clinical information, and EGFR mutation status, were divided into training (n = 404) and test sets (n = 112). Several deep learning models were developed utilizing transfer learning, involving CT-only and PET-only models. A dual-stream model fusing PET and CT and a three-stream transfer learning model (TS_TL) integrating clinical data were also developed. Image preprocessing includes semi-automatic segmentation, resampling, and image cropping. Considering the impact of class imbalance, the performance of the model was evaluated using ROC curves and AUC values.ResultsTS_TL model demonstrated promising performance in predicting the EGFR mutation status, with an AUC of 0.883 (95%CI = 0.849–0.917) in the training set and 0.730 (95%CI = 0.629–0.830) in the independent test set. Particularly in advanced LADC, the model achieved an AUC of 0.871 (95%CI = 0.823–0.919) in the training set and 0.760 (95%CI = 0.638–0.881) in the test set. The model identified distinct activation areas in solid or subsolid lesions associated with wild and mutant types. Additionally, the patterns captured by the model were significantly altered by effective tyrosine kinase inhibitors treatment, leading to notable changes in predicted mutation probabilities.ConclusionPET/CT deep learning model can act as a tool for predicting EGFR mutation in LADC. Additionally, it offers clinicians insights for treatment decisions through evaluations both before and after treatment.

Common and distinct patterns underlying different linguistic tasks: multivariate disconnectome symptom mapping in poststroke patients.

Numerous studies have been devoted to neural mechanisms of a variety of linguistic tasks (e.g. speech comprehension and production). To date, however, whether and how the neural patterns underlying different linguistic tasks are similar or differ remains elusive. In this study, we compared the neural patterns underlying 3 linguistic tasks mainly concerning speech comprehension and production. To address this, multivariate regression approaches with lesion/disconnection symptom mapping were applied to data from 216 stroke patients with damage to the left hemisphere. The results showed that lesion/disconnection patterns could predict both poststroke scores of speech comprehension and production tasks; these patterns exhibited shared regions on the temporal pole of the left hemisphere as well as unique regions contributing to the prediction for each domain. Lower scores in speech comprehension tasks were associated with lesions/abnormalities in the superior temporal gyrus and middle temporal gyrus, while lower scores in speech production tasks were associated with lesions/abnormalities in the left inferior parietal lobe and frontal lobe. These results suggested an important role of the ventral and dorsal stream pathways in speech comprehension and production (i.e. supporting the dual stream model) and highlighted the applicability of the novel multivariate disconnectome-based symptom mapping in cognitive neuroscience research.

Asymmetrical Contrastive Learning Network via Knowledge Distillation for No-Service Rail Surface Defect Detection.

Owing to extensive research on deep learning, significant progress has recently been made in trackless surface defect detection (SDD). Nevertheless, existing algorithms face two main challenges. First, while depth features contain rich spatial structure features, most models only accept red-green-blue (RGB) features as input, which severely constrains performance. Thus, this study proposes a dual-stream teacher model termed the asymmetrical contrastive learning network (ACLNet-T), which extracts both RGB and depth features to achieve high performance. Second, the introduction of the dual-stream model facilitates an exponential increase in the number of parameters. As a solution, we designed a single-stream student model (ACLNet-S) that extracted RGB features. We leveraged a contrastive distillation loss via knowledge distillation (KD) techniques to transfer rich multimodal features from the ACLNet-T to the ACLNet-S pixel by pixel and channel by channel. Furthermore, to compensate for the lack of contrastive distillation loss that focuses exclusively on local features, we employed multiscale graph mapping to establish long-range dependencies and transfer global features to the ACLNet-S through multiscale graph mapping distillation loss. Finally, an attentional distillation loss based on the adaptive attention decoder (AAD) was designed to further improve the performance of the ACLNet-S. Consequently, we obtained the ACLNet-S ∗ , which achieved performance similar to that of ACLNet-T, despite having a nearly eightfold parameter count gap. Through comprehensive experimentation using the industrial RGB-D dataset NEU RSDDS-AUG, the ACLNet-S ∗ (ACLNet-S with KD) was confirmed to outperform 16 state-of-the-art methods. Moreover, to showcase the generalization capacity of ACLNet-S ∗ , the proposed network was evaluated on three additional public datasets, and ACLNet-S ∗ achieved comparable results. The code is available at https://github.com/Yuride0404127/ACLNet-KD.

Dual-stream complex-valued convolutional network for authentic dehazed image quality assessment.

Effectively evaluating the perceptual quality of dehazed images remains an under-explored research issue. In this paper, we propose a no-reference complex-valued convolutional neural network (CV-CNN) model to conduct automatic dehazed image quality evaluation. Specifically, a novel CV-CNN is employed that exploits the advantages of complex-valued representations, achieving better generalization capability on perceptual feature learning than real-valued ones. To learn more discriminative features to analyze the perceptual quality of dehazed images, we design a dual-stream CV-CNN architecture. The dual-stream model comprises a distortion-sensitive stream that operates on the dehazed RGB image, and a haze-aware stream on a novel dark channel difference image. The distortion-sensitive stream accounts for perceptual distortion artifacts, while the haze-aware stream addresses the possible presence of residual haze. Experimental results on three publicly available dehazed image quality assessment (DQA) databases demonstrate the effectiveness and generalization of our proposed CV-CNN DQA model as compared to state-of-the-art no-reference image quality assessment algorithms.

Insights Into Detecting Adult ADHD Symptoms Through Advanced Dual-Stream Machine Learning.

Advancements in machine learning offer promising avenues for the identification of ADHD symptoms in adults, an endeavour traditionally encumbered by the intricacies of human behavioural patterns. In this paper, we introduce three innovative dual-stream models. The proposed approach utilises a novel multi-modal dataset recorded for ADHD symptoms detection, leveraging RGB video alongside facial, body posture and hand landmark data. The fusion of these different sub-modalities within video enhances the discriminative capability of the ADHD symptoms detection system. A primary objective was to maintain minimal model depth while achieving competitive performance. Through randomised search cross-validation and a rigorous leave-one-out validation scheme, the proposed model achieves high generalisability and robust symptom identification, suggesting strong potential for application in clinical environments. Evaluation boasts the state-of-the-art performance of the proposed model, demonstrating an accuracy of 98.67%, a precision of 98.01%, and a recall of 98.88%. These metrics attest to the model's ability to consistently identify ADHD symptoms while maintaining a minimal parameter footprint. This delicate balance provides a significant step forward in behavioural health analytics.

MCMNET: Multi-Scale Context Modeling Network for Temporal Action Detection.

Temporal action detection is a very important and challenging task in the field of video understanding, especially for datasets with significant differences in action duration. The temporal relationships between the action instances contained in these datasets are very complex. For such videos, it is necessary to capture information with a richer temporal distribution as much as possible. In this paper, we propose a dual-stream model that can model contextual information at multiple temporal scales. First, the input video is divided into two resolution streams, followed by a Multi-Resolution Context Aggregation module to capture multi-scale temporal information. Additionally, an Information Enhancement module is added after the high-resolution input stream to model both long-range and short-range contexts. Finally, the outputs of the two modules are merged to obtain features with rich temporal information for action localization and classification. We conducted experiments on three datasets to evaluate the proposed approach. On ActivityNet-v1.3, an average mAP (mean Average Precision) of 32.83% was obtained. On Charades, the best performance was obtained, with an average mAP of 27.3%. On TSU (Toyota Smarthome Untrimmed), an average mAP of 33.1% was achieved.