Unified Network Research Articles

Unified Cross-Structural Motion Retargeting for Humanoid Characters.

Motion retargeting for animation characters has potential applications in fields such as animation production and virtual reality. However, current methods either assume that the source and target characters have the same skeletal structure, or require designing and training specific model architectures for each structure. In this paper, we aim to address the challenge of motion retargeting across previously unseen skeletal structures with a unified dynamic graph network. The proposed approach utilizes a dynamic graph transformation module to dynamically transfer latent motion features to different structures. We also take into consideration for intricate hand movements and model both torso and hand joints as graphs in a unified manner for whole-body motion retargeting. Our model allows the use of motion data from different structures to train a unified model and learns cross-structural motion retargeting in an unsupervised manner with unpaired data. Experimental results demonstrate the superiority of the proposed method in terms of data efficiency and performance on both seen and unseen structures.

Deblurring Videos Using Spatial-Temporal Contextual Transformer with Feature Propagation.

We present a simple and effective approach to explore both local spatial-temporal contexts and non-local temporal information for video deblurring. First, we develop an effective spatial-temporal contextual transformer to explore local spatial-temporal contexts from videos. As the features extracted by the spatial-temporal contextual transformer does not model the non-local temporal information of video well, we then develop a feature propagation method to aggregate useful features from the long-range frames so that both local spatial-temporal contexts and non-local temporal information can be better utilized for video deblurring. Finally, we formulate the spatial-temporal contextual transformer with the feature propagation into a unified deep convolutional neural network (CNN) and train it in an end-to-end manner. We show that using the spatial-temporal contextual transformer with the feature propagation is able to generate useful features and makes the deep CNN model more compact and effective for video deblurring. Extensive experimental results show that the proposed method performs favorably against state-of-the-art ones on the benchmark datasets in terms of accuracy and model parameters.

ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection.

Recent camouflaged object detection (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. Apart from the high intrinsic similarity between camouflaged objects and their background, objects are usually diverse in scale, fuzzy in appearance, and even severely occluded. To this end, we propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and videos, i.e. zooming in and out. Specifically, our approach employs the zooming strategy to learn discriminative mixed-scale semantics by the multi-head scale integration and rich granularity perception units, which are designed to fully explore imperceptible clues between candidate objects and background surroundings. The former's intrinsic multi-head aggregation provides more diverse visual patterns. The latter's routing mechanism can effectively propagate inter-frame differences in spatiotemporal scenarios and be adaptively deactivated and output all-zero results for static representations. They provide a solid foundation for realizing a unified architecture for static and dynamic COD. Moreover, considering the uncertainty and ambiguity derived from indistinguishable textures, we construct a simple yet effective regularization, uncertainty awareness loss, to encourage predictions with higher confidence in candidate regions. Our highly task-friendly framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.

VatLM: Visual-Audio-Text Pre-Training With Unified Masked Prediction for Speech Representation Learning

Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.

Continual All-in-One Adverse Weather Removal With Knowledge Replay on a Unified Network Structure

Continual All-in-One Adverse Weather Removal With Knowledge Replay on a Unified Network Structure

Efficient Reflectance Capture With a Deep Gated Mixture-of-Experts.

We present a novel framework to efficiently acquire anisotropic reflectance in a pixel-independent fashion, using a deep gated mixture-of-experts. While existing work employs a unified network to handle all possible input, our network automatically learns to condition on the input for enhanced reconstruction. We train a gating module that takes photometric measurements as input and selects one out of a number of specialized decoders for reflectance reconstruction, essentially trading generality for quality. A common pre-trained latent-transform module is also appended to each decoder, to offset the burden of the increased number of decoders. In addition, the illumination conditions during acquisition can be jointly optimized. The effectiveness of our framework is validated on a wide variety of challenging near-planar samples with a lightstage. Compared with the state-of-the-art technique, our quality is improved with the same number of input images, and our input image number can be reduced to about 1/3 for equal-quality results. We further generalize the framework to enhance a state-of-the-art technique on non-planar reflectance scanning.

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

Emotion Recognition in Conversation (ERC) plays a significant part in Human-Computer Interaction (HCI) systems since it can provide empathetic services. Multimodal ERC can mitigate the drawbacks of uni-modal approaches. Recently, Graph Neural Networks (GNNs) have been widely used in a variety of fields due to their superior performance in relation modeling. In multimodal ERC, GNNs are capable of extracting both long-distance contextual information and inter-modal interactive information. Unfortunately, since existing methods such as MMGCN directly fuse multiple modalities, redundant information may be generated and diverse information may be lost. In this work, we present a directed Graph based Cross-modal Feature Complementation (GraphCFC) module that can efficiently model contextual and interactive information. GraphCFC alleviates the problem of heterogeneity gap in multimodal fusion by utilizing multiple subspace extractors and Pair-wise Cross-modal Complementary (PairCC) strategy. We extract various types of edges from the constructed graph for encoding, thus enabling GNNs to extract crucial contextual and interactive information more accurately when performing message passing. Furthermore, we design a GNN structure called GAT-MLP, which can provide a new unified network framework for multimodal learning. The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.

A Practical Unified Network for Localization and Recognition of Arbitrary-Oriented Container Code and Type

A Practical Unified Network for Localization and Recognition of Arbitrary-Oriented Container Code and Type

All in One: A Unified Network for Hyperspectral Image Fusion

All in One: A Unified Network for Hyperspectral Image Fusion

Vacancy-induced phonon localization in boron arsenide using a unified neural network interatomic potential

Vacancy-induced phonon localization in boron arsenide using a unified neural network interatomic potential

An integrated neural network method for landslide susceptibility assessment based on time-series InSAR deformation dynamic features

ABSTRACT We develop an integrated neural network landslide susceptibility assessment (LSA) method that integrates temporal dynamic features of interferometry synthetic aperture radar (InSAR) deformation data and the spatial features of landslide influencing factors. We construct a time-distributed convolutional neural network (TD-CNN) and bidirectional gated recurrent unit (Bi-GRU) to better understand the temporal dynamic features of InSAR cumulative deformation, and construct a multi-scale convolutional neural network (MSCNN) to determine the spatial features of landslide influencing factors, and construct a parallel unified deep learning network model to fuse these temporal and spatial features for LSA. Compared with the traditional MSCNN method, the accuracy of the proposed model is improved by 1.20%. The performance of the proposed model is preferable to MSCNN. The area under the receiver operating characteristic curve (AUC) of the testing set reaches 0.91. Our LSA results show that the proposed model clearly depicts areas with very high susceptibility landslides. Further, only 10.18% of the study area accurately covers 84.79% of historical landslide areas. Subjective consequences and objective indicators show that the proposed model that is integrated time-series InSAR deformation dynamic features can make full use of landslide characteristics and effectively improve the reliability of LSA.

A Y-shaped network based single-shot absolute phase recovery method for fringe projection profilometry

Fringe projection profilometry (FPP) is a widely used non-contact 3D measurement method. Though maturing in the last decade, it remains a significant challenge when facing the phase unwrapping of measured object surfaces in a single-shot measurement setting. With the rapid development of deep learning techniques, the adoption of a data-driven approach is gaining popularity in the field of optical metrology. This study proposes a new absolute phase recovery method based on the devised single-stage deep learning network. The aim is to ensure high-quality absolute phase recovery from a single-shot fringe projection measurement. Unlike most existing approaches, where the numerators and denominators of the wrapped phases and the fringe orders are predicted in various stages, the proposed method acquires the wrapped phases and the corresponding fringe orders within a single network, i.e. it can predict both wrapped phases and the corresponding fringe orders directly and simultaneously from the single fringe pattern projected in the single-shot mode based on a unified Y-shaped network. Experiments on benchmark datasets and models have demonstrated the effectiveness and efficiency of the technique, especially in terms of high-quality recovery of absolute phase information by using the lightweight single-stage network, and enabling the FPP-based phase 3D measurements in an online manner.

Unified ANN model for heat transfer factor (j) and friction factor (f) prediction in offset strip and wavy fin PFHEs

Accurate prediction of the dimensionless heat transfer (j factor) and friction factor (f factor) is crucial for optimizing plate-fin heat exchanger (PFHE) performance. However, current models, including existing ANN models, have difficulty predicting these factors for complex fin geometries such as offset strip fins and wavy fins. To address this challenge, this research proposes a novel, unified artificial neural network (ANN) model, specifically a multilayer perceptron (MLP), capable of accurately predicting both factors across various fin types, including offset strip fins and wavy fins. The ANN was trained and validated using a comprehensive dataset that encompasses a wide range of fin configurations and Reynolds numbers, from 300 to 10,000, covering both laminar and turbulent flow conditions. Advanced learning techniques, like Bayesian regularization (BR), were employed to enhance the model's performance. The proposed model surpasses existing models, achieving a predictive accuracy with a deviation of just 1–2% from actual values, compared to the typical ±20 % seen in previous models. This significant improvement implies the new ANN model could enhance PFHE performance, energy efficiency, and cost-effectiveness across diverse applications. Introducing a unified ANN model, the research underscores machine learning's potential in thermal engineering and paves the way for future advancements.

Advanced continuous threat detection in public networks of university laboratories based on deep learning

The rapid development of computer technology in China has led many universities to establish a common network within their schools to meet the different needs of middle school students. There may be some security risks in the common network of university laboratories, so this article will study the public network of laboratories and propose corresponding response policies and management strategies. This article will use deep learning techniques to extract relevant data and model some data. This technology mainly addresses the issue of non-standard user behavior, proposing some uneven training data and corresponding training methods, which can then build a unified laboratory public network to detect potential threats to the network, and extract and organize these abnormal data. This chapter can effectively improve the security of the entire network system and timely detect some threats in the network platform

Hybrid-Order Anomaly Detection on Attributed Networks

Anomaly detection on attributed networks has received an increasing amount of attention in recent years. Despite the success, most of the existing methods only focus on detecting the abnormal nodes while fail to detect the abnormal subgraphs. In this paper, we define a new problem of hybrid-order anomaly detection on attributed networks, which aims to detect both of the abnormal nodes and subgraphs. To this end, a new deep learning model called Hybrid-Order Graph Attention Network (HO-GAT) is developed, which is able to simultaneously detect the abnormal nodes and motif instances in an attributed network. In order to model the mutual influence between nodes and motif instances, the learning procedures of the node representation and the motif instance representation are integrated into a unified graph attention network with a novel hybrid-order self-attention mechanism. After learning the node representation and the motif instance representation, two decoders are respectively designed to reconstruct the attribute information of the nodes and motif instances, and the hybrid-order topological structure among nodes and motif instances. And finally, the reconstruction errors are utilized as the abnormal score of nodes and motif instances respectively. Extensive experiments conducted on real-world datasets have confirmed the effectiveness of the HO-GAT method.

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities.

Accurate segmentation of brain tumors is of critical importance in clinical assessment and treatment planning, which requires multiple MR modalities providing complementary information. However, due to practical limits, one or more modalities may be missing in real scenarios. To tackle this problem, existing methods need to train multiple networks or a unified but fixed network for various possible missing modality cases, which leads to high computational burdens or sub-optimal performance. In this paper, we propose a unified and adaptive multi-modal MR image synthesis method, and further apply it to tumor segmentation with missing modalities. Based on the decomposition of multi-modal MR images into common and modality-specific features, we design a shared hyper-encoder for embedding each available modality into the feature space, a graph-attention-based fusion block to aggregate the features of available modalities to the fused features, and a shared hyper-decoder for image reconstruction. We also propose an adversarial common feature constraint to enforce the fused features to be in a common space. As for missing modality segmentation, we first conduct the feature-level and image-level completion using our synthesis method and then segment the tumors based on the completed MR images together with the extracted common features. Moreover, we design a hypernet-based modulation module to adaptively utilize the real and synthetic modalities. Experimental results suggest that our method can not only synthesize reasonable multi-modal MR images, but also achieve state-of-the-art performance on brain tumor segmentation with missing modalities.

MiddleNet: A Unified, High-Performance NFV and Middlebox Framework With eBPF and DPDK

Traditional network resident functions (e.g., firewalls, network address translation) and middleboxes (caches, load balancers) have moved from purpose-built appliances to softwarebased components. However, L2/L3 network functions (NFs) are being implemented on Network Function Virtualization (NFV) platforms that extensively exploit kernel-bypass technology. They often use DPDK for zero-copy delivery and high performance. On the other hand, L4/L7 middleboxes, which have a greater emphasis on functionality, take advantage of a full-fledged kernelbased system. L2/L3 NFs and L4/L7 middleboxes continue to be handled by distinct platforms on different nodes. This paper proposes MiddleNet that develops a unified network resident function framework that supports L2/L3 NFs and L4/L7 middleboxes. MiddleNet supports function chains that are essential in both NFV and middlebox environments. MiddleNet uses the Data Plane Development Kit (DPDK) library for zero-copy packet delivery without interrupt-based processing, to enable the ’bumpin-the-wire’ L2/L3 processing performance required of NFV. To support L4/L7 middlebox functionality, MiddleNet utilizes a consolidated, kernel-based protocol stack for processing, avoiding a dedicated protocol stack for each function. MiddleNet fully exploits the event-driven capabilities of the extended Berkeley Packet Filter (eBPF) and seamlessly integrates it with shared memory for high-performance communication in L4/L7 middlebox function chains. The overheads for MiddleNet in L4/L7 are strictly load-proportional, without needing the dedicated CPU cores of DPDK-based approaches. MiddleNet supports flow-dependent packet processing by leveraging Single Root I/O Virtualization (SR-IOV) to dynamically select the packet processing needed (Layers 2 -7). Our experimental results show that MiddleNet achieves high performance in such a unified environment.

XGraphCDS: An explainable deep learning model for predicting drug sensitivity from gene pathways and chemical structures

Cancer is a highly complex disease characterized by genetic and phenotypic heterogeneity among individuals. In the era of precision medicine, understanding the genetic basis of these individual differences is crucial for developing new drugs and achieving personalized treatment. Despite the increasing abundance of cancer genomics data, predicting the relationship between cancer samples and drug sensitivity remains challenging. In this study, we developed an explainable graph neural network framework for predicting cancer drug sensitivity (XGraphCDS) based on comparative learning by integrating cancer gene expression information and drug chemical structure knowledge. Specifically, XGraphCDS consists of a unified heterogeneous network and multiple sub-networks, with molecular graphs representing drugs and gene enrichment scores representing cell lines. Experimental results showed that XGraphCDS consistently outperformed most state-of-the-art baselines (R2 = 0.863, AUC = 0.858). We also constructed a separate in vivo prediction model by using transfer learning strategies with in vitro experimental data and achieved good predictive power (AUC = 0.808). Simultaneously, our framework is interpretable, providing insights into resistance mechanisms alongside accurate predictions. The excellent performance of XGraphCDS highlights its immense potential in aiding the development of selective anti-tumor drugs and personalized dosing strategies in the field of precision medicine.

Polarimetric Synthetic Aperture Radar Image Classification Based on Double-Channel Convolution Network and Edge-Preserving Markov Random Field

Deep learning methods have gained significant popularity in the field of polarimetric synthetic aperture radar (PolSAR) image classification. These methods aim to extract high-level semantic features from the original PolSAR data to learn the polarimetric information. However, using only original data, these methods cannot learn multiple scattering features and complex structures for extremely heterogeneous terrain objects. In addition, deep learning methods always cause edge confusion due to the high-level features. To overcome these limitations, we propose a novel approach that combines a new double-channel convolutional neural network (CNN) with an edge-preserving Markov random field (MRF) model for PolSAR image classification, abbreviated to “DCCNN-MRF”. Firstly, a double-channel convolution network (DCCNN) is developed to combine complex matrix data and multiple scattering features. The DCCNN consists of two subnetworks: a Wishart-based complex matrix network and a multi-feature network. The Wishart-based complex matrix network focuses on learning the statistical characteristics and channel correlation, and the multi-feature network is designed to learn high-level semantic features well. Then, a unified network framework is designed to fuse two kinds of weighted features in order to enhance advantageous features and reduce redundant ones. Finally, an edge-preserving MRF model is integrated with the DCCNN network. In the MRF model, a sketch map-based edge energy function is designed by defining an adaptive weighted neighborhood for edge pixels. Experiments were conducted on four real PolSAR datasets with different sensors and bands. The experimental results demonstrate the effectiveness of the proposed DCCNN-MRF method.

A novel spatiotemporal 3D CNN framework with multi-task learning for efficient structural damage detection

In recent years, convolutional neural networks (CNNs) have demonstrated promising results in detecting structural damage. However, their architectures often overlook spatial and temporal effects simultaneously. This limitation can result in the loss of valuable information and an incapability to fully capture the complexity of the data, ultimately leading to reduced accuracy and suboptimal performance. This study proposes an intuitive three-dimensional CNN architecture that takes into account vibration history along with sensor spatial relations based on their relative positions. Furthermore, a multi-task learning (MTL) approach is suggested, which is a powerful approach for performing multiple tasks with a single network. The proposed 3D CNN method has been employed to detect single and double damage cases in an experimental steel frame through conventional classification alongside the transfer learning (TL). Moreover, MTL is used to detect single and double damage scenarios with a single unified network, which evaluates damage presence in separate tasks. The 3D CNN fulfilled state-of-the-art performance and 100% accuracy in detecting structural damage in almost all experiments. Additionally, the MTL model achieved promising results even in the presence of severe imbalanced classes of data. Furthermore, it was observed that the utilization of TL resulted in a notable reduction of computation time by 68% and the number of trainable parameters by 90% with the same level of accuracy in double-damage cases.