Alignment Network Research Articles

Networked scatter plot bidirectional coupling detection and its application in the diagnosis of depression

Identifying the bidirectional coupling of bivariate time series is a basic problem in the analysis of complex systems in contemporary science. As an intuitive statistical graphical analysis tool, scatter plot allows direct analysis of the correlation between two bivariate time series. However, the scatter plot itself have the problems of not being able to conduct bidirectional coupling analysis and data points overlap. In this work, by introducing inner composition alignment (IOTA) and ordinal pattern transition network (OPTN) into scatter plot, the limitations of scatter plot are alleviated and a new approach for measuring bivariate time series bidirectional coupling is proposed. To facilitate the integration of the three methods, the IOTA scatter plot proposed and the new scatter plot been partitioned and coded. The coded partitions are defined as network nodes and a directed weighted network is constructed based on the temporal adjacency of the nodes. For the constructed OPTN, the mean output degree that commonly used in OPTN is used as a quantitative indicator of coupling strength. To test the characteristics and performance of the new method, it is used to analyse the unidirectional coupled Lorentz model and to compare it with existing methods. The results showed the new method had the advantages of easy parameter setting, efficiency, and suitable for short time series. Finally, the real EEG dataset from depressed patients at rest are analysed and some useful and interesting results are obtained by treating the relevant brain areas as routers.

FreeEnricher: Enriching Face Landmarks without Additional Cost

Recent years have witnessed significant growth of face alignment. Though dense facial landmark is highly demanded in various scenarios, e.g., cosmetic medicine and facial beautification, most works only consider sparse face alignment. To address this problem, we present a framework that can enrich landmark density by existing sparse landmark datasets, e.g., 300W with 68 points and WFLW with 98 points. Firstly, we observe that the local patches along each semantic contour are highly similar in appearance. Then, we propose a weakly-supervised idea of learning the refinement ability on original sparse landmarks and adapting this ability to enriched dense landmarks. Meanwhile, several operators are devised and organized together to implement the idea. Finally, the trained model is applied as a plug-and-play module to the existing face alignment networks. To evaluate our method, we manually label the dense landmarks on 300W testset. Our method yields state-of-the-art accuracy not only in newly-constructed dense 300W testset but also in the original sparse 300W and WFLW testsets without additional cost.

Joint Multimodal Entity-Relation Extraction Based on Edge-Enhanced Graph Alignment Network and Word-Pair Relation Tagging

Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task. However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them. This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction (JMERE) task . Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships. To address the above challenges, we propose an edge-enhanced graph alignment network and a word-pair relation tagging (EEGA) for the JMERE task. Specifically, we first design a word-pair relation tagging to exploit the bidirectional interaction between MNER and MRE and avoid error propagation. Then, we propose an edge-enhanced graph alignment network to enhance the JMERE task by aligning nodes and edges in the cross-graph. Compared with previous methods, the proposed method can leverage the edge information to auxiliary alignment between objects and entities and find the correlations between entity-entity relationships and object-object relationships. Experiments are conducted to show the effectiveness of our model.

MT-FANet: A Morphology and Topology-Based Feature Alignment Network for SAR Ship Rotation Detection

In recent years, ship target detection in synthetic aperture radar (SAR) images has significantly progressed due to the rapid development of deep learning (DL). However, since only the spatial feature information of ship targets is utilized, the current DL-based SAR ship detection approaches cannot achieve a satisfactory performance, especially in the case of multiscale, rotations, or complex backgrounds. To address these issues, in this paper, a novel deep-learning network for SAR ship rotation detection, called a morphology and topology-based feature alignment network, is proposed which can better exploit the morphological features and inherent topological structure information. This network consists of the following three main steps: First, deformable convolution is introduced to improve the representational ability for irregularly shaped ship targets, and subsequently, a morphology and topology feature pyramid network is developed to extract inherent topological structure information. Second, based on the aforementioned features, a rotation alignment feature head is devised for fine-grained processing as well as aligning and distinguishing the features; to enable regression prediction of rotated bounding boxes; and to adopt a parameter-sharing mechanism to improve detection efficiency. Therefore, utilizing morphological and inherent topological structural information enables a superior detection performance to be achieved. Finally, we evaluate the effectiveness of the proposed method using the rotated ship detection dataset in SAR images (RSDD-SAR). Our method outperforms other DL-based algorithms with fewer parameters. The overall average precision is 90.84% and recall is 92.21%. In inshore and offshore scenarios, our method performs well for the detection of multi-scale and rotation-varying ship targets, with its average precision reaching 66.87% and 95.72%, respectively.

Optimising for Interpretability: Convolutional Dynamic Alignment Networks.

We introduce a new family of neural network models called Convolutional Dynamic Alignment Networks (CoDA Nets), which are performant classifiers with a high degree of inherent interpretability. Their core building blocks are Dynamic Alignment Units (DAUs), which are optimised to transform their inputs with dynamically computed weight vectors that align with task-relevant patterns. As a result, CoDA Nets model the classification prediction through a series of input-dependent linear transformations, allowing for linear decomposition of the output into individual input contributions. Given the alignment of the DAUs, the resulting contribution maps align with discriminative input patterns. These model-inherent decompositions are of high visual quality and outperform existing attribution methods under quantitative metrics. Further, CoDA Nets constitute performant classifiers, achieving on par results to ResNet and VGG models on e.g. CIFAR-10 and TinyImagenet. Lastly, CoDA Nets can be combined with conventional neural network models to yield powerful classifiers that more easily scale to complex datasets such as Imagenet whilst exhibiting an increased interpretable depth, i.e., the output can be explained well in terms of contributions from intermediate layers within the network.

In Your Eyes: Modality Disentangling for Personality Analysis in Short Video

With the dramatic growth of various short video platforms, users are more likely to share their social stream online and make their social connections stronger. To better understand their preferences, personality analysis has attracted more attention. Unlike single modal data such as text or images, which is hard to comprehensively uncover one's personal traits, personality analysis on short video is verified to be much more accurate but also more challenging because of the huge gap between incompatible data modalities. We have noticed that the key problem is how to disentangle the complexity from multimodal data to find their consistency and uniqueness. In this article, we propose a novel video analysis framework for personality detection with visual, acoustic, and textual neural networks. Specifically, to enhance our model's sensitivity to personality detection, we first propose three deep learning channels to learn modal features. The framework can not only extract each modal feature but also learn time-varying pattern via a temporal alignment network. To identify the consistency and uniqueness across multiple modalities, we creatively propose to maximize the similarity of common information learned by a shared neural network across multiple modalities and extend the distance of exclusive information learned by private networks of different modalities. Extensive experiments on the real-world dataset demonstrate that our model can outperform existing baselines.

Towards Cross-Lingual Multi-Modal Misinformation Detection for E-Commerce Management

The misinformation detection systems are increasingly important in E-commerce management, which detect misinformation on the commodity display page. Misinformation in E-commerce is usually presented as a mismatch between multi-modal information, the detection systems need to find the misinformation across the multi-model information. Furthermore, with the development of E-commerce globalization, we hope to deploy the system in the international E-commerce platform, which may face the difficulty caused by multi-lingual data. To this end, we propose a Cross-lingual Multi-modal Misinformation Detection (CMMD) framework for E-commerce management. The CMMD framework includes a word alignment network that embeds information from different languages into the same feature space and a multimodal fusion structure that fuses text representations and image representations through two self-supervised tasks. With the cooperation of these two modules, the CMMD model could extract rich cross-lingual multi-modal features to achieve accurate misinformation detection. We conduct experiments on the public multi-modal dataset and further apply the proposed CMMD to the real-world E-commerce international platform. The experimental results show that the proposed framework CMMD achieves better performance on the public dataset than some benchmarks and gets satisfactory results in real-world E-commerce management.

Robot learning from human demonstrations with inconsistent contexts

Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Multi-modal policy fusion for end-to-end autonomous driving

Multi-modal learning has made impressive progress in autonomous driving by leveraging information from multiple sensors. Existing feature fusion methods make decisions by integrating perceptions from different sensors. However, autonomous driving systems could be risky since the fused feature are unreliable when one of the sensors fails. Moreover, these methods require either sophisticated geometric designs to align features or complex neural networks to effectively fuse features, significantly increasing the training cost. In this paper, we propose PolicyFuser, a policy fusion method for end-to-end autonomous driving to address these issues. PolicyFuser retains an independent decision for each sensor, and no feature alignment or complex neural networks are required. To focus on the best policy, we use reinforcement learning to select the action with the highest Q-value as the primary decision, and the remaining actions as the secondary decisions. Then the secondary decisions are used to fine-tune the primary decision through a primary and secondary policy fusion (PSF) module. To bridge the gap between the decisions from different sensors and improve the stability of policy fusion, we use a conditional variational autoencoder (CVAE) to generate pseudo-expert decisions. We demonstrate the effectiveness of our method in CARLA, and our method achieves the highest driving scores and handles sensor failures with excellence.

Video super-resolution via nonlocal deformable alignment and frame recursive progressive fusion network

Video super-resolution (VSR) is a process of high-resolution reconstruction of low-resolution video. To address the problems of the previous VSR methods with poor temporal consistency and unsatisfactory SR results, we proposed a nonlocal deformable alignment and frame recursive progressive fusion (RPF) network combining sliding window and recursive methods, which uses nonlocal operations to align sequential frame features and later applies recursion to temporally model the hidden information and alignment features of the previous moment, thus improving temporal consistency. The RPF unit is used to fully fuse the hidden information with the currently aligned features, acquiring more supporting information to be obtained from adjacent frames, resulting in better SR results. The results were evaluated on the three public VSR datasets of Vid4, udm10, and Vimeo-90K, and the experimental results show that the proposed method can achieve state-of-the-art performance on VSR task.

Printing‐Induced Alignment Network Design of Polymer Matrix for Stretchable Perovskite Solar Cells with Over 20% Efficiency

AbstractPolymer matrix is felicitously applied into the active layer and transporting layer of perovskite solar cells (PSCs) to enable a stretchable function. However, the chaotic deposition of polymer chains is the main cause for the inferior photoelectric performance. When the stretchable PSCs are in a working state, the stress cannot be removed effectively due to the random polymer chain deposition. The stress accumulation will cause irreversible damage to the stretchable PSCs. Herein, the structural bionics and patterned‐meniscus coating technology are combined to print the polymer chain‐oriented deposition in the stretchable PSCs. Based on this approach, the conducting polymer electrode is printed with both significant mechanical stability and conductivity. More importantly, the oriented polyurethane with self‐healing property can enhance the crystal quality of perovskite films and repair perovskite cracks caused by stress destruction. Thus, the corresponding stretchable PSCs achieve a stabilized power conversion efficiency (PCE) of 20.04% (1.0 cm2) and 16.47% (9 cm2) with minor efficiency discrepancy. Notably, the stretchable PSCs can maintain 86% of the primitive PCE after 1000 cycles of bending with a stretch ratio of 30%. This directional growth of polymer chain strategy provides guidance for printing prominent‐performance stretchable PSCs.

Adversarial-based latent space alignment network for left atrial appendage segmentation in transesophageal echocardiography images

Left atrial appendage (LAA) is a leading cause of atrial fibrillation and thrombosis in cardiovascular disease. Clinicians can rely on LAA occlusion (LAAO) to effectively prevent and treat ischaemic strokes attributed to the LAA. The correct selection of the LAAO is one of the most critical stages in the successful surgical process, which relies on the quantification of the anatomical structure of the LAA for successful intervention in LAAO. In this paper, we propose an adversarial-based latent space alignment framework for LAA segmentation in transesophageal echocardiography (TEE) images by introducing prior knowledge from the label. The proposed method consists of an LAA segmentation network, a label reconstruction network, and a latent space alignment loss. To be specific, we first employ ConvNeXt as the backbone of the segmentation and reconstruction network to enhance the feature extraction capability of the encoder. The label reconstruction network then encodes the prior shape features from the LAA labels to the latent space. The latent space alignment loss consists of the adversarial-based alignment and the contrast learning losses. It can motivate the segmentation network to learn the prior shape features of the labels, thus improving the accuracy of LAA edge segmentation. The proposed method was evaluated on a TEE dataset including 1,783 images and the experimental results showed that the proposed method outperformed other state-of-the-art LAA segmentation methods with Dice coefficient, AUC, ACC, G-mean, and Kappa of 0.831, 0.917, 0.989, 0.911, and 0.825, respectively.

A Facial Landmark Detection Method Based on Deep Knowledge Transfer.

Facial landmark detection is a crucial preprocessing step in many applications that process facial images. Deep-learning-based methods have become mainstream and achieved outstanding performance in facial landmark detection. However, accurate models typically have a large number of parameters, which results in high computational complexity and execution time. A simple but effective facial landmark detection model that achieves a balance between accuracy and speed is crucial. To achieve this, a lightweight, efficient, and effective model is proposed called the efficient face alignment network (EfficientFAN) in this article. EfficientFAN adopts the encoder-decoder structure, with a simple backbone EfficientNet-B0 as the encoder and three upsampling layers and convolutional layers as the decoder. Moreover, deep dark knowledge is extracted through feature-aligned distillation and patch similarity distillation on the teacher network, which contains pixel distribution information in the feature space and multiscale structural information in the affinity space of feature maps. The accuracy of EfficientFAN is further improved after it absorbs dark knowledge. Extensive experimental results on public datasets, including 300 Faces in the Wild (300W), Wider Facial Landmarks in the Wild (WFLW), and Caltech Occluded Faces in the Wild (COFW), demonstrate the superiority of EfficientFAN over state-of-the-art methods.

Cross-Modal Generation and Pair Correlation Alignment Hashing

Cross-modal hashing is an effective cross-modal retrieval approach because of its low storage and high efficiency. However, most existing methods mainly utilize pre-trained networks to extract modality-specific features, while ignore the position information and lack information interaction between different modalities. To address those problems, in this paper, we propose a novel approach, named cross-modal generation and pair correlation alignment hashing (CMGCAH), which introduces transformer to exploit position information and utilizes cross-modal generative adversarial networks (GAN) to boost cross-modal information interaction. Concretely, a cross-modal interaction network based on conditional generative adversarial network and pair correlation alignment networks are proposed to generate cross-modal common representations. On the other hand, a transformer-based feature extraction network (TFEN) is designed to exploit position information, which can be propagated to text modality and enforce the common representation to be semantically consistent. Experiments are performed on widely used datasets with text-image modalities, and results show that the proposed method achieved competitive performance compared with many existing methods.

HAAN: Learning a Hierarchical Adaptive Alignment Network for Image-Text Retrieval

Image-text retrieval aims to search related results of one modality by querying another modality. As a fundamental and key problem in cross-modal retrieval, image-text retrieval is still a challenging problem owing to the complementary and imbalanced relationship between different modalities (i.e., Image and Text) and different granularities (i.e., Global-level and Local-level). However, existing works have not fully considered how to effectively mine and fuse the complementarities between images and texts at different granularities. Therefore, in this paper, we propose a hierarchical adaptive alignment network, whose contributions are as follows: (1) We propose a multi-level alignment network, which simultaneously mines global-level and local-level data, thereby enhancing the semantic association between images and texts. (2) We propose an adaptive weighted loss to flexibly optimize the image-text similarity with two stages in a unified framework. (3) We conduct extensive experiments on three public benchmark datasets (Corel 5K, Pascal Sentence, and Wiki) and compare them with eleven state-of-the-art methods. The experimental results thoroughly verify the effectiveness of our proposed method.

Reverse-Net: Few-Shot Learning with Reverse Teaching for Deformable Medical Image Registration

Multimodal medical image registration has an important role in monitoring tumor growth, radiotherapy, and disease diagnosis. Deep-learning-based methods have made great progress in the past few years. However, its success depends on large training datasets, and the performance of the model decreases due to overfitting and poor generalization when only limited data are available. In this paper, a multimodal medical image registration framework based on few-shot learning is proposed, named reverse-net, which can improve the accuracy and generalization ability of the network by using a few segmentation labels. Firstly, we used the border enhancement network to enhance the ROI (region of interest) boundaries of T1 images to provide high-quality data for the subsequent pixel alignment stage. Secondly, through a coarse registration network, the T1 image and T2 image were roughly aligned. Then, the pixel alignment network generated more smooth deformation fields. Finally, the reverse teaching network used the warped T1 segmentation labels and warped images generated by the deformation field to teach the border enhancement network more structural knowledge. The performance and generalizability of our model have been evaluated on publicly available brain datasets including the MRBrainS13DataNii-Pro, SRI24, CIT168, and OASIS datasets. Compared with VoxelMorph, the reverse-net obtained performance improvements of 4.36% in DSC on the publicly available MRBrainS13DataNii-Pro dataset. On the unseen dataset OASIS, the reverse-net obtained performance improvements of 4.2% in DSC compared with VoxelMorph, which shows that the model can obtain better generalizability. The promising performance on dataset CIT168 indicates that the model is practicable.

AnANet: Association and Alignment Network for Modeling Implicit Relevance in Cross-Modal Correlation Classification

With the explosive increase of multimodal data, cross-modal correlation classification has become an important research topic and is in great demand in many cross-modal applications. A variety of classification schemes and predictive models have been built based on the existing cross-modal correlation categorization. However, these classification schemes typically follow the prior assumption that the paired cross-modal samples are strictly related, and thus pay great attention to the fine-grained relevant types of cross-modal correlation, ignoring the high volume of implicitly relevant data which are often wrongly classified into irrelevant types. Even more, previous predictive models fall short of reflecting the essence of cross-modal correlation according to their definitions, especially in the modeling of network structure. Thus in this paper, by comprehensively investigating the current image-text correlation classification research, we redefine a new classification scheme for cross-modal correlation based on the implicit and explicit relevance. To predict the types of image-text correlation based on our proposed definition, we further devise the Association and Alignment Network (namely AnANet) to model the implicit and explicit relevance, which captures both the implicit association of global discrepancy and commonality between image and text and explicit alignment of cross-modal local relevance. Experimental studies on our constructed new image-text correlation dataset verify the effectiveness of our proposed model.

FRAN-X: An improved diagnostic transfer learning approach with application to ball bearings fault diagnosis

Data-driven diagnostic methods are attractive from an industrial and practical perspective due to their limited amount of required prior knowledge about the process or component under monitoring. However, these methods usually require a large amount of healthy and possibly faulty labeled data. Often, gathering and manually labeling a vast dataset is not feasible in real scenarios. Transfer learning has emerged as an answer to the labeling problem, exploiting the idea that the diagnostic knowledge could be reused across multiple different, but related, machines and operating conditions. In this work, we introduce several improvements to the Feature Representation and Alignment Network (FRAN) architecture described in (Chen et al., 2020) devised with the diagnostic transfer learning purpose. Our approach, named FRAN-X, presents improved transfer and diagnostics performance between identical machines in different operating conditions, and it is computationally lighter than its original counterpart. The FRAN-X approach is evaluated on the CWRU-bearing dataset and on experimental data collected from a Computerized Numerical Control (CNC) workcenter machine.

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel Multi-Modal Cross-Domain Alignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Decoder-Side Cross Resolution Synthesis for Video Compression Enhancement

This paper proposes a decoder-side Cross Resolution Synthesis (CRS) module to pursue better compression efficiency beyond the latest Versatile Video Coding (VVC), where we encode intra frames at original high resolution (HR), compress inter frames at a lower resolution (LR), and then super-resolve decoded LR inter frames with the help from preceding HR intra and neighboring LR inter frames. For a LR inter frame, a motion alignment and aggregation network (MAN) is devised to produce temporally aggregated motion representation to best guarantee the temporal smoothness; Another texture compensation network (TCN) is utilized to generate texture representation from decoded HR intra frame for better augmenting spatial details; Finally, a similarity-driven fusion engine synthesizes motion and texture representations to upscale LR inter frames for the removal of compression and resolution re-sampling noises. We enhance the VVC using proposed CRS, showing averaged 8.76% and 11.93% Bjntegaard Delta Rate (BD-Rate) gains against the latest VVC anchor in Random Access (RA) and Low-delay P (LDP) settings respectively. In addition, experimental comparisons to the state-of-the-art super-resolution (SR) based VVC enhancement methods, and ablation studies are conducted to further report superior efficiency and generalization of the proposed algorithm. All materials will be made to public at https://njuvision.github.io/CRS for reproducible research.