Information Embedding Research Articles

LAGSwin: Local attention guided Swin-transformer for thermal infrared sports object detection.

Compared with visible light images, thermal infrared images have poor resolution, low contrast, signal-to-noise ratio, blurred visual effects, and less information. Thermal infrared sports target detection methods relying on traditional convolutional networks capture the rich semantics in high-level features but blur the spatial details. The differences in physical information content and spatial distribution of high and low features are ignored, resulting in a mismatch between the region of interest and the target. To address these issues, we propose a local attention-guided Swin-transformer thermal infrared sports object detection method (LAGSwin) to encode sports objects' spatial transformation and orientation information. On the one hand, Swin-transformer guided by local attention is adopted to enrich the semantic knowledge of low-level features by embedding local focus from high-level features and generating high-quality anchors while increasing the embedding of contextual information. On the other hand, an active rotation filter is employed to encode orientation information, resulting in orientation-sensitive and invariant features to reduce the inconsistency between classification and localization regression. A bidirectional criss-cross fusion strategy is adopted in the feature fusion stage to enable better interaction and embedding features of different resolutions. At last, the evaluation and verification of multiple open-source sports target datasets prove that the proposed LAGSwin detection framework has good robustness and generalization ability.

A Self-Defense Copyright Protection Scheme for NFT Image Art Based on Information Embedding

Non-convertible tokens (NFTs) have become a fundamental part of the metaverse ecosystem due to its uniqueness and immutability. However, existing copyright protection schemes of NFT image art relied on the NFTs itself minted by third-party platforms. A minted NFT image art only tracks and verifies the entire transaction process, but the legitimacy of the source and ownership of its mapped digital image art cannot be determined. The original author or authorized publisher lack an active defense mechanism to prove ownership of the digital image art mapped by the unauthorized NFT. Therefore, we propose a self-defense copyright protection scheme for NFT image art based on information embedding in this paper, called SDCP-IE. The original author or authorized publisher can embed the copyright information into the published digital image art without damaging its visual effect in advance. Different from the existing information embedding works, the proposed SDCP-IE can generally enhance the invisibility of copyright information with different embedding capacity. Furthermore, considering the scenario of copyright information being discovered or even destroyed by unauthorized parties, the designed SDCP-IE can efficiently generate enhanced digital image art to improve the security performance of embedded image, thus resisting the detection of multiple known and unknown detection models simultaneously. The experimental results have also shown that the PSNR values of enhanced embedded image are all over 57db on three datasets BOSSBase, BOWS2 and ALASKA#2. Moreover, compared with existing information embedding works, the enhanced embedded images generated by SDCP-IE reaches the best transferability performance on the advanced CNN-based detection models. When the target detector is the pre-trained SRNet at 0.4bpp, the test error rate of SDCP-IE at 0.4bpp on the evaluated detection model YeNet reaches 53.38%, which is 4.92%, 28.62% and 7.05% higher than that of the UTGAN, SPS-ENH and Xie-Model, respectively.

A Multiinterest and Social Interest-Field Framework for Financial Security

Online payment has become an influential method of transaction. While improving convenience, this method of payment also brings great financial risks. Prevalidating transactions can be effective in reducing the number of frauds. For this reason, recommendation algorithms have been introduced to measure the credibility of transactions by predicting users’ ratings of items. However, most algorithms deal with the relationships in social networks without distinction, mixing positive and negative information into the recommender system, which brings huge noise. And, they only generate a single interest representation for each user to measure the similarity between users and spread interest, ignoring the diversity of user interests. Moreover, they did not consider the propagation of different interests would be different. In this article, we propose a multiinterest and social interest-field framework (MISIF) for social recommendations in financial security, which introduces capsule networks into social recommendation and extends the traditional single-interest representation to user multiinterest embedding by dynamic routing (DR) and other methods to improve the expressiveness of user embedding. After that, we construct social interest fields to integrate social interests based on multiinterest embedding, which alleviates the noise in social networks and user data sparsity problems. Finally, we aggregate user multiinterest embedding and additional information through neural networks to obtain the final prediction scores. Experiments with three publicly available datasets show that our proposed MISIF framework outperforms the state-of-the-art social recommendation methods.

Increasing the Efficiency of Blind Decoding of the Steganographic Method with Code Control of Additional Information Embedding

Modern energy systems are often used as a medium for information transmission, including confidential information, which makes relevant the task of ensuring its security, which today is solved using not only cryptographic but also steganographic means that ensure concealment of the very fact of confidential information transmission. The steganographic method with code control of the additional information embedding possesses practical important qualities, but in its original form requires the presence of a container for successful information extraction, which is not always desirable in practice, while the known modification of this steganographic method allows blind decoding is characterized by an insufficient level of resistance to attacks against the embedded message. The purpose of this paper is to improve the efficiency of the steganographic method with code control of the addition-al information embedding and blind decoding. This purpose was achieved by identifying two factors that determine the occurrence of errors during blind decoding of additional information in the steganographic method with code control: variation of sub-blocks, and errors caused by attacks against the embedded message. We propose the theoretical and practical rationale for codewords that provide the best level of resistance to both factors. The most significant result of the paper is a reduction in the number of decoding errors of the steganographic method with code control of the additional information embedding and blind decoding by 12.01% compared to the result known in the literature through a reasonable choice of codewords used to embed the additional information.

Benchmarking antibody clustering methods using sequence, structural, and machine learning similarity measures for antibody discovery applications.

Antibodies are proteins produced by our immune system that have been harnessed as biotherapeutics. The discovery of antibody-based therapeutics relies on analyzing large volumes of diverse sequences coming from phage display or animal immunizations. Identification of suitable therapeutic candidates is achieved by grouping the sequences by their similarity and subsequent selection of a diverse set of antibodies for further tests. Such groupings are typically created using sequence-similarity measures alone. Maximizing diversity in selected candidates is crucial to reducing the number of tests of molecules with near-identical properties. With the advances in structural modeling and machine learning, antibodies can now be grouped across other diversity dimensions, such as predicted paratopes or three-dimensional structures. Here we benchmarked antibody grouping methods using clonotype, sequence, paratope prediction, structure prediction, and embedding information. The results were benchmarked on two tasks: binder detection and epitope mapping. We demonstrate that on binder detection no method appears to outperform the others, while on epitope mapping, clonotype, paratope, and embedding clusterings are top performers. Most importantly, all the methods propose orthogonal groupings, offering more diverse pools of candidates when using multiple methods than any single method alone. To facilitate exploring the diversity of antibodies using different methods, we have created an online tool-CLAP-available at (clap.naturalantibody.com) that allows users to group, contrast, and visualize antibodies using the different grouping methods.

Mono3DVG: 3D Visual Grounding in Monocular Images

We introduce a novel task of 3D visual grounding in monocular RGB images using language descriptions with both appearance and geometry information. Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D object targets with their corresponding geometric text descriptions, generated by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR, an end-to-end transformer-based network, which takes advantage of both the appearance and geometry information in text embeddings for multi-modal learning and 3D object localization. Depth predictor is designed to explicitly learn geometry features. The dual text-guided adapter is proposed to refine multiscale visual and geometry features of the referred object. Based on depth-text-visual stacking attention, the decoder fuses object-level geometric cues and visual appearance into a learnable query. Comprehensive benchmarks and some insightful analyses are provided for Mono3DVG. Extensive comparisons and ablation studies show that our method significantly outperforms all baselines. The dataset and code will be released.

Cross-Modal Match for Language Conditioned 3D Object Grounding

Language conditioned 3D object grounding aims to find the object within the 3D scene mentioned by natural language descriptions, which mainly depends on the matching between visual and natural language. Considerable improvement in grounding performance is achieved by improving the multimodal fusion mechanism or bridging the gap between detection and matching. However, several mismatches are ignored, i.e., mismatch in local visual representation and global sentence representation, and mismatch in visual space and corresponding label word space. In this paper, we propose crossmodal match for 3D grounding from mitigating these mismatches perspective. Specifically, to match local visual features with the global description sentence, we propose BEV (Bird’s-eye-view) based global information embedding module. It projects multiple object proposal features into the BEV and the relations of different objects are accessed by the visual transformer which can model both positions and features with long-range dependencies. To circumvent the mismatch in feature spaces of different modalities, we propose crossmodal consistency learning. It performs cross-modal consistency constraints to convert the visual feature space into the label word feature space resulting in easier matching. Besides, we introduce label distillation loss and global distillation loss to drive these matches learning in a distillation way. We evaluate our method in mainstream evaluation settings on three datasets, and the results demonstrate the effectiveness of the proposed method.

Disentangled Partial Label Learning

Partial label learning (PLL) induces a multi-class classifier from training examples each associated with a set of candidate labels, among which only one is valid. The formation of real-world data typically arises from heterogeneous entanglement of series latent explanatory factors, which are considered intrinsic properties for discriminating between different patterns. Though learning disentangled representation is expected to facilitate label disambiguation for partial-label (PL) examples, few existing works were dedicated to addressing this issue. In this paper, we make the first attempt towards disentangled PLL and propose a novel approach named TERIAL, which makes predictions according to derived disentangled representation of instances and label embeddings. The TERIAL approach formulates the PL examples as an undirected bipartite graph where instances are only connected with their candidate labels, and employs a tailored neighborhood routing mechanism to yield disentangled representation of nodes in the graph. Specifically, the proposed routing mechanism progressively infers the explanatory factors that contribute to the edge between adjacent nodes and augments the representation of the central node with factor-aware embedding information propagated from specific neighbors simultaneously via iteratively analyzing the promising subspace clusters formed by the node and its neighbors. The estimated labeling confidence matrix is also introduced to accommodate unreliable links owing to the inherent ambiguity of PLL. Moreover, we theoretically prove that the neighborhood routing mechanism will converge to the point estimate that maximizes the marginal likelihood of observed PL training examples. Comprehensive experiments over various datasets demonstrate that our approach outperforms the state-of-the-art counterparts.

Medical knowledge graph completion via fusion of entity description and type information

Medical Knowledge Graphs (MKGs) are vital in propelling big data technologies in healthcare and facilitating the realization of medical intelligence. However, large-scale MKGs often exhibit characteristics of data sparsity and missing facts. Following the latest advances, knowledge embedding addresses these problems by performing knowledge graph completion. Most knowledge embedding algorithms rely solely on triplet structural information, overlooking the rich information hidden within entity property sets, leading to bottlenecks in performance enhancement when dealing with the intricate relations of MKGs. Inspired by the semantic sensitivity and explicit type constraints unique to the medical domain, we propose BioBERT-based graph embedding model. This model represents an evolvable framework that integrates graph embedding, language embedding, and type information, thereby optimizing the utility of MKGs. Our study utilizes not only WordNet as a benchmark dataset but also incorporates MedicalKG to compare and corroborate the specificity of medical knowledge. Experimental results on these datasets indicate that the proposed fusion framework achieves state-of-art (SOTA) performance compared to other baselines. We believe that this incremental improvement provides promising insights for future medical knowledge graph completion endeavors.

DIEANet: an attention model for histopathological image grading of lung adenocarcinoma based on dimensional information embedding

Efficient and rapid auxiliary diagnosis of different grades of lung adenocarcinoma is conducive to helping doctors accelerate individualized diagnosis and treatment processes, thus improving patient prognosis. Currently, there is often a problem of large intra-class differences and small inter-class differences between pathological images of lung adenocarcinoma tissues under different grades. If attention mechanisms such as Coordinate Attention (CA) are directly used for lung adenocarcinoma grading tasks, it is prone to excessive compression of feature information and overlooking the issue of information dependency within the same dimension. Therefore, we propose a Dimension Information Embedding Attention Network (DIEANet) for the task of lung adenocarcinoma grading. Specifically, we combine different pooling methods to automatically select local regions of key growth patterns such as lung adenocarcinoma cells, enhancing the model's focus on local information. Additionally, we employ an interactive fusion approach to concentrate feature information within the same dimension and across dimensions, thereby improving model performance. Extensive experiments have shown that under the condition of maintaining equal computational expenses, the accuracy of DIEANet with ResNet34 as the backbone reaches 88.19%, with an AUC of 96.61%, MCC of 81.71%, and Kappa of 81.16%. Compared to seven other attention mechanisms, it achieves state-of-the-art objective metrics. Additionally, it aligns more closely with the visual attention of pathology experts under subjective visual assessment.

Hierarchical bottleneck for heterogeneous graph representation

Heterogeneous graphs (HGs) contain many nodes and their interaction relationships, which can model complex systems and provide rich semantic and structural information for task execution. Among these, HG representation stands as the fundamental and pivotal component. Existing HG representation methods primarily employ graph neural networks to acquire the semantics of nodes along various meta-paths and fuse them to represent the nodes. The most prevalent HG representation methods encompass two steps: semantic information extraction within meta-paths and semantic fusion between meta-paths. However, these methods overlooked the consideration of node heterogeneity within meta-paths and the simultaneous semantic correlation between meta-paths. Specifically, node heterogeneity within meta-paths signifies that the meta-path-based neighbors do not consistently contain information that positively influences the target node, and the semantic correlation between meta-paths indicates that different meta-path spaces are not entirely independent. Disregarding either of these issues leads to the propagation of irrelevant or redundant information and potential disruption of HG embedding. Consequently, in this study, we propose the HBHG, which is a hierarchical bottleneck for heterogeneous graph representation. HBHG primarily employs the information bottleneck (IB) as a guiding principle, constraining the propagation of irrelevant information within and between meta-paths while preserving relevant information. The central concept of the IB revolves around viewing model learning as the preservation of relevant information and compression of irrelevant information, accomplished by minimizing the dependency between input and hidden features through mutual information (MI) and maximizing the dependency between hidden features and ground-truth. Considering the complexity associated with MI estimation, this paper introduces a novel dependency index, namely the Hilbert-Schmidt independence criterion (HSIC), which offers ease of calculation. Specifically, HBHG comprises two primary components: a semantic bottleneck within meta-paths and a semantic bottleneck between meta-paths. The semantic bottleneck within meta-paths relies primarily on the HSIC-based limitations of dependencies at different layers of the graph neural network on various meta-paths, thereby maximizing the extraction of information relevant to the target node from neighboring nodes. The semantic bottleneck between meta-paths enables flexible extraction and fusion of semantic information based on downstream tasks, achieved by managing the trade-off of dependencies with HSIC between different meta-path semantic spaces. In summary, the proposed HBHG integrates hierarchical bottleneck constraints within and between meta-paths. This integration serves to maximize the aggregation of relevant information while effectively compressing irrelevant information, thereby enhancing the quality of heterogeneous graph embedding. The effectiveness of HBHG was validated through performance and ablation experiments conducted on multiple datasets.

MagView++: Data Exfiltration via CPU Magnetic Signals Under Video Decoding

Air-gapped networks achieve security by using physical isolation to keep the computers and network from the Internet. However, magnetic covert channels based on CPU utilization have been proposed to help secret data to exfiltrate from the Faraday-cage and the air gap. Despite the success of such covert channels, they suffer from the high risk of being detected by the transmitter computer and the challenge of installing malware into such a computer. In this paper, we propose <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> , where sensitive information is embedded in other data such as video and can be transmitted over the internal network. When any computer uses the data such as playing the video, the sensitive information will leak through the magnetic signals. The “separation” of information embedding and leaking, combined with the fact that the data can be exfiltrated from any computer in a distributed manner, overcomes these limitations. We demonstrate that CPU utilization for video decoding can be effectively controlled by changing the video frame type, reducing the quantization parameter, and changing the timestamp of the frame, without video quality degradation. We prototype <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> and achieve 8.9 bps throughput with 0.0057 BER when using a smartphone as the receiver, and 59 bps throughput with 0.0025 BER when using a dedicated devices with high sampling rate as the receiver. Experiments under various environments are conducted to show the robustness of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> . Limitations and possible countermeasures are also discussed.

Prediction of linear B-cell epitopes based on protein sequence features and BERT embeddings

Linear B-cell epitopes (BCEs) play a key role in the development of peptide vaccines and immunodiagnostic reagents. Therefore, the accurate identification of linear BCEs is of great importance in the prevention of infectious diseases and the diagnosis of related diseases. The experimental methods used to identify BCEs are both expensive and time-consuming and they do not meet the demand for identification of large-scale protein sequence data. As a result, there is a need to develop an efficient and accurate computational method to rapidly identify linear BCE sequences. In this work, we developed the new linear BCE prediction method LBCE-BERT. This method is based on peptide chain sequence information and natural language model BERT embedding information, using an XGBoost classifier. The models were trained on three benchmark datasets. The model was training on three benchmark datasets for hyperparameter selection and was subsequently evaluated on several test datasets. The result indicate that our proposed method outperforms others in terms of AUROC and accuracy. The LBCE-BERT model is publicly available at: https://github.com/Lfang111/LBCE-BERT.

Single Channel Based Interference-Free and Self-Powered Human-Machine Interactive Interface Using Eigenfrequency-Dominant Mechanism.

The recent development of wearable devices is revolutionizing the way of human-machine interaction (HMI). Nowadays, an interactive interface that carries more embedded information is desired to fulfill the increasing demand in era of Internet of Things. However, present approach normally relies on sensor arrays for memory expansion, which inevitably brings the concern of wiring complexity, signal differentiation, power consumption, and miniaturization. Herein, a one-channel based self-powered HMI interface, which uses the eigenfrequency of magnetized micropillar (MMP) as identification mechanism, is reported. When manually vibrated, the inherent recovery of the MMP causes a damped oscillation that generates current signals because of Faraday's Law of induction. The time-to-frequency conversion explores the MMP-related eigenfrequency, which provides a specific solution to allocate diverse commands in an interference-free behavior even with one electric channel. A cylindrical cantilever model is built to regulate the MMP eigenfrequencies via precisely designing the dimensional parameters and material properties. It is shown that using one device and two electrodes, high-capacity HMI interface can be realized when the magnetic micropillars (MMPs) with different eigenfrequencies have been integrated. This study provides the reference value to design the future HMI system especially for situations that require a more intuitive and intelligent communication experience with high-memory demand.

A reversible natural language watermarking for sensitive information protection

Existing methods have evolved from using synonym substitution to incorporating arbitrary word substitution to achieve reversible natural language watermarking. However, a notable limitation is that they are prone to overlook the sensitivity of information associated with the original words, with a tendency to prefer non-sensitive words for substitution. As a result, a potential risk of sensitive information leakage contained in the original text is posed. Furthermore, while aiming for reversibility, the overall performance of the watermarking method may be inadvertently compromised. In response to the above problems, this paper puts forward a novel reversible natural language watermarking method that combines a Keyword Substitution scheme and a Prediction Error Expansion algorithm (KSPEE) to protect sensitive information, verify content integrity, protect copyright, and so on. Specifically, KSPEE leverages a keyword extraction algorithm to identify important content containing sensitive information in the original text, thereby determining the potential positions for watermark information embedding. Subsequently, a masked language model is utilized to predict appropriate substitution words based on the surrounding semantic information of the embedding position. In addition, the prediction error expansion algorithm is employed to select appropriate words for substituting the original keywords, ensuring the successful embedding of watermark information while maintaining the recoverability of the original keywords. By identifying keywords and substituting them, a suitable method of protecting the original sensitive information is provided. Extensive experiments demonstrate that, under the promise of semantic distortion and lossless restoration of the original content, the proposed method KSPEE achieves outstanding watermarked text quality. A higher watermark embedding rate is achieved and strong security is shown by KSPEE. More importantly, KSPEE effectively prevents the leakage of sensitive information.

Texture-aware and color-consistent learning for underwater image enhancement

Texture and color are pivotal factors for evaluating the quality of underwater image enhancement. However, current methods for enhancing underwater images still exhibit deficiencies in the restoration of texture and color. Diverging from previous approaches, we conduct heuristic modeling specifically targeting color and texture, culminating in the proposal of a texture-aware and color-consistent network (TACC-Net). Specifically, it first obtains preliminary enhanced images of more global features to facilitate feature decoupling. Then, we designed a color feature decoupling (CFD) sub-module to decouple the preliminary enhanced images into color-consistent color histograms to generate textured gray images. Finally, we designed a color space adjustment embedding (CSAE) sub-module, which is used to evaluate the similarity between features when embedding color information, and adjust the embedding of color information on textures. The experimental results indicate that our method enhances images and exhibits strong competitiveness in improving visual quality.

High-capacity medical information hiding scheme with P-tensor product secret image sharing for wireless body area networks

Given the escalating demand for information transmission in the resource-restricted wireless body area networks (WBANs), securing medical data and minimizing energy consumption are paramount. We present an innovative information hiding scheme that employs P-tensor product secret image sharing (PTP-SIS) and multiple information-hiders for WBANs. Considering the remarkable compression performance demonstrated by compressed sensing (CS), as well as its link to Shamir's (k, n) secret sharing method, using CS rather than traditional secret sharing techniques for image processing purposes could potentially help alleviate transmission burden of sensor nodes. However, utilizing CS to process high-dimensional signals poses a significant challenge due to the large dimensionality of the measurement matrix. The P-tensor product (PTP) is a tool that offers outstanding universality and can overcome the dimension limitations of traditional matrix multiplication, it is applied in the secret sharing stage. Moreover, applying the ideas of secret sharing leads to less stress on an individual device as well as lower communication spending, and multiple hiders have the ability to independently embed additional secrets into shadow images for efficient information backup. Theoretical analysis and experiments validate that the proposed scheme ensures information security and transmission efficiency while achieving superior image recovery performance and higher information embedding rates compared to other approaches.

A novel method for linguistic steganography by English translation using attention mechanism and probability distribution theory.

To enhance our ability to model long-range semantical dependencies, we introduce a novel approach for linguistic steganography through English translation. This method leverages attention mechanisms and probability distribution theory, known as NMT-stega (Neural Machine Translation-steganography). Specifically, to optimize translation accuracy and make full use of valuable source text information, we employ an attention-based NMT model as our translation technique. To address potential issues related to the degradation of text quality due to secret information embedding, we have devised a dynamic word pick policy based on probability variance. This policy adaptively constructs an alternative set and dynamically adjusts embedding capacity at each time step, guided by variance thresholds. Additionally, we have incorporated prior knowledge into the model by introducing a hyper-parameter that balances the contributions of the source and target text when predicting the embedded words. Extensive ablation experiments and comparative analyses, conducted on a large-scale Chinese-English corpus, validate the effectiveness of the proposed method across several critical aspects, including embedding rate, text quality, anti-steganography, and semantical distance. Notably, our numerical results demonstrate that the NMT-stega method outperforms alternative approaches in anti-steganography tasks, achieving the highest scores in two steganalysis models, NFZ-WDA (with score of 53) and LS-CNN (with score of 56.4). This underscores the superiority of NMT-stega in the anti-steganography attack task. Furthermore, even when generating longer sentences, with average lengths reaching 47 words, our method maintains strong semantical relationships, as evidenced by a semantic distance of 87.916. Moreover, we evaluate the proposed method using two metrics, Bilingual Evaluation Understudy and Perplexity, and achieve impressive scores of 42.103 and 23.592, respectively, highlighting its exceptional performance in the machine translation task.

ISTR: Mask-Embedding-Based Instance Segmentation Transformer.

Transformer-based instance-level recognition has attracted increasing research attention recently due to the superior performance. However, although attempts have been made to encode masks as embeddings into Transformer-based frameworks, how to combine mask embeddings and spatial information for a transformer-based approach is still not fully explored. In this paper, we revisit the design of mask-embedding-based pipelines and propose an Instance Segmentation TRansformer (ISTR) with Mask Meta-Embeddings (MME), leveraging the strengths of transformer models in encoding embedding information and incorporating spatial information from mask embeddings. ISTR incorporates a recurrent refining head that consists of a Dynamic Box Predictor (DBP), a Mask Information Generator (MIG), and a Mask Meta-Decoder (MMD). To improve the quality of mask embeddings, MME interprets the mask encoding-decoding processes as a mutual information maximization problem, which unifies the objective functions of different decoding schemes such as Principal Component Analysis (PCA) and Discrete Cosine Transform (DCT) with a meta-formulation. Under the meta-formulation, a learnable Spatial Mask Tuner (SMT) is further proposed, which fuses the spatial and embedding information produced from MIG and can significantly boost the segmentation performance. The resulting varieties, i.e., ISTR-PCA, ISTR-DCT, and ISTR-SMT, demonstrate the effectiveness and efficiency of incorporating mask embeddings with the query-based instance segmentation pipelines. On the COCO dataset, ISTR surpasses all predominant mask-embedding-based models by a large margin, and achieves competitive performance compared to concurrent state-of-the-art models. On the Cityscapes dataset, ISTR also outperforms several strong baselines. Our code has been made available at: https://github.com/hujiecpp/ISTR.

Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering.

Answering semantically complicated questions according to an image is challenging in a visual question answering (VQA) task. Although the image can be well represented by deep learning, the question is always simply embedded and cannot well indicate its meaning. Besides, the visual and textual features have a gap for different modalities, it is difficult to align and utilize the cross-modality information. In this article, we focus on these two problems and propose a graph matching attention (GMA) network. First, it not only builds graph for the image but also constructs graph for the question in terms of both syntactic and embedding information. Next, we explore the intramodality relationships by a dual-stage graph encoder and then present a bilateral cross-modality GMA to infer the relationships between the image and the question. The updated cross-modality features are then sent into the answer prediction module for final answer prediction. Experiments demonstrate that our network achieves the state-of-the-art performance on the GQA dataset and the VQA 2.0 dataset. The ablation studies verify the effectiveness of each module in our GMA network.