Information Embedding Research Articles

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Adaptive class token knowledge distillation for efficient vision transformer

The Vision Transformer (ViT) outperforms Convolutional Neural Networks (CNNs) but at the cost of significantly higher computational demands. Knowledge Distillation (KD) has shown promise in compressing complex networks by transferring knowledge from a large pre-trained model to a smaller one. However, current KD methods for ViT often rely on CNNs as teachers or neglect the importance of class token ([CLS]) information, resulting in ineffective distillation of ViT’s unique knowledge. In this paper, we propose Adaptive Class token Knowledge Distillation ([CLS]-KD), which fully exploits information from the class token and patches in ViT. For class embedding (CLS) distillation, the intermediate CLS of the student model is aligned with the corresponding CLS of the teacher model through a projector. Furthermore, we introduce CLS-patch attention map distillation, where an attention map between the CLS and patch embeddings is generated and matched at each layer. This empowers the student model to learn how to dynamically extract patch embedding information into the CLS under teacher guidance. Finally, we propose Adaptive Layer-wise Distillation (ALD) to mitigate the imbalance in distillation effects varying with the depth of layers. This method assigns greater weight to the losses in layers where the training discrepancies between the teacher and student models are larger during distillation. Through these strategies, [CLS]-KD consistently surpasses existing state-of-the-art methods on the ImageNet-1K dataset across various teacher-student configurations. Furthermore, the proposed method demonstrates its generalization capability through transfer learning experiments on the CIFAR-10, CIFAR-100, and CALTECH-256 datasets.

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

Dual-perspective multi-instance embedding learning with adaptive density distribution mining

Multi-instance learning (MIL) is a potent framework for solving weakly supervised problems, with bags containing multiple instances. Various embedding methods convert each bag into a vector in the new feature space based on a representative bag or instance, aiming to extract useful information from the bag. However, since the distribution of instances is related to labels, these methods rely solely on the overall perspective embedding without considering the different distribution characteristics, which will conflate the varied distributions of instances and thus lead to poor classification performance. In this paper, we propose the dual-perspective multi-instance embedding learning with adaptive density distribution mining (DPMIL) algorithm with three new techniques. First, the mutual instance selection technique consists of adaptive density distribution mining and discriminative evaluation. The distribution characteristics of negative instances and heterogeneous instance dissimilarity are effectively exploited to obtain instances with strong representativeness. Second, the embedding technique mines two crucial information of the bag simultaneously. Bags are converted into sequence invariant vectors according to the dual-perspective such that the distinguishability is maintained. Finally, the ensemble technique trains a batch of classifiers. The final model is obtained by weighted voting with the contribution of the dual-perspective embedding information. The experimental results demonstrate that the DPMIL algorithm has higher average accuracy than other compared algorithms, especially on web datasets.

A high-quality visual image encryption algorithm utilizing the conservative chaotic system and adaptive embedding

Regarding the issue of encrypted images in the channel attracting the attention of attackers and making them vulnerable to attacks. This paper proposes a high-quality visual image encryption algorithm utilizing the conservative chaotic system, two-dimensional compressive sensing, optimization local of binary patterns, and adaptive embedding method. Firstly, a conservative chaotic system with excellent performance and resistance to reconstruction attacks is proposed. Secondly, the pseudo-random sequences generated by the chaotic system dynamically generate measurement matrices, which are optimized before compressing the image. Then, during the encryption process, by utilizing the newly proposed composite DNA computing rules, chaotic sequences dynamically encode and compute image information, which enriches the coding criteria and improves the security of encryption algorithms. Finally, in the information embedding stage, the newly proposed OLBP algorithm can identify important textured and non-textured regions of the host image, prioritize embedding non-textured regions, and then embed textured regions, which can improve the amount of information embedding and reduce damage to the host image. Experimental simulation and analysis exhibit that the encryption algorithm has high security, strong robustness and high efficiency. Meanwhile the imperceptible analysis of the steganographic images is exceed 52 dB, so the algorithm presents a high-quality visual effect of imperceptibility.

UAT:Unsupervised object tracking based on graph attention information embedding

An excellent unsupervised tracker includes a powerful base tracker and an effective unsupervised tracking strategy. However, most base trackers lack internal feature representations for information embedding processes. Most unsupervised trackers are not robust enough in complex environments and lack an effective template update strategy. We propose an unsupervised object tracking based on graph attention information embedding (UAT) to solve the above problems. UAT combines graph attention mechanism with multi-scale features to construct a multi-scale graph attention module (MGA). MGA module dynamically and efficiently completes the information embedding between the template branch and the search area branch. The response map obtained by fusing the feature maps of the two branches is more informative about the location of the target. An attention based information reinforcement update module (RUM) improves the robustness of the tracker. RUM enhances the representation of the feature map in both the spatial dimension and the channel dimension. Template features are also updated indirectly through information transfer between the two branches. RUM suppresses background interference and improves network perception during tracking. Experiments on challenging benchmarks such as VOT2018, VOT2019, TrackingNet, OTB100, LaSOT and UAV123 demonstrate that the proposed UAT achieves state-of-the-art performance in unsupervised trackers.

Word vector embedding and self-supplementing network for Generalized Few-shot Semantic Segmentation

Under the condition of sufficient base class samples and a few novel class samples, Generalized Few-shot Semantic Segmentation (GFSS) classifies each pixel in the query image as base class, novel class, or background. A standard GFSS approach involves two training stages: base class learning and novel class updating. However, inter-class interference and information loss which contribute to the poor performance of GFSS, have not been synthetical considered. To address the problem, we propose an Embedded-Self-Supplementing Network (ESSNet), i.e., semantic word embedding and query set self-supplementing information to enhance segmentation accuracy. Specifically, the semantic word embedding module employs distance information between word vectors to assist the model in learning the distance between class prototypes. In order to transform the semantic word vector prototypes from the semantic space to the visual embedding space, we designed a triplet loss function to supervise the word vector embedding module, where the word vector prototype serves as an anchor and positive-negative samples are collected among the general features of the support image. To compensate for the information loss caused by using prototypes to represent classes, we propose a self-supplementing module to mine the information contained in the query image. Specifically, this module first makes a preliminary prediction on the query image, then selects high-confidence area to form pseudo labels, and finally uses pseudo labels to extract query prototypes to supplement the missing information. Extensive experiments on PASCAL-5i and COCO-20i show that ESSNet has superior performance and outperforms state-of-the-art methods in all settings.

A Hybrid System Based on Three Levels to Hide Information using JPEG Color Images

Nowadays, due to the circulation of information among people, information security has become an important requirement to protect information from hacking. Many techniques have been discovered to protect this information and maintain its confidentiality from hacking, the most prominent of which are cryptography and steganography techniques. In this paper, a hybrid system based on three levels to hide information using Joint Photographic Experts Group (JPEG) color images, where this system works in both processes sender and receiver. The sender process at the first level uses a mixing of the Advanced Encryption Standard (AES) and Rivest Shamir Adleman (RSA), while at the second level uses a Fuzzy Stream Algorithm (FSA) adds a higher complexity and eliminates the non-linearity of the encrypted information. The third level is to hide encrypted information using the Least Significant Bit (LSB) technique, while the receiver process, performs the reverse process of three levels. This system provides high information embedding capability and the inability to perceive hidden information, the system was evaluated based on criteria like Peak Signal Noise Ratio (PSNR), Mean Square Error (MSE), Structure Similarity (SSIM), and correlation, which were characterized by high and good rates. The study was compared with modern steganography techniques.

Trust enhanced POI recommendation with collaborative learning

With the rapid development and popularization of smart mobile devices, users tend to share their visited points-of-interest (POIs) on the network with attached location information, which forms a location-based social network (LBSN). LBSNs contain a wealth of valuable information, including the geographical coordinates of POIs and the social connections among users. Nowadays, lots of trust-enhanced approaches have fused the trust relationships of users together with other auxiliary information to provide more accurate recommendations. However, in the traditional trust-aware approaches, the embedding processes of the information on different graphs with different properties (e.g., user-user graph is an isomorphic graph, user-POI graph is a heterogeneous graph) are independent of each other and different embedding information is directly fused together without guidance, which limits their performance. More effective information fusion strategies are needed to improve the performance of trust-enhanced recommendation. To this end, we propose a Trust Enhanced POI recommendation approach with Collaborative Learning (TECL) to merge geographic information and social influence. Our proposed model integrates two modules, a GAT-based graph autoencoder as trust relationships embedding module and a multi-layer deep neural network as a user-POI graph learning module. By applying collaborative learning strategy, these two modules can interact with each other. The trust embedding module can guide the selection of user’s potential features, and in turn the user-POI graph learning module enhances the embedding process of trust relationships. Different information is fused through the two-way interaction of information, instead of travelling in one direction. Extensive experiments are conducted using real-world datasets, and results illustrate that our suggested approach outperforms state-of-the-art methods.

Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

An optimized watermarking scheme based on genetic algorithm and elliptic curve

Digital watermarking serves as a crucial tool for tracing copyright infringements and ensuring the authenticity and integrity of sensitive information. The fundamental concept involves embedding a watermark in the host information, ensuring its undetectability by unauthorized parties. The efficacy of a watermarking scheme mainly depends on achieving high levels of imperceptibility, robustness, and embedding capacity. These attributes are intricately linked to both the selection of the host information segment and the embedding factor. Existing schemes often (i) employ the entire host information for embedding, incurring computational expenses, and (ii) optimize the embedding factor without considering imperceptibility, robustness, and embedding capacity simultaneously, resulting in less secure watermarks. To address these limitations, we introduce a novel watermarking scheme leveraging elliptic curves (ECs) and genetic algorithms (GA). We model the choice of the embedding part by generating pseudo-random numbers over ECs, taking advantage of their proven sensitivity, security, and low computational complexity. Due to parallel search and adaptability to non-linear relationships of GA, the scheme employs genetic optimization with a multivariate objective function to establish a balance between imperceptibility, robustness, and embedding capacity for optimal watermarked generation. Rigorous analysis and comparisons demonstrate that our proposed scheme attains significantly higher imperceptibility, robustness, and embedding capacity compared to existing optimized schemes. Furthermore, our scheme exhibits a speed advantage, being up to 278 and 21 times faster than optimized and non-optimized schemes, respectively, thereby affirming its practical applicability.

ELSNC: A semi-supervised community detection method with integration of embedding-enhanced links and node content in attributed networks

In complex network analysis, detecting communities is becoming increasingly important. However, it is difficult to fuse multiple types of information to enhance the community-detection performance in real-world applications. Besides the nodes and the edges, a network also contains the structure of communities, its networking topological structure, and the network embeddings. Note that existing works on community detection have limited usage of all these information types in combination. In this work, we designed a novel unified model called embedding-enhanced link-based semi-supervised community detection with node content (ELSNC). ELSNC integrates the structure of the topology, the priori information, the network embeddings, and the node content. First, we employ two non-negative matrix factorization (NMF)–based stochastic models to characterize the node-community membership and the content-community membership (by performing similarity detection between a topic model and the NMF). Second, we introduce the nodes’ and networking embeddings’ topological similarity into the model as topological information. To model the topological similarity, we introduce a strong constraint (i.e., the priori information) and apply matrix completion to identify the community membership with the network embeddings’ representation ability. Finally, we present a semi-supervised community-detection method based on NMF that combines the network topology, content information, and the network embeddings. Our work’s innovation can be captured in two points: 1) As a type of semi-supervised community detection method, we extend the theory of semi-supervised methods on attributed networks and propose a unified model that integrates multiple information types. 2) The community membership obtained by the unified model simultaneously contains different information, including the topological, content, priori, and embedding information, which can more robustly be explored in the community structure in real-world scenarios. Furthermore, we performed a comprehensive evaluation of our proposed approach compared with state-of-the-art methods on both synthetic and real-world networks. The results show that our proposed method significantly outperformed the baseline methods.

FastUNet: Fast hierarchical multi-patch underwater enhancement network for industrial recirculating aquaculture

The industrialized aquaculture with recirculating water systems has gained increasing attention and rapid development in recent years. However, the existing underwater enhancement methods are time-consuming and severely hinder their online application in industrialized aquaculture due to the requirement for real-time processing. We found that their main limitations are the insufficient utilization of the hierarchical network approach and valuable prior information in underwater scenarios. To address this problem, we propose a novel architecture called FastUNet and introduce a visual enhancement benchmark dataset specifically designed for industrialized aquaculture with recirculating water systems. Surprisingly, by learning the fast multi-patch hierarchical module and employing prior constraints including color loss and homology loss, FastUNet achieves comparable or even better results than many deep learning methods. In terms of runtime, it is up to 30 times faster than current underwater enhancement methods. The processing time for images with a resolution of 5474 × 3653 is only 0.137s. The real application scenario of factory recirculating aquaculture is considered by FastUNet, and the embedding of prior information under the fast network framework is explored. In this paper, 14 state-of-the-art underwater enhancement methods are compared, and excellent results are obtained on 2 Full-reference datasets and 2 No-referenced datasets. The application test results of SIFT significant point detection, geometric rotation, edge detection, target segmentation and target detection were also presented, which strongly verified the feasibility of FastUNet in the field of factory recirculating aquaculture.

Anomalous behavior detection based on optimized graph embedding representation in social networks

Anomalous behaviors in social networks can lead to privacy leaks and the spread of false information. In this paper, we propose an anomalous behavior detection method based on optimized graph embedding representation. Specifically, the user behavior logs are first extracted into a social network user behavior temporal knowledge graph, based on which the graph embedding representation method is used to transform the network topology and temporal information in the user behavior knowledge graph into structural embedding vectors and temporal information embedding vectors, and the hybrid attention mechanism is used to merge the two types of vectors to obtain the final entity embedding to complete the prediction and complementation of the temporal knowledge graph of user behavior. We use graph neural networks, which use the temporal information of user behaviors as a time constraint and capture both user behavioral and semantic information. It converts the two parts of information into vectors for concatenation and linear transformation to obtain a comprehensive representation vector of the whole subgraph, as well as joint deep learning models to evaluate abnormal behavior. Finally, we perform experiments on the Yelp dataset to validate that our method achieves a 9.56% improvement in the F1-score.

Multi-stakeholder recommendation system through deep learning-based preference evaluation and aggregation model with multi-view information embedding

Learning the preferences of consumers, providers, and system stakeholders is a challenging problem in the Multi-Stakeholder Recommendation System (MSRS). Existing MSRS methods lack the ability to generate equitable recommendations and investigate implicit relationships between stakeholders and items. Hence, this study addresses this issue by proposing a multi-stakeholder preference learning-based recommendation model that exploits information from multiple views to evaluate stakeholders' preferences. The proposed model learns consumer preferences using users' ratings and reviews of an item, provider preferences with provider utility, and provider-item interaction. Furthermore, the proposed model learns the system-level preference of promoting long-tail items through the probabilistic evaluation of stakeholders' interest in popular and unpopular items. Finally, this study develops a multi-stakeholder, multi-view deep neural network model to aggregate stakeholders' preferences and deliver equitable recommendations. This work utilizes benchmark Movie Lens (ML) 25M, ML-100K, ML-1M, and TripAdvisor datasets to validate and compare the proposed model's performance with other baseline methods using standard evaluation metrics for each stakeholder. Examining the precision metrics, the proposed model attains the minimum enhancement of 7.91%, 18.24%, 10.72%, and 20.12% across the ML-25M, ML-100K, ML-1M, and TripAdvisor datasets. Further concerning the exposure, hit, and reach metrics, the model exhibits a substantial minimum improvement of 19.12%, 14.73%, 5.37%, and 28.46% over the ML-25M, ML-100K, ML-1M, and TripAdvisor datasets. Finally, the proposed model excels in promoting long-tail items and enhancing the cumulative utility gain of the stakeholders, surpassing the baseline methods.

Image style migration method based on transformer generative adversarial network combined with contrast learning strategy

Abstract Aiming at the problems of large loss of image patch embedding information and lack of stylized features in the generated image during transformer style migration, a style migration model combining contrast learning with transformer backbone generative adversarial network is established. Firstly, overlapping patch embedding is used to serialize the input images to get richer features; then the style and content contrast loss of contrast learning is used to mine the non-significant information between stylized images to reduce the style and content loss; finally, a multi-scale discriminator discriminates the true and false of stylized images to optimize the generator to get more stylistic features. The results on MS-COCO and WikiArt datasets show that the model improves the effectiveness and transformation efficacy of stylized images in style migration.

Relational-branchformer: Novel framework for audio-visual speech recognition

This study embraced the state-of-the-art Branchformer series architecture within the realm of automatic speech recognition, supplanting the widely utilized Conformer architecture. This substitution offers an innovative remedy tailored to audio-visual speech recognition tasks. Building upon the Branchformer architecture, enhancements were made, culminating in the proposal of the Relational-Branchformer (R-Branchformer). The convolutional attention relation module was innovatively incorporated to augment the connectivity between the local and global branches by meticulously considering their interrelations and interplays. Consequently, this module facilitates the mutual embedding of local and global contextual information, ultimately leading to a substantial enhancement in model performance. Our model was grounded in the utilization of the connectionist temporal classification (CTC) loss, wherein intermediate CTC losses were incorporated between blocks. Moreover, through the reference and enhancement of the gated interlayer collaboration module, which superseded the inter CTC module, the conditional independence assumption intrinsic to the CTC model was effectively relaxed. As a consequence, this augmentation markedly bolstered the overall performance of our model. Furthermore, the audio-visual output enhancement module was proposed, which adeptly assimilates information from both audio and visual modalities to enrich the representation of audio-visual information. Consequently, the R-Branchformer model achieved remarkable word error rates of 1.7% and 1.5% on the LRS2 and LRS3 test sets, respectively, exemplifying its state-of-the-art performance in audio-visual speech recognition tasks.

Copyright protection framework for federated learning models against collusion attacks

Federated learning (FL) models are constructed by multiple participants who provide their training datasets and collaborate in joint training. However, training and deployment processes have encountered various challenges in terms of intellectual property protection, such as illegal theft and data leakage. Existing FL protection frameworks focus on each client independently and verify the model ownership. When they are under collusion attacks (i.e., when multiple clients negotiate to steal together), they cannot accurately identify the clients that have stolen the model. To address this challenge, a novel watermarking protection scheme against collusion attacks for federated learning is proposed in this work. It employs anti-collusion coding to design unique watermark information for each client, which can effectively detect colluders. Furthermore, it utilizes a specific regularized loss function for watermark information embedding along with the incorporation of skip connections to embed the watermark information within each batch normalization layer. The experimental results demonstrated that embedding different watermark information into each client did not affect the accuracy of the original task. The accuracy was approximately 100% when identifying the colluders. The embedding and extraction times for the original task were only 1.53% and 0.29%, respectively. Further, it exhibited high robustness against various common attacks, including fine-tuning, shearing, and collusion attacks.

A Robust Dual Watermarking using Grey Wolf Optimization, Selective Encryption and Fast Flexible De-Noising Convolution Neural Network

Digital data interchange in IoT systems has flourished with the advancement of industrial internet technologies. Particularly, more and more digital images created by intelligent and industrial equipment are sent there are security concerns related to the website, server, and cloud. To accomplish this issue, in this article a secure watermarking approach is suggested in this research to effectively improve security, invisibility, and resilience at the same time. The adequate coefficient for information embedding is first determined using an assortment of transform domain techniques Discrete-Wavelet-Transform (DWT), Heisenberg- decomposition (HD), and Tensor-singular-value-decomposition (T-SVD). Using the grey wolf optimization (GWO) approach, we estimated the appropriate embedding factors to provide a reasonable compromise between robustness and invisibility. To enable the suggested approach to offer an additional level of security, a selective encryption technique is used on the watermark image. Moreover, FFDNet—a quick and adaptable de-noising convolutional-neural–network is working to increase the robustness-of-the suggested algorithm. The results demonstrate that the recommended watermarking method generates exceptional imperceptibility, resilience, and security against standard attacks. Additionally, the comparison demonstrates that the suggested algorithm performs better than alternative strategies. The following metrics were reached: 51.6966 dB, 0.9944, 0.9961, and 0.2849 for the peak-signal- to-noise ratio (PSNR), Structural-Similarity-Index (SSIM), number of changing pixels per second (NPCR), and unified-averaged-changed-intensity (UACI) average scores.

TC-Match: Fast Time-Constrained Continuous Subgraph Matching

Continuously monitoring structural patterns in streaming graphs is a critical task in many real-time graph-based applications. In this paper, we study the problem of time-constrained continuous subgraph matching (shorted as TCSM) over streaming graphs. Given a query graph Q with timing order constraint and a data graph stream G , TCSM aims to report all incremental matches of Q in G for each update of G , where a match should obey both structure constraint (i.e., isomorphism) and timing order constraint of Q. Although TCSM has a wide range of applications, such as cyber-attack detection and credit card fraud detection, we note that this problem has not been well addressed. The state-of-the-art bears the limitations of high index space cost and intermediate result maintenance cost. In this paper, we propose TC-Match, an effective approach to TCSM. First, we design a space and time cost-effective index CSS, which is essentially a k -partite graph structure where a node corresponds to an edge in G. By carefully creating links between nodes, we can encapsulate into CSS the partial embedding and timing order information between edges in G. We theoretically show that CSS has polynomial space and construction time complexities. Second, based on the property of CSS, we develop an efficient incremental matching algorithm with an effective node merging optimization. Extensive experiments show that TC-Match can achieve up to 3 orders of magnitude query performance improvement over the baseline methods, and meanwhile the memory consumption is reduced by 48.7%-86.7%.