Masking Model Research Articles

Enhancing Contextual Masking in Reversible Linguistic Steganography with Ensemble Methods

Various cybercrimes can be prevented by text authentication that is responsible for preserving digital identities and contents. Digital signatures come in handy as a way of authenticating texts, which is an extensively used method. One approach to this problem is linguistic steganography, which allows hiding the signature in other words within the text and thereby facilitating efficient data management. However, it should be noted that there is a danger that these kinds of changes may result in inappropriate decisions being taken by automated computing systems not to mention change their final outputs (unseen). As such, many people are becoming more concerned with the possibility of reversing steganography so that it becomes possible to eliminate any distortions made during the process. This paper uses Contextual masking instead of masking randomly with BERT model. The goal behind this research was developing a natural language text specific Reversible Steganographic System. Our model uses pre-trained BERT as a transformer based masked language model and reversibly embeds messages through predictive word substitution. To quantify predictive uncertainty, we introduce an adaptive steganographic technique using Bayesian deep learning. This experiment shows us how our proposed system balances imperceptibility with capacity while maintaining near semantics at all times. Also, we integrate ensemble methods instead of Monte Carlo to balance the imperceptibility.

MolPROP: Molecular Property prediction with multimodal language and graph fusion

Pretrained deep learning models self-supervised on large datasets of language, image, and graph representations are often fine-tuned on downstream tasks and have demonstrated remarkable adaptability in a variety of applications including chatbots, autonomous driving, and protein folding. Additional research aims to improve performance on downstream tasks by fusing high dimensional data representations across multiple modalities. In this work, we explore a novel fusion of a pretrained language model, ChemBERTa-2, with graph neural networks for the task of molecular property prediction. We benchmark the MolPROP suite of models on seven scaffold split MoleculeNet datasets and compare with state-of-the-art architectures. We find that (1) multimodal property prediction for small molecules can match or significantly outperform modern architectures on hydration free energy (FreeSolv), experimental water solubility (ESOL), lipophilicity (Lipo), and clinical toxicity tasks (ClinTox), (2) the MolPROP multimodal fusion is predominantly beneficial on regression tasks, (3) the ChemBERTa-2 masked language model pretraining task (MLM) outperformed multitask regression pretraining task (MTR) when fused with graph neural networks for multimodal property prediction, and (4) despite improvements from multimodal fusion on regression tasks MolPROP significantly underperforms on some classification tasks. MolPROP has been made available at https://github.com/merck/MolPROP.Scientific contributionThis work explores a novel multimodal fusion of learned language and graph representations of small molecules for the supervised task of molecular property prediction. The MolPROP suite of models demonstrates that language and graph fusion can significantly outperform modern architectures on several regression prediction tasks and also provides the opportunity to explore alternative fusion strategies on classification tasks for multimodal molecular property prediction.

Enhancing image quality prediction with self‐supervised visual masking

AbstractFull‐reference image quality metrics (FR‐IQMs) aim to measure the visual differences between a pair of reference and distorted images, with the goal of accurately predicting human judgments. However, existing FR‐IQMs, including traditional ones like PSNR and SSIM and even perceptual ones such as HDR‐VDP, LPIPS, and DISTS, still fall short in capturing the complexities and nuances of human perception. In this work, rather than devising a novel IQM model, we seek to improve upon the perceptual quality of existing FR‐IQM methods. We achieve this by considering visual masking, an important characteristic of the human visual system that changes its sensitivity to distortions as a function of local image content. Specifically, for a given FR‐IQM metric, we propose to predict a visual masking model that modulates reference and distorted images in a way that penalizes the visual errors based on their visibility. Since the ground truth visual masks are difficult to obtain, we demonstrate how they can be derived in a self‐supervised manner solely based on mean opinion scores (MOS) collected from an FR‐IQM dataset. Our approach results in enhanced FR‐IQM metrics that are more in line with human prediction both visually and quantitatively.

Completing and Balancing Database Excerpted Chemical Reactions with a Hybrid Mechanistic-Machine Learning Approach.

Computer-aided synthesis planning (CASP) development of reaction routes requires an understanding of complete reaction structures. However, most reactions in the current databases are missing reaction coparticipants. Although reaction prediction and atom mapping tools can predict major reaction participants and trace atom rearrangements in reactions, they fail to identify the missing molecules to complete reactions. This is because these approaches are data-driven models trained on the current reaction databases, which comprise incomplete reactions. In this work, a workflow was developed to tackle the reaction completion challenge. This includes a heuristic-based method to identify balanced reactions from reaction databases and complete some imbalanced reactions by adding candidate molecules. A machine learning masked language model (MLM) was trained to learn from simplified molecular input line entry system (SMILES) sentences of these completed reactions. The model predicted missing molecules for the incomplete reactions, a workflow analogous to predicting missing words in sentences. The model is promising for the prediction of small- and middle-sized missing molecules in incomplete reaction records. The workflow combining both the heuristic and machine learning methods completed more than half of the entire reaction space.

Multi-Modal Latent Diffusion.

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence-quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign.

Masked Self-Attention Dynamic Imputation Model for Physiological Indicators

Accurate imputation of missing and outlier values in intraoperative physiological indicators can assist doctors in swiftly taking measures to prevent risks during surgery, which in turn helps improve postoperative prognosis for patients. Traditional data imputation methods have focused too much on the current data itself, neglecting the contextual information provided by surrounding data points. Addressing this issue, the imputation process for intraoperative physiological indicator data obtained through monitoring has been enhanced with a model that employs a self-attention mechanism focusing on the data context for dynamic weighting during imputation. Experimental results indicate that the proposed model achieves a MAE of 2.06%, a RMSE of 7.08%, and a MRE of 2.8%, outperforming other comparative models.

Stitching Segments and Sentences towards Generalization in Video-Text Pre-training

Video-language pre-training models have recently achieved remarkable results on various multi-modal downstream tasks. However, most of these models rely on contrastive learning or masking modeling to align global features across modalities, neglecting the local associations between video frames and text tokens. This limits the model’s ability to perform fine-grained matching and generalization, especially for tasks that selecting segments in long videos based on query texts. To address this issue, we propose a novel stitching and matching pre-text task for video-language pre-training that encourages fine-grained interactions between modalities. Our task involves stitching video frames or sentences into longer sequences and predicting the positions of cross-model queries in the stitched sequences. The individual frame and sentence representations are thus aligned via the stitching and matching strategy, encouraging the fine-grained interactions between videos and texts. in the stitched sequences for the cross-modal query. We conduct extensive experiments on various benchmarks covering text-to-video retrieval, video question answering, video captioning, and moment retrieval. Our results demonstrate that the proposed method significantly improves the generalization capacity of the video-text pre-training models.

Hydrophobicity classification of polymeric insulators using a masked autoencoder model in vision transformer

The loss of hydrophobicity on the polymeric insulator’s surface leads to a continuous water channel formation. This water channel formation can initiate dry band arcing and subsequent flashover. Therefore, accurately identifying the hydrophobicity class is crucial as it provides the surface ageing information of the polymeric insulators. This study proposed a novel approach utilizing a masked autoencoder-based vision transformer image classifier to classify the hydrophobic condition of polymeric insulators accurately. A comprehensive series of tests were conducted on deep learning image classifier models to assess robustness. The experimental findings demonstrated that the vision transformer model exhibited a significant recognition accuracy of 99.69%, surpassing the performance of the CNN, pre-trained CNN, and ViT models.

Dual-channel neural network for instance segmentation of synapse

Detection and segmentation of neural synapses in electron microscopy images are the committed steps for analyzing neural ultrastructure. To date, manual annotation of the structure in synapses has been the primary method, which is time-consuming and restricts the throughput of data acquisition. Recent studies have utilized a series of deformations based on a segmentation model for the detection and segmentation of transmission electron microscope images. However, the analysis of synaptic segmentation and statistics still lacks sufficient automation and high-throughput. Therefore, we developed a dual-channel neural network instance segmentation model with weighted top-down and multi-scale bottom-up schemes, which aid in accurately detecting and segmenting synaptic vesicles and their active zones within presynaptic membranes in complex environments. In addition, we proposed a masked self-supervised pre-training model based on the latest convolutional architecture to improve performance in downstream segmentation tasks. By comparing our model to other state-of-the-art methods, we determined its viability and accuracy. The applicability of our model is thoroughly demonstrated by distinct application scenarios for neurobiological research. These findings indicate that the dual-channel neural network could facilitate the analysis of synaptic structures for the advancement of biomedical research and electron microscope reconstruction techniques.

CD-MAE: Contrastive Dual-Masked Autoencoder Pre-Training Model for PCB CT Image Element Segmentation

Element detection is an important step in the process of the non-destructive testing of printed circuit boards (PCB) based on computed tomography (CT). Compared with the traditional manual detection method, the image semantic segmentation method based on deep learning greatly improves efficiency and accuracy. However, semantic segmentation models often require a large amount of data for supervised training to generalize better model performance. Unlike natural images, the PCB CT image annotation task is more time-consuming and laborious than the semantic segmentation task. In order to reduce the cost of labeling and improve the ability of the model to utilize unlabeled data, unsupervised pre-training is a very reasonable and necessary choice. The masked image reconstruction model represented by a masked autoencoder is pre-trained on the unlabeled data, learning a strong feature representation ability by recovering the masked image, and shows a good generalization ability in various downstream tasks. In the PCB CT image element segmentation task, considering the characteristics of the image, it is necessary to use a model with strong feature robustness in the pre-training stage to realize the representation learning on a large number of unlabeled PCB CT images. Based on the above purposes, we proposed a contrastive dual-masked autoencoder (CD-MAE) pre-training model, which can learn more robust feature representation on unlabeled PCB CT images. Our experiments show that the CD-MAE outperforms the baseline model and fully supervised models in the PCB CT element segmentation task.

Convolutions are competitive with transformers for protein sequence pretraining

Pretrained protein sequence language models have been shown to improve the performance of many prediction tasks and are now routinely integrated into bioinformatics tools. However, these models largely rely on the transformer architecture, which scales quadratically with sequence length in both run-time and memory. Therefore, state-of-the-art models have limitations on sequence length. To address this limitation, we investigated whether convolutional neural network (CNN) architectures, which scale linearly with sequence length, could be as effective as transformers in protein language models. With masked language model pretraining, CNNs are competitive with, and occasionally superior to, transformers across downstream applications while maintaining strong performance on sequences longer than those allowed in the current state-of-the-art transformer models. Our work suggests that computational efficiency can be improved without sacrificing performance, simply by using a CNN architecture instead of a transformer, and emphasizes the importance of disentangling pretraining task and model architecture. A record of this paper's transparent peer review process is included in the supplemental information.

AOA based Masked Region-CNN model for Detection of Parking Space in IoT Environment

Uneven illumination has a significant impact on vision-based automatic parking systems, making it impossible to conduct a correct assessment of parking places in the presence of complicated picture data. In to address this issue, this work provides a deep learning-based system for visual recognition of parking spaces and picture processing. Artificial intelligence (AI) approaches can be used to identify a less expensive and easier-to-implement solution to the parking spot identification challenge, especially since the discipline of deep learning is reshaping the world. Using deep learning techniques, this study offers a dynamic, straightforward, and cost-effective algorithm for the detection of parking spots. In order to determine which parking spots are available and which are occupied, this method employs a Masked Region Based Convolutional Neural Network (MR-CNN) and the intersection over union approach. Cars in the training dataset were spaced more apart than those actually seen, which increased the accuracy of the identification between cars and parking spots. The AOA mechanism enhances the model's ability to focus on relevant regions within an image, improving accuracy in detecting parking spaces. This leads to precise identification of parking slots, reducing false positives and negatives. The sequence and quantity of parking spots, as well as the capacity to predict empty spots, were tested in a case study and found to be accurate. In the experimental results as the AOA based MR-CNN model stretched the accuracy as 98.50 and the recall value as 40.59 then the precision as 96.34 F1-measure as 57.95 correspondingly.

A reversible natural language watermarking for sensitive information protection

Existing methods have evolved from using synonym substitution to incorporating arbitrary word substitution to achieve reversible natural language watermarking. However, a notable limitation is that they are prone to overlook the sensitivity of information associated with the original words, with a tendency to prefer non-sensitive words for substitution. As a result, a potential risk of sensitive information leakage contained in the original text is posed. Furthermore, while aiming for reversibility, the overall performance of the watermarking method may be inadvertently compromised. In response to the above problems, this paper puts forward a novel reversible natural language watermarking method that combines a Keyword Substitution scheme and a Prediction Error Expansion algorithm (KSPEE) to protect sensitive information, verify content integrity, protect copyright, and so on. Specifically, KSPEE leverages a keyword extraction algorithm to identify important content containing sensitive information in the original text, thereby determining the potential positions for watermark information embedding. Subsequently, a masked language model is utilized to predict appropriate substitution words based on the surrounding semantic information of the embedding position. In addition, the prediction error expansion algorithm is employed to select appropriate words for substituting the original keywords, ensuring the successful embedding of watermark information while maintaining the recoverability of the original keywords. By identifying keywords and substituting them, a suitable method of protecting the original sensitive information is provided. Extensive experiments demonstrate that, under the promise of semantic distortion and lossless restoration of the original content, the proposed method KSPEE achieves outstanding watermarked text quality. A higher watermark embedding rate is achieved and strong security is shown by KSPEE. More importantly, KSPEE effectively prevents the leakage of sensitive information.

Recursive signal denoising method for predictive maintenance of equipment by using deep learning based temporal masking

In the data driven predictive maintenance, high quality data is the premise of high accuracy diagnosis and prediction. In the industrial practice, reducing the noise is of great significance to ensure data quality. This paper proposes a recursive denoising method for manufacturing equipment signals in the data driven predictive maintenance. First, in signal decomposition method, equipment mixed signal is decomposed by temporal masking with dilated convolution neural network to generate a noise mask, which realizes signal decomposition of using recursive operation of temporal masking model. Second, in signal components recognition method, signal component features similarities are calculated, which act on the parameter regulation of signal recognition meta-learning model. The experimental results demonstrated that the proposed method effectively solves the noise reduction problem of the equipment signal. Further engineering tests of a chemical winding machine vibration signal decomposition and recognition show that the proposed method has strong adaptive performance for noise reduction.

Automatic Segmentation of Membranous Glottal Gap Area with U-Net-Based Architecture.

While videostroboscopy is recognized as the most popular approach for investigating vocal fold function, evaluating the numerical values, such as the membranous glottal gap area, remains too time consuming for clinical applications. We used a total of 2507 videostroboscopy images from 137 patients and developed five U-Net-based deep-learning image segmentation models for automatic masking of the membranous glottal gap area. To further validate the models, we used another 410 images from 41 different patients. During development, all five models exhibited acceptable and similar metrics. While the VGG19 U-Net had a long inference time of 1654 ms, the other four models had more practical inference times, ranging from 16 to 138 ms. During further validation, Efficient U-Net demonstrated the highest intersection over union of 0.8455, the highest Dice coefficient of 0.9163, and the lowest Hausdorff distance of 1.5626. The normalized membranous glottal gap area index was also calculated and validated. Efficient U-Net and VGG19 U-Net exhibited the lowest mean squared errors (3.5476 and 3.3842) and the lowest mean absolute errors (1.8835 and 1.8396). Automatic segmentation of the membranous glottal gap area can be achieved through U-net-based architecture. Considering the segmentation quality and speed, Efficient U-Net is a reasonable choice for this task, while the other four models remain valuable competitors. The models' masked area enables possible calculation of the normalized membranous glottal gap area and analysis of the glottal area waveform, revealing promising clinical applications for this model. NA Laryngoscope, 134:2835-2843, 2024.

Dance movement error correction based on decision tree algorithm

Abstract In an environment where action recognition technology is widely used, dance action recognition technology has also received extensive attention. Due to the complexity of dance movements, there will be a large number of errors in dance movement recognition. Therefore, this paper uses the decision tree algorithm to optimize dance movement recognition. In this paper, the general framework of dance movement recognition is first constructed, and the human body features are extracted and calculated by using the Laban motion analysis method for the changes in the joints of dance movements. Then, use the decision tree algorithm to classify and identify the dance movements, and use it to optimize the self-obscuring motion information repair algorithm to solve the problem of dance movement occlusion and error, and complete the construction of the dance movement recognition model based on CNN network. Empirical analysis shows that the model method's Top-1 and Top-5 accuracy are both 84.7% and 89.32%, respectively. The average probability of incorrect recognition, error recognition, and correct recognition of the optimized dance action recognition in this paper is 8.84%, 3.78 and 87.38 respectively. The average recognition correctness is improved by 12.02%, which indicates that the effect of recognition of the optimized model of dance action masking and error correction in this paper is excellent. It has achieved a better construction in the field of dance action error correction.

Masked Duration Model for Utterance Duration-Controllable Text-to-Speech

Masked Duration Model for Utterance Duration-Controllable Text-to-Speech

Masked co-attention model for audio-visual event localization

Masked co-attention model for audio-visual event localization

Δ BP-Net: Monitoring "Changes" in Blood Pressure Using PPG with Self-contrastive Masking.

Estimating blood pressure (BP) values from physiological signals (e.g., photoplethysmogram (PPG)) using deep learning models has recently received increased attention, yet challenges remain in terms of models' generalizability. Here, we propose taking a new approach by framing the problem as tracking the "changes" in BP over an interval, rather than directly estimating its value. Indeed, continuous monitoring of acute changes in BP holds promising implications for clinical applications (e.g., hypertensive emergencies). As a solution, we first present a self-contrastive masking (SCM) model, designed to perform pair-wise temporal comparisons within the input signal. We then leverage the proposed SCM model to introduce ΔBPNet, a model trained to detect elevations/drops greater than a given threshold in the systolic blood pressure (SBP) over an interval, from PPG. Using data from PulseDB, 1) we evaluate the performance of ΔBP-Net on previously unseen subjects, 2) we test ΔBP-Net's ability to generalize across domains by training and testing on different datasets, and 3) we compare the performance of ΔBP-Net with existing PPG-based BP-estimation models in detecting over-threshold SBP changes. Formulating the problem as a binary classification task (i.e., over-threshold SBP elevation/ drop or not), ΔBP-Net achieves 75.97%/73.19% accuracy on data from subjects unseen during training. Additionally, the proposed ΔBP-Net outperforms ΔSBP estimations derived from existing PPG-based BP-estimation methods. Overall, by shifting the focus from estimating the value of SBP to detecting overthreshold "changes" in SBP, this work introduces a new potential for using PPG in clinical BP monitoring, and takes a step forward in addressing the challenges related to the generalizability of PPG-based BP-estimation models.

Efficient Text Style Transfer through Robust Masked Language Model and Iterative Inference

Efficient Text Style Transfer through Robust Masked Language Model and Iterative Inference