Masked Images Research Articles

Mathematical Modelling for Neural Network-Based Textural Feature Extraction from Normalized Ocular Images for Improved Biometric Authentication

Biometric authentication systems have become increasingly prevalent in security and identity verification applications. Among various biometric traits, the ocular region, specifically the iris, offers a unique and stable source for individual identification. The intricate patterns within the iris are highly distinctive and remain consistent over a person's lifetime, making them ideal for biometric applications. In the field of biometric authentication, the ocular region has emerged as a robust and reliable source for identity verification due to its unique and stable features. This study focuses on enhancing the accuracy of biometric systems through neural network-based textural feature extraction from normalized ocular images. The proposed approach utilizes Mask Image Extraction (Hough circle) and an efficient U-Net model to achieve superior performance in biometric authentication. Initially, the ocular images are pre-processed to normalize lighting and orientation, ensuring consistency across the dataset. Mask Image Extraction, particularly using the Hough circle transform, is employed to isolate the region of interest (ROI) within the ocular images. This method accurately detects and segments the circular iris region, which is crucial for subsequent feature extraction. The core of the proposed system is the Efficient U-Net model, a variant of the standard U-Net architecture designed for enhanced efficiency and accuracy. The U-Net model is trained to extract intricate textural features from the segmented ocular regions. By leveraging its encoder-decoder structure, the U-Net captures both local and global features, resulting in a detailed and discriminative representation of the ocular texture. Experimental results demonstrate that the integration of Mask Image Extraction and the Efficient U-Net model significantly improves the accuracy of biometric authentication systems. The proposed method outperforms traditional approaches by providing more precise and reliable feature extraction, thereby enhancing the overall robustness and reliability of ocular-based biometric systems.

Correlation-Embedded Transformer Tracking: A Single-Branch Framework.

Developing robust and discriminative appearance models has been a long-standing research challenge in visual object tracking. In the prevalent Siamese-based paradigm, the features extracted by the Siamese-like networks are often insufficient to model the tracked targets and distractor objects, thereby hindering them from being robust and discriminative simultaneously. While most Siamese trackers focus on designing robust correlation operations, we propose a novel single-branch tracking framework inspired by the transformer. Unlike the Siamese-like feature extraction, our tracker deeply embeds cross-image feature correlation in multiple layers of the feature network. By extensively matching the features of the two images through multiple layers, it can suppress non-target features, resulting in target-aware feature extraction. The output features can be directly used to predict target locations without additional correlation steps. Thus, we reformulate the two-branch Siamese tracking as a conceptually simple, fully transformer-based Single-Branch Tracking pipeline, dubbed SBT. After conducting an in-depth analysis of the SBT baseline, we summarize many effective design principles and propose an improved tracker dubbed SuperSBT. SuperSBT adopts a hierarchical architecture with a local modeling layer to enhance shallow-level features. A unified relation modeling is proposed to remove complex handcrafted layer pattern designs. SuperSBT is further improved by masked image modeling pre-training, integrating temporal modeling, and equipping with dedicated prediction heads. Thus, SuperSBT outperforms the SBT baseline by 4.7%,3.0%, and 4.5% AUC scores in LaSOT, TrackingNet, and GOT-10K. Notably, SuperSBT greatly raises the speed of SBT from 37 FPS to 81 FPS. Extensive experiments show that our method achieves superior results on eight VOT benchmarks. Code is available at https://github.com/phiphiphi31/SBT.

Rethinking masked image modelling for medical image representation

Masked Image Modelling (MIM), a form of self-supervised learning, has garnered significant success in computer vision by improving image representations using unannotated data. Traditional MIMs typically employ a strategy of random sampling across the image. However, this random masking technique may not be ideally suited for medical imaging, which possesses distinct characteristics divergent from natural images. In medical imaging, particularly in pathology, disease-related features are often exceedingly sparse and localized, while the remaining regions appear normal and undifferentiated. Additionally, medical images frequently accompany reports, directly pinpointing pathological changes’ location. Inspired by this, we propose Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that employs radiological reports to guide the masking and restore the informative areas of images, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge-driven masking (KDM), and sentence-driven masking (SDM). KDM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify symptom clues mapped to MeSH words (e.g., cardiac, edema, vascular, pulmonary) and guide the mask generation. Recognizing that radiological reports often comprise several sentences detailing varied findings, SDM integrates sentence-level information to identify key regions for masking. MedIM reconstructs images informed by this masking from the KDM and SDM modules, promoting a comprehensive and enriched medical image representation. Our extensive experiments on seven downstream tasks covering multi-label/class image classification, pneumothorax segmentation, and medical image–report analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image–report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.

MIMTracking: Masked image modeling enhanced vision transformer for visual object tracking

Recently, one-stage trackers achieve state-of-the-art tracking results due to more sufficient integration of search and template representations. These methods usually adopt an encoder for synchronous feature generation and interaction. Despite their high performance, the one-stage approaches tend to feed the encoder full input representations that are highly redundant and dense during training. With a focus on tackling this problem, a novel algorithm termed MIMTracking is developed for visual target tracking. MIMTracking exploits an encoder and a decoder for masked image modeling during training. Randomly sampled discrete search embeddings and template embeddings serve as input of the encoder. The lightweight decoder takes full representations as input and progressively highlights the target region. This design alleviates input redundancy and reduces the computational cost of the training process, thereby allowing for more efficient learning of useful representations. The proposed MIMTracking achieves state-of-the-art tracking results on numerous tracking datasets, e.g., 51.5 % area under curve (AUC) on LaSOText, outperforming the previous top tracker OSTrack by 2 %. Especially, our large model MIMTracking-L further improves the AUC to 53.4 % on LaSOText.

Neural Correlates of the Attentional Bias Towards Subliminal Pornographic Cues in Individuals with Tendencies Toward Problematic Pornography Use: An ERP Study Using a Dot-Probe Task.

Attentional bias toward addiction-related stimuli has been implicated in the development and maintenance of addiction disorders. Several previous studies have reported an attentional bias toward pornographic cues in individuals with problematic pornography use (PPU). Since attentional bias can occur without conscious awareness, the purpose of this study was to use electroencephalography to examine whether individuals with a high tendency for PPU exhibit attentional bias at the level of the preconscious processing. Event-related potentials (ERPs) were recorded while male participants with high (n = 24) and low (n = 23) levels of subclinical PPU performed a masked version of the dot-probe task measuring attentional bias toward subliminally presented pornographic stimuli. Behavioral data revealed that participants from both groups with high and low tendencies for PPU reacted faster to probes replacing pornographic images than to probes replacing neutral images. ERPs revealed that individuals with a high tendency for PPU exhibited larger probe-locked P1 amplitudes following masked pornographic images (valid condition) compared with masked neutral images (invalid condition). Additionally, PPU symptom severity correlated positively with the P1 amplitude difference between valid and invalid conditions. These results highlight the automaticity of attentional capture by pornographic stimuli and support the hypothesis of an addiction-related attentional bias during preconscious processes. The implication of these findings for understanding the clinical phenomenon of out-of-control addictive behavior are discussed.

Opti-CAM: Optimizing saliency maps for interpretability

Methods based on class activation maps (CAM) provide a simple mechanism to interpret predictions of convolutional neural networks by using linear combinations of feature maps as saliency maps. By contrast, masking-based methods optimize a saliency map directly in the image space or learn it by training another network on additional data. In this work we introduce Opti-CAM, combining ideas from CAM-based and masking-based approaches. Our saliency map is a linear combination of feature maps, where weights are optimized per image such that the logit of the masked image for a given class is maximized. We also fix a fundamental flaw in two of the most common evaluation metrics of attribution methods. On several datasets, Opti-CAM largely outperforms other CAM-based approaches according to the most relevant classification metrics. We provide empirical evidence supporting that localization and classifier interpretability are not necessarily aligned.

Fusing Ground-Penetrating Radar Images for Improving Image Characteristics Fidelity

The analysis of ground-penetrating radar (GPR) data is of vital importance for detecting various subsurface features that might manifest as hyperbolic peaks, which are indicators of a buried object or grayscale variation in the case of contrast in the soil texture. This method focuses on identifying exaggerated patterns through a series of image-processing steps. Two GPR images are initially read and preprocessed by extracting channels, flipping, and resizing. Then, specific regions of interest (ROIs) are cropped, and the Fourier transform is further applied to turn them into the frequency domain. With the help of their frequency signatures, these patterns are extracted from the images, and binary masks are constructed to obtain features of interest. These masked images were reconstructed and merged to make hyperbolic features visible. Finally, Local Binary Pattern (LBP) analysis is used to emphasize these hyperbolic peaks, thereby facilitating their recognition across the whole image. The proposed approach improves the detection of performance subsurface features in GPR data; hence, it is an important tool for geophysical surveys and other related applications. The results prove the high performance of the proposed procedure in improving GPR image characteristics.

Demonstration of MaskSearch: Efficiently Querying Image Masks for Machine Learning Workflows

We demonstrate MaskSearch, a system designed to accelerate queries over databases of image masks generated by machine learning models. MaskSearch formalizes and accelerates a new category of queries for retrieving images and their corresponding masks based on mask properties, which support various applications, from identifying spurious correlations learned by models to exploring discrepancies between model saliency and human attention. This demonstration makes the following contributions: (1) the introduction of MaskSearch's graphical user interface (GUI), which enables interactive exploration of image databases through mask properties, (2) hands-on opportunities for users to explore MaskSearch's capabilities and constraints within machine learning workflows, and (3) an opportunity for conference attendees to understand how MaskSearch accelerates queries over image masks.

Self-Supervised Medical Image Segmentation Using Deep Reinforced Adaptive Masking.

Self-supervised learning aims to learn transferable representations from unlabeled data for downstream tasks. Inspired by masked language modeling in natural language processing, masked image modeling (MIM) has achieved certain success in the field of computer vision, but its effectiveness in medical images remains unsatisfactory. This is mainly due to the high redundancy and small discriminative regions in medical images compared to natural images. Therefore, this paper proposes an adaptive hard masking (AHM) approach based on deep reinforcement learning to expand the application of MIM in medical images. Unlike predefined random masks, AHM uses an asynchronous advantage actor-critic (A3C) model to predict reconstruction loss for each patch, enabling the model to learn where masking is valuable. By optimizing the non-differentiable sampling process using reinforcement learning, AHM enhances the understanding of key regions, thereby improving downstream task performance. Experimental results on two medical image datasets demonstrate that AHM outperforms state-of-the-art methods. Additional experiments under various settings validate the effectiveness of AHM in constructing masked images.

Self-supervised Domain Adaptation with Significance-Oriented Masking for Pelvic Organ Prolapse detection

Pelvic Organ Prolapse (POP) is a common disease in middle-aged and elderly women. The detection of POP is a challenging task, and using deep learning for detection has its practical significance. However, medical image detection tasks always face many problems, i.e., small sample size, data imbalance, and unobvious pathological characteristics. In this paper, we propose a new training framework, called self-supervised Domain Adaptation with Significance-Oriented Masking (DASOM), to address these problems and improve the performance of POP intelligent detection. DASOM includes a new pre-training process based on the masked image modeling task, and redesigns the masking strategy, bringing the local induction capability required for detection to the model. Meanwhile, we also adopt the data process method fitting the pelvic floor ultrasonic dataset to effectively solve the problem of data shortage and imbalance. Extensive experimental results and analysis confirm that the proposed method significantly improves the performance and reliability of POP detection.

EVA-02: A visual representation for neon genesis

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well as extensive pre-training from an open & accessible giant CLIP vision encoder, EVA-02 demonstrates superior performance compared to prior state-of-the-art approaches across various representative vision tasks, while utilizing significantly fewer parameters and compute budgets. Notably, using exclusively publicly accessible training data, EVA-02 with only 304 M parameters achieves a phenomenal 90.0 fine-tuning top-1 accuracy on ImageNet-1 K val set. Additionally, our EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1 K, outperforming the previous largest & best open-sourced CLIP with only ∼1/6 parameters and ∼ 1/6 image-text training data. We offer four EVA-02 variants in various model sizes, ranging from 6 M to 304 M parameters, all with impressive performance. To facilitate open access and open research, we release the complete suite of EVA-02 to the community at https://github.com/baaivision/EVA/tree/master/EVA-02.

Single-shot lensless masked imaging with enhanced self-calibrated phase retrieval.

Single-shot lensless imaging with a binary amplitude mask enables a low-cost and miniaturized configuration for wave field recovery. However, the mask only allows a part of the wave field to be captured, and thus the inverse decoding process becomes a highly ill-posed problem. Here we propose an enhanced self-calibrated phase retrieval (eSCPR) method to realize single-shot joint recovery of mask distribution and the sample's wavefront. In our method, a sparse regularized phase retrieval (SrPR) algorithm is designed to calibrate the mask distribution. Then, a denoising regularized phase retrieval (DrPR) algorithm is constructed to reconstruct the wavefront of the sample. Compared to conventional single-shot methods, our method shows robust and flexible image recovery. Experimental results of different samples are given to demonstrate the superiority of our method.

Light Reflections Detection And Correction For Robotic Volumetric PIV

Laser light reflection mitigation in Particle Image Velocimetry (PIV) is crucial for accurate flow field measurements. While numerous methods exist for planar PIV, fewer have been developed for volumetric PIV systems, and in particular for coaxial setups like robotic volumetric PIV. Light reflections in volumetric PIV experiments result in high-intensity regions that corrupt particle detection and analysis. This study presents a novel approach for treating light reflections in robotic volumetric PIV experiments. The proposed method uses image filtering and masking techniques in the wavenumber space to separate particle images from reflection regions. The process involves decomposing the image signal into low- and high-wavenumber components using the 2D discrete Fourier transform (DFT) to then use a high-pass filter to attenuate the intensity of the reflection regions. Finally, a step of automated adaptive masking is applied to remove residual reflection areas that the filtering is not able to eliminate. The proposed approach is tested on experimental data obtained from experiments performed using robotic volumetric PIV on two different geometries: a side-view mirror and a rotating two-blade propeller. Comparison between raw and pre-processed images, as well as particle tracking results, is presented. The results from this data comparison show successful removal of reflection-induced artifacts in instantaneous images by using the spatial Fourier filter automated masking approach. The developed image pre-processing strategy effectively removes unsteady light reflection regions in robotic volumetric PIV images, preventing the appearance of spurious particle tracks and improving the accuracy of flow field measurements. The spatial gaps introduced by the masking procedure can be easily filled in via measurements from multiple directions, which are promptly achieved via the robotic volumetric PIV approach.

Attention module incorporated transfer learning empowered deep learning-based models for classification of phenotypically similar tropical cattle breeds (Bos indicus).

Accurate breed identification in dairy cattle is essential for optimizing herd management and improving genetic standards. A smart method for correctly identifying phenotypically similar breeds can empower farmers to enhance herd productivity. A convolutional neural network (CNN) based model was developed for the identification of Sahiwal and Red Sindhi cows. To increase the classification accuracy, first, cows's pixels were segmented from the background using CNN model. Using this segmented image, a masked image was produced by retaining cows' pixels from the original image while eliminating the background. To improve the classification accuracy, models were trained on four different images of each cow: front view, side view, grayscale front view, and grayscale side view. The masked images of these views were fed to the multi-input CNN model which predicts the class of input images. The segmentation model achieved intersection-over-union (IoU) and F1-score values of 81.75% and 85.26%, respectively with an inference time of 296ms. For the classification task, multiple variants of MobileNet and EfficientNet models were used as the backbone along with pre-trained weights. The MobileNet model achieved 80.0% accuracy for both breeds, while MobileNetV2 and MobileNetV3 reached 82.0% accuracy. CNN models with EfficientNet as backbones outperformed MobileNet models, with accuracy ranging from 84.0% to 86.0%. The F1-scores for these models were found to be above 83.0%, indicating effective breed classification with fewer false positives and negatives. Thus, the present study demonstrates that deep learning models can be used effectively to identify phenotypically similar-looking cattle breeds. To accurately identify zebu breeds, this study will reduce the dependence of farmers on experts.

Evaluating the Diagnostic Accuracy of a Portable, Motorized, and Remotely Controlled Slit Lamp Imaging Adaptor Prototype for Head-Mounted Displays.

The purpose of this study was to validate the performance of a portable and remotely controlled slit lamp imaging adaptor. Twenty patients with anterior eye segment conditions participated in a randomized masked clinical trial. Imaging was performed using a Haag-Streit AG, BX 900 slit lamp biomicroscope and a new slit lamp prototype. Three ophthalmologists independently reviewed masked and randomized 2D images from both instruments and conducted physical eye examinations based on these images. Inter- and intra-grader reliability were assessed using kappa statistics, and sensitivity and specificity were determined with reference to the clinical eye examinations performed during the patients' visits. The sensitivity and specificity of the evaluations with the prototype were 47.8% and 64.1%. Similarly, the evaluations from the conventional system obtained a sensitivity and specificity of 49.5% and 66.2%. The differences in the sensitivity and specificity between imaging modalities were not statistically significant (P > 0.05). The intra-grader reliability showed moderate to substantial agreement between the systems (κ = 0.522-0.708). The inter-grader reliability also indicated moderate agreement for the evaluations with the conventional system (κ = 0.552) and the prototype (κ = 0.474). This study presents a new prototype that exhibits diagnostic accuracy on par with conventional slit lamps and moderate reliability. Further studies with larger sample sizes are required to characterize the prototype's performance. However, its remote functionality and accessibility suggest the potential to extend eye care. The development of portable and remotely controlled eye imaging systems will enhance teleophthalmology services and broaden access to eye care at the primary care level.

APW: An ensemble model for efficient wheat spike counting in unmanned aerial vehicle images

The accurate estimation of wheat ear is of great significance to gain high yield for ensuring food security. The use of computer vision for wheat ear counting to improve the estimation efficiency and accuracy has been explored by many scholars. However, existing single-computer vision recognition methods have insufficient robustness and recognition of adherent wheat ears. This study mainly focused on to solve the problem of adherent wheat ears, and proposed a combined algorithm called “APW” that ensemble with alternating direction methods of multipliers (ADMM), Potts algorithm and Watershed to achieve fast recognition and counting of wheat ears. The images of wheat ears were collected under three different scenarios including growth stages, planting densities, and flight heights by using unmanned aerial vehicle (UAV), and the Potts model was used to recognise the images. The masked images of the wheat ear were converted into a greyscale image, and were layered using a greyscale gradient. Finally, the watershed algorithm was used to segment adherent wheat ears in the layered image, and the centroids were labelled and counted. The results shows that the best accuracy was obtained under low planting density scenario, with an R2 of 0.89 and a root mean square error (RMSE) of 3.72, suggesting that APW can handle scenarios with large background differences and low target adherence. These findings provides a new approach for efficient counting of wheat ears in UAV acquired images.

A deep learning-based image analysis for assessing the extent of abduction in abducens nerve palsy patients before and after strabismus surgery

PurposeThis study aimed to propose a novel deep learning-based approach to assess the extent of abduction in patients with abducens nerve palsy before and after strabismus surgery. MethodsThis study included 13 patients who were diagnosed with abducens nerve palsy and underwent strabismus surgery in a tertiary hospital. Photographs of primary, dextroversion and levoversion position were collected before and after strabismus surgery. The eye location and eye segmentation network were trained via recurrent residual convolutional neural networks with attention gate connection based on U-Net (R2AU-Net). Facial images of abducens nerve palsy patients were used as the test set and parameters were measured automatically based on the masked images. Absolute abduction also was measured manually, and relative abduction was calculated. Agreements between manual and automatic measurements, as well as repeated automatic measurements were analyzed. Preoperative and postoperative results were compared. ResultsThe intraclass correlation coefficients (ICCs) between manual and automatic measurements of absolute abduction ranged from 0.985 to 0.992 (P＜0.001), and the bias ranged from −0.25 ​mm to −0.05 ​mm. The ICCs between two repeated automatic measurements ranged from 0.994 to 0.997 (P＜0.001), and the bias ranged from −0.11 ​mm to 0.05 ​mm. After strabismus surgery, absolute abduction of affected eye increased from 2.18 ​± ​1.40 ​mm to 3.36 ​± ​1.93 ​mm (P＜0.05). The relative abduction was improved in 76.9% patients (10/13) after surgery (P＜0.01). ConclusionsThis image analysis technique demonstrated excellent accuracy and repeatability for automatic measurements of ocular abduction, which has promising application prospects in objectively assessing surgical outcomes in patients with abducens nerve palsy.

Interreader and Intrareader Reproducibility of 18F-Flotufolastat Image Interpretation in Patients with Newly Diagnosed or Recurrent Prostate Cancer: Data from Two Phase 3 Prospective Multicenter Studies.

Interreader and intrareader reproducibility of 18F-flotufolastat PET/CT scans in newly diagnosed and recurrent prostate cancer patients was assessed from masked image evaluations from two phase 3 studies. Methods: 18F-flotufolastat PET/CT images of newly diagnosed (n = 352) or recurrent (n = 389) patients were evaluated by 3 masked readers. Cohen κ was used to assess pairwise patient- and region-level interreader agreement. Agreement among all readers was assessed using Fleiss κ. Intrareader agreement between the first and repeat read (20% of images, ≥4 wk later) was assessed using Cohen κ. Results: Pairwise interreader agreement was 95% or better (newly diagnosed) and 75% or better (recurrent). The κ coefficients were impacted by the high-agreement-low-κ paradox: Cohen κ ranged from not estimable to 0.55, whereas Fleiss κ was 0.50 (newly diagnosed) and 0.41 (recurrent). Agreement was highest in the prostate of newly diagnosed patients (≥95%) and in the pelvic lymph nodes in recurrent patients (≥87%). Intrareader agreement was 86% or better across both populations. Conclusion: 18F-flotufolastat PET/CT images can be reliably interpreted, with a high degree of inter- and intrareader agreement.

DH-GAN: Image manipulation localization via a dual homology-aware generative adversarial network

Image manipulation localization is a binary segmentation task that sensitive to the tampered artifacts other than awareness of the object. Thus, both traditional and learning-based methods highly rely on hand-crafted features. However, these specifically-defined features limit the ability of the network for general scenes. To tackle this problem, we propose a dual homology-aware generative adversarial network (DH-GAN), a novel GAN-based framework to localize the manipulated region. Firstly, we localize the forgery region via re-calibrating the multi-scale encoded features with a selective pyramid generator. Then, we perform the homology identification in the discriminator. The proposed homology-aware discriminators contain a stack of masked convolution (MConv) layers and learn to identify the real/fake of the segmented pixels on the predicted/target masked image in a hard-gating manner. Overall, the networks are optimized under a standard GAN. Experiments show that the proposed method outperforms other state-of-the-art algorithms on four popular image manipulation datasets.

An Image Quality Evaluation and Masking Algorithm Based On Pretrained Deep Neural Networks

With the growing amount of astronomical data, there is an increasing need for automated data processing pipelines, which can extract scientific information from observation data without human interventions. A critical aspect of these pipelines is the image quality evaluation and masking algorithm, which evaluate image qualities based on various factors such as cloud coverage, sky brightness, scattering light from the optical system, point-spread-function size and shape, and read-out noise. Occasionally, the algorithm requires masking of areas severely affected by noise. However, the algorithm often necessitates significant human interventions, reducing data processing efficiency. In this study, we present a deep-learning-based image quality evaluation algorithm that uses an autoencoder to learn features of high quality astronomical images. The trained autoencoder enables automatic evaluation of image quality and masking of noise affected areas. We have evaluated the performance of our algorithm using two test cases: images with point spread functions of varying full width half magnitude, and images with complex backgrounds. In the first scenario, our algorithm could effectively identify variations of the point spread functions, which can provide valuable reference information for photometry. In the second scenario, our method could successfully mask regions affected by complex regions, which could significantly increase the photometry accuracy. Our algorithm can be employed to automatically evaluate image quality obtained by different sky surveying projects, further increasing the speed and robustness of data processing pipelines.