Pixel-wise Differences Research Articles

Bayesian Pairwise Comparison of High-Dimensional Images

A fundamental task in the automated analysis of images is the development of effective image pair comparison techniques. For two high-dimensional images, a statistical method must automatically label them as “similar” or “different” depending on whether random error and spatial dependencies could account for the pixel-wise differences. We develop a Bayesian strategy by constructing a novel extension of Dirichlet processes called the spatial random partition model (sRPM). The process groups spatially proximal image pixels with similar intensities into clusters, thereby achieving dimension reduction in the large number of pixels. Next, we apply the sRPM-based analytical procedure to compare two images. The image comparison problem is formulated as a hypothesis test involving a univariate metric adaptive to spatial correlations and robust to random variability in the pixel intensities. To handle the computational burden, we foster a two-stage technique for MCMC analysis and hypothesis testing of image pairs. A simulation study analyzes artificial datasets and finds compelling evidence for the high accuracy of sRPM in image comparison. We demonstrate the effectiveness of the technique by statistically analyzing satellite image data.

Anomaly Detection for Medical Images Using Heterogeneous Auto-Encoder.

Anomaly detection is an important task for medical image analysis, which can alleviate the reliance of supervised methods on large labelled datasets. Most existing methods use a pixel-wise self-reconstruction framework for anomaly detection. However, there are two challenges of these studies: 1) they tend to overfit learning an identity mapping between the input and output, which leads to failure in detecting abnormal samples; 2) the reconstruction considers the pixel-wise differences which may lead to an undesirable result. To mitigate the above problems, we propose a novel heterogeneous Auto-Encoder (Hetero-AE) for medical anomaly detection. Our model utilizes a convolutional neural network (CNN) as the encoder and a hybrid CNN-Transformer network as the decoder. The heterogeneous structure enables the model to learn the intrinsic information of normal data and enlarge the difference on abnormal samples. To fully exploit the effectiveness of Transformer in the hybrid network, a multi-scale sparse Transformer block is proposed to trade off modelling long-range feature dependencies and high computational costs. Moreover, the multi-stage feature comparison is introduced to reduce the noise of pixel-wise comparison. Extensive experiments on four public datasets (i.e., retinal OCT, chest X-ray, brain MRI, and COVID-19) verify the effectiveness of our method on different imaging modalities for anomaly detection. Additionally, our method can accurately detect tumors in brain MRI and lesions in retinal OCT with interpretable heatmaps to locate lesion areas, assisting clinicians in diagnosing abnormalities efficiently.

Action Recognition Network Based on Local Spatiotemporal Features and Global Temporal Excitation

Temporal modeling is a key problem in action recognition, and it remains difficult to accurately model temporal information of videos. In this paper, we present a local spatiotemporal extraction module (LSTE) and a channel time excitation module (CTE), which are specially designed to accurately model temporal information in video sequences. The LSTE module first obtains difference features by computing the pixel-wise differences between adjacent frames within each video segment and then obtains local motion features by stressing the effect of the feature channels sensitive to difference information. The local motion features are merged with the spatial features to represent local spatiotemporal features of each segment. The CTE module adaptively excites time-sensitive channels by modeling the interdependencies of channels in terms of time to enhance the global temporal information. Further, the above two modules are embedded into the existing 2DCNN baseline methods to build an action recognition network based on local spatiotemporal features and global temporal excitation (LSCT). We conduct experiments on the temporal-dependent Something-Something V1 and V2 datasets. We compare the recognition results with those obtained by the current methods, which proves the effectiveness of our methods.

Sequential Image Recovery Using Joint Hierarchical Bayesian Learning

Recovering temporal image sequences (videos) based on indirect, noisy, or incomplete data is an essential yet challenging task. We specifically consider the case where each data set is missing vital information, which prevents the accurate recovery of the individual images. Although some recent (variational) methods have demonstrated high-resolution image recovery based on jointly recovering sequential images, there remain robustness issues due to parameter tuning and restrictions on the type of sequential images. Here, we present a method based on hierarchical Bayesian learning for the joint recovery of sequential images that incorporates prior intra- and inter-image information. Our method restores the missing information in each image by “borrowing” it from the other images. More precisely, we couple sequential images by penalizing their pixel-wise difference. The corresponding penalty terms (one for each pixel and pair of subsequent images) are treated as weakly-informative random variables that favor small pixel-wise differences but allow occasional outliers. As a result, all of the individual reconstructions yield improved accuracy. Our method can be used for various data acquisitions and allows for uncertainty quantification. Some preliminary results indicate its potential use for sequential deblurring and magnetic resonance imaging.

A temporal–spatial analysis on the socioeconomic development of rural villages in Thailand and Vietnam based on satellite image data

Obtaining accurate and timely estimates of socio-economic status at fine geographical resolutions is essential for global sustainable development and the fight against poverty. However, data related to local socio-economic dynamics in rural villages is often either unavailable or outdated. To fill this gap, predicting local economic well-being with satellite imagery and machine learning has shown promising results. While most state-of-the-art analyses focus on predicting the levels of socio-economic status, finding temporal changes in rural villages’ economic well-being is essential for tracking the impacts of public policies (targeting e.g., poverty alleviation or access to various public services). In this paper, we propose an approach that utilizes pixel-wise differences in satellite images to classify temporal changes in average and median consumption expenditures (and income) in rural villages in Thailand and Vietnam between 2007 and 2017. This approach is shown to be able to distinguish between “Decline”, “Stagnation” and “Growth” in these outcomes with an F1 score of up to 63.2% using an Extreme Gradient Boosting Classifier model. In addition, we employ regression-based approaches which achieve an R2 of up to 39.5% when predicting actual changes in these outcomes with an Extreme Gradient Boosting Regressor. Our study demonstrates the feasibility of satellite-based estimates for measuring changes in local socio-economic dynamics.

Predicting visual difference maps for computer‐generated images by integrating human visual system model and deep learning

The quality of images generated by computer graphics rendering algorithms is mainly affected by visible distortion at some pixel locations. Image quality assessment (IQA) metrics are commonly utilized to assess the quality of rendered images, but their results are a global difference value, which does not provide pixel-wise differences to optimize the renderings. In contrast, visibility difference models including visual perception models and deep learning models can calculate pixel-wise visibility difference between distorted images and reference images. However, they either are only applied to a single type of visible distortion or are seriously dependent on datasets. To this end, the authors propose a novel model, dubbed Human Visual Perception and Deep Learning Image Difference Metric (HPDL-IDM), which combines the Human Visual System (HVS) model and deep learning. HPDL-IDM primarily consists of two modules: (i) the visual perception feature calculation module, which calculates difference maps between various kinds of features extracted from the reference image and the distorted image according to the visual characteristics of human eyes and concatenates them, and (ii) the deep learning module, which utilizes a neural network of encoder–decoder structure to train on the LocvisVC and VisTexRes datasets whose input and output are these concatenated feature difference maps and the final image distortion visibility difference map respectively. Additionally, the authors pool the final difference map into a global difference value between 0 and 1 to apply their model to many image processing tasks related to Image quality metrics (IQMs). Experimental results show that HPDL-IDM's generalization capacity and accuracy are improved by a large margin compared to other models.

Coloring Panchromatic Nighttime Satellite Images: Comparing the Performance of Several Machine Learning Methods

Artificial light-at-night (ALAN), emitted from the ground and visible from space, marks human presence on Earth. Since the launch of the Suomi National Polar Partnership satellite with the Visible Infrared Imaging Radiometer Suite Day/Night Band (VIIRS/DNB) onboard, global nighttime images have significantly improved; however, they remained panchromatic. Although multispectral images are also available, they are either commercial or free of charge, but sporadic. In this paper, we use several machine learning techniques, such as linear, kernel, random forest regressions, and elastic map approach, to transform panchromatic VIIRS/DBN into Red Green Blue (RGB) images. To validate the proposed approach, we analyze RGB images for eight urban areas worldwide. We link RGB values, obtained from ISS photographs, to panchromatic ALAN intensities, their pixel-wise differences, and several land-use type proxies. Each dataset is used for model training, while other datasets are used for the model validation. The analysis shows that model-estimated RGB images demonstrate a high degree of correspondence with the original RGB images from the ISS database. Yet, estimates, based on linear, kernel and random forest regressions, provide better correlations, contrast similarity and lower WMSEs levels, while RGB images, generated using elastic map approach, provide higher consistency of predictions.

EAR: Efficient action recognition with local-global temporal aggregation

Temporal modeling in videos is crucial for action recognition. Traditionally, it involves feature aggregation for both local motion and global semantic. In this paper, we propose an Efficient Action Recognition network (EAR), which includes a Persistence of Appearance (PA) module and a Various-timescale Aggregation (VA) module for local and global temporal aggregations respectively. For local motion aggregation, instead of using the previous time-consuming optical flow, our PA calculates pixel-wise differences in feature space as the motion representation, which is much more efficient (8196 fps vs. 8 fps in optical flow). Besides, to capture global semantic hints, we propose VA module which adaptively emphasizes expressive features and suppresses less informative ones across various timescales. Empowered by the local-global temporal aggregation, our EAR achieves competitive results on six challenging action recognition benchmarks at low FLOPs.

CG-SENSE revisited: Results from the first ISMRM reproducibility challenge.

The aim of this work is to shed light on the issue of reproducibility in MR image reconstruction in the context of a challenge. Participants had to recreate the results of "Advances in sensitivity encoding with arbitrary k-space trajectories" by Pruessmann et al. METHODS: The task of the challenge was to reconstruct radially acquired multicoil k-space data (brain/heart) following the method in the original paper, reproducing its key figures. Results were compared to consolidated reference implementations created after the challenge, accounting for the two most common programming languages used in the submissions (Matlab/Python). Visually, differences between submissions were small. Pixel-wise differences originated from image orientation, assumed field-of-view, or resolution. The reference implementations were in good agreement, both visually and in terms of image similarity metrics. While the description level of the published algorithm enabled participants to reproduce CG-SENSE in general, details of the implementation varied, for example, density compensation or Tikhonov regularization. Implicit assumptions about the data lead to further differences, emphasizing the importance of sufficient metadata accompanying open datasets. Defining reproducibility quantitatively turned out to be nontrivial for this image reconstruction challenge, in the absence of ground-truth results. Typical similarity measures like NMSE of SSIM were misled by image intensity scaling and outlier pixels. Thus, to facilitate reproducibility, researchers are encouraged to publish code and data alongside the original paper. Future methodological papers on MR image reconstruction might benefit from the consolidated reference implementations of CG-SENSE presented here, as a benchmark for methods comparison.

Fine-grained facial landmark detection exploiting intermediate feature representations

Facial landmark detection has been an active research subject over the last decade. In this paper, we present a new approach for Fine-grained Facial Landmark Detection (FFLD) improving on the precision of the detected points. A high spatial precision of facial landmarks is crucial for many applications related to aesthetic rendering, such as face modeling, face animation, virtual make-up, etc. In this paper, we present an approach that improves the detection precision. Since most facial landmarks are positioned on visible boundary lines, we train a model that encourages the detected landmarks to stay on these boundaries. Our proposed Convolutional Neural Networks (CNN) effectively exploits lower-level feature maps containing abundant boundary information. To this end, beside the main CNN predicting facial landmark positions, we use several additional components, called CropNets. CropNet receives patches cropped from feature maps at different stages of this CNN, and estimate fine corrections of its predicted positions. We also introduce a novel robust spatial loss function based on pixel-wise differences between patches cropped from predicted and ground-truth positions. To further improve the landmark localization, our framework uses several loss functions optimizing the precision at several stages in different ways. Extensive experiments show that our framework significantly increases the local precision of state-of-the-art deep coordinate regression models.

Deep Homography for License Plate Detection

The orientation of plate images in license plate recognition is one of the factors that influence its accuracy. In particular, tilted plate images are harder to detect and recognize characters with than aligned ones. To this end, the rectification of plates in a preprocessing step is essential to improve their performance. We propose deep models to estimate four-corner coordinates of tilted plates. Since the predicted corners can then be used to rectify plate images, they can help improve plate recognition in plate recognition. The main contributions of this work are a set of open-structured hybrid networks to predict corner positions and a novel loss function that combines pixel-wise differences with position-wise errors, producing performance improvements. Regarding experiments using proprietary plate images, one of the proposed modes produces a 3.1% improvement over the established warping method.

Background Extraction Based on Joint Gaussian Conditional Random Fields

Background extraction is generally the first step in many computer vision and augmented reality applications. Most existing methods, which assume the existence of a clean background during the reconstruction period, are not suitable for video sequences such as highway traffic surveillance videos, whose complex foreground movements may not meet the assumption of a clean background. Therefore, we propose a novel joint Gaussian conditional random field (JGCRF) background extraction algorithm for estimating the optimal weights of frame composition for a fixed-view video sequence. A maximum a posteriori problem is formulated to describe the intra- and inter-frame relationships among all pixels of all frames based on their contrast distinctness and spatial and temporal coherence. Because all background objects and elements are assumed to be static, patches that are motionless are good candidates for the background. Therefore, in the algorithm method, a motionless extractor is designed by computing the pixel-wise differences between two consecutive frames and thresholding the accumulation of variation across the frames to remove possible moving patches. The proposed JGCRF framework can flexibly link extracted motionless patches with desired fusion weights as extra observable random variables to constrain the optimization process for more consistent and robust background extraction. The results of quantitative and qualitative experiments demonstrated the effectiveness and robustness of the proposed algorithm compared with several state-of-the-art algorithms; the proposed algorithm also produced fewer artifacts and had a lower computational cost.

Depth Estimation of Video Sequences With Perceptual Losses

3-D vision plays an important role in intelligent perception of robot, while it requires extra 3-D sensors. Depth estimation from monocular videos provides an alternative mechanism to recover the 3-D information. In this paper, we propose an unsupervised learning framework that uses the perceptual loss for depth estimation. Depth and pose networks are first trained to estimate the depth and the camera motion of the video sequence, respectively. With the estimated depth and pose of the original frame, the adjacent frame can be reconstructed. The pixel-wise differences between the constructed frame and the original frame are used as per-pixel loss. Meanwhile, reconstructed views and original views can be used to extract advanced features from a pre-trained network to define and optimize perceptual loss functions to assess the quality of reconstructions. We combine the respective advantages of these two methods and present an approach of generating a depth map by training the feed-forward network with per-pixel loss function and perceptual loss function. The experimental results show that our method can significantly improve the estimation accuracy of depth map.

Sparse analysis model based multiplicative noise removal with enhanced regularization

The multiplicative noise removal problem for a corrupted image has recently been considered under the framework of regularization based approaches, where the regularizations are typically defined on sparse dictionaries and/or total variation (TV). This framework was demonstrated to be effective. However, the sparse regularizers used so far are based overwhelmingly on the synthesis model, and the TV based regularizer may induce the stair-casing effect in the reconstructed image. In this paper, we propose a new method using a sparse analysis model. Our formulation contains a data fidelity term derived from the distribution of the noise and two regularizers. One regularizer employs a learned analysis dictionary, and the other regularizer is an enhanced TV by introducing a parameter to control the smoothness constraint defined on pixel-wise differences. To address the resulting optimization problem, we adapt the alternating direction method of multipliers (ADMM) framework, and present a new method where a relaxation technique is developed to update the variables flexibly with either image patches or the whole image, as required by the learned dictionary and the enhanced TV regularizers, respectively. Experimental results demonstrate the improved performance of the proposed method as compared with several recent baseline methods, especially for relatively high noise levels.

Efficient, Edge-Aware, Combined Color Quantization and Dithering.

In this paper, we present a novel algorithm to simultaneously accomplish color quantization and dithering of images. This is achieved by minimizing a perception-based cost function, which considers pixel-wise differences between filtered versions of the quantized image and the input image. We use edge aware filters in defining the cost function to avoid mixing colors on the opposite sides of an edge. The importance of each pixel is weighted according to its saliency. To rapidly minimize the cost function, we use a modified multi-scale iterative conditional mode (ICM) algorithm, which updates one pixel a time while keeping other pixels unchanged. As ICM is a local method, careful initialization is required to prevent termination at a local minimum far from the global one. To address this problem, we initialize ICM with a palette generated by a modified median-cut method. Compared with previous approaches, our method can produce high-quality results with a fewer visual artifacts but also requires significantly less computational effort.

Histology validation of mapping depth-resolved cardiac fiber orientation in fresh mouse heart using optical polarization tractography.

Myofiber organization in cardiac muscle plays an important role in achieving normal mechanical and electrical heart functions. An imaging tool that can reveal microstructural details of myofiber organization is valuable for both basic research and clinical applications. A high-resolution optical polarization tractography (OPT) was recently developed based on Jones matrix optical coherence tomography (JMOCT). In this study, we validated the accuracy of using OPT for measuring depth-resolved fiber orientation in fresh heart samples by comparing directly with histology images. Systematic image processing algorithms were developed to register OPT with histology images. The pixel-wise differences between the two tractographic results were analyzed in details. The results indicate that OPT can accurately image depth-resolved fiber orientation in fresh heart tissues and reveal microstructural details at the histological level.

A Feasibility Study of Parametric Response Map Analysis of Diffusion-Weighted Magnetic Resonance Imaging Scans of Head and Neck Cancer Patients for Providing Early Detection of Therapeutic Efficacy

The parametric response map (PRM) was evaluated as an early surrogate biomarker for monitoring treatment-induced tissue alterations in patients with head and neck squamous cell carcinoma (HNSCC). Diffusion-weighted magnetic resonance imaging (DW-MRI) was performed on 15 patients with HNSCC at baseline and 3 weeks after treatment initiation of a nonsurgical organ preservation therapy (NSOPT) using concurrent radiation and chemotherapy. PRM was applied on serial apparent diffusion coefficient (ADC) maps that were spatially aligned using a deformable image registration algorithm to measure the tumor volume exhibiting significant changes in ADC (PRM(ADC)). Pretherapy and midtherapy ADC maps, quantified from the DWIs, were analyzed by monitoring the percent change in whole-tumor mean ADC and the PRM metric. The prognostic values of percentage change in tumor volume and mean ADC and PRM(ADC) as a treatment response biomarker were assessed by correlating with tumor control at 6 months. Pixel-wise differences as part of PRM(ADC) analysis revealed regions where water mobility increased. Analysis of the tumor ADC histograms also showed increases in mean ADC as early as 3 weeks into therapy in patients with a favorable outcome. Nevertheless, the percentage change in mean ADC was found to not correlate with tumor control at 6 months. In contrast, significant differences in PRM(ADC) and percentage change in tumor volume were observed between patients with pathologically different outcomes. Observations from this study have found that diffusion MRI, when assessed by PRM(ADC), has the potential to provide both prognostic and spatial information during NSOPT of head and neck cancer.

Colorization algorithm using probabilistic relaxation

This paper presents a method of colorizing a black and white imagery based on the probabilistic relaxation algorithm. Since the colorization is an ill-posed problem, a user specifies a suitable color on each isolated pixel of an image as prior information in this paper. Then other pixels in the image are colorized automatically. The colorizing process is done by assuming local Markov property on the images. By minimizing a total of RGB pixel-wise differences, the problem can be considered as a combinatorial optimization problem and it is solved by using the probabilistic relaxation. The proposed algorithm works very well when a few percent color pixels are known with confidence.