Pixel Domain Research Articles

One-index vector quantization based adversarial attack on image classification

To improve storage and transmission, images are generally compressed. Vector quantization (VQ) is a popular compression method as it has a high compression ratio that suppresses other compression techniques. Despite this, existing adversarial attack methods on image classification are mostly performed in the pixel domain with few exceptions in the compressed domain, making them less applicable in real-world scenarios. In this paper, we propose a novel one-index attack method in the VQ domain to generate adversarial images by a differential evolution algorithm, successfully resulting in image misclassification in victim models. The one-index attack method modifies a single index in the compressed data stream so that the decompressed image is misclassified. It only needs to modify a single VQ index to realize an attack, which limits the number of perturbed indexes. The proposed method belongs to a semi-black-box attack, which is more in line with the actual attack scenario. We apply our method to attack three popular image classification models, i.e., Resnet, NIN, and VGG16. On average, 55.9 % and 77.4 % of the images in CIFAR-10 and Fashion MNIST, respectively, are successfully attacked, with a high level of misclassification confidence and a low level of image perturbation.

Sensitivity-improved Polarization Maps at 40 GHz with CLASS and WMAP Data

Improved polarization measurements at frequencies below 70 GHz with degree-level angular resolution are crucial for advancing our understanding of the Galactic synchrotron radiation and the potential polarized anomalous microwave emission and ultimately benefiting the detection of primordial B modes. In this study, we present sensitivity-improved 40 GHz polarization maps obtained by combining the CLASS 40 GHz and Wilkinson Microwave Anisotropy Probe (WMAP) Q-band data through a weighted average in the harmonic domain. The decision to include WMAP Q-band data stems from similarities in the bandpasses. Leveraging the accurate large-scale measurements from the WMAP Q band and the high-sensitivity information from the CLASS 40 GHz band at intermediate scales, the noise level at ℓ ∈ [30, 100] is reduced by a factor of 2–3 in the map space. A pixel domain analysis of the polarized synchrotron spectral index (β s ) using the WMAP K band and the combined maps (mean and 16th/84th percentiles across the β s map: −3.08−0.20+0.20 ) reveals a stronger preference for spatial variation (probability to exceed for a uniform β s hypothesis smaller than 0.001) than the results obtained using WMAP K and Ka bands ( −3.08−0.14+0.14 ). The cross-power spectra of the combined maps follow the same trend as other low-frequency data, and validation through simulations indicates negligible bias introduced by the combination method (subpercent level in the power spectra). The products of this work are publicly available on LAMBDA (https://lambda.gsfc.nasa.gov/product/class/class_prod_table.html).

Reconstruction-free Image Compression for Machine Vision via Knowledge Transfer

Reconstruction-free image compression for machine vision aims to perform machine vision tasks directly on compressed-domain representations instead of reconstructed images. Existing reports have validated the feasibility of compressed-domain machine vision. However, we observe that when using recent learned compression models, the performance gap between compressed-domain and pixel-domain vision tasks is still large due to the lack of some natural inductive biases in pixel-domain convolutional neural networks. In this paper, we attempt to address this problem by transferring knowledge from pixel domain to compressed domain. A knowledge transfer loss defined at both output level and feature level is proposed to narrow the gap between compressed domain and pixel domain. In addition, we modify neural networks for pixel-domain vision tasks to better suit compressed-domain inputs. Experimental results on several machine vision tasks show that the proposed method improves the accuracy of compressed-domain vision tasks significantly, which even outperforms learning on reconstructed images while avoiding the computational cost of image reconstruction.

An efficient steganography scheme based on wavelet transformation for side-information estimation

Previous studies have specifically demonstrated that incorporating high-quality side-information can notably enhance the steganographic security of JPEG images. To obtain more precise side-information estimates, researchers utilize sophisticated deblocking or denoising filters, which enhance security but require a substantial amount of time. The challenge lies in achieving a significant improvement in security and efficiency. This paper introduces a steganographic network named WTSNet, which leverages wavelet transform for estimating side-information through iterative optimization in both the pixel and wavelet domains. The wavelet domain is purposely designed to capture intricate high-frequency texture details and facilitate precise estimation of the side-information. Furthermore, the embedding cost is adjusted based on the polarity of the estimated rounding error, ensuring both secure and efficient performance. Experimental results demonstrate that the proposed scheme substantially enhances the security of conventional JPEG steganography, which relies on additive distortion. Moreover, it outperforms the state-of-the-art JPEG steganography methods based on estimating side-information while consuming less time.

Information Density Enhancement Using Lossy Compression in DNA Data Storage.

This study develops two deoxyribonucleic acid (DNA)lossy compression models, Models A and B, to encode grayscale images into DNA sequences, enhance information density, and enable high-fidelity image recovery. These models, distinguished by their handling of pixel domains and interpolation methods, offer a novel approach to data storage for DNA. Model A processes pixels in overlapped domains using linear interpolation (LI), whereas Model B uses non-overlapped domains with nearest-neighbor interpolation (NNI). Through a comparative analysis with Joint Photographic Experts Group (JPEG)compression, the DNA lossy compression models demonstrate competitive advantages in terms of information density and image quality restoration. The application of these models to the Modified National Institute of Standards and Technology (MNIST) dataset reveals their efficiency and the recognizability of decompressed images, which is validated by convolutional neural network (CNN) performance. In particular, Model B2, a version of Model B, emerges as an effective method for balancing high information density (surpassing over 20 times the typical densities of two bits per nucleotide) with reasonably good image quality. These findings highlight the potential of DNA-based data storage systems for high-density and efficient compression, indicating a promising future for biological data storage solutions.

Wavelet–pixel domain progressive fusion network for underwater image enhancement

Due to the complexity of the underwater environment, underwater images suffer from low-illumination, color deviation, and blurred details. Conventional underwater image enhancement (UIE) approaches have achieved remarkable success in improving color and illumination in the pixel domain. However, they ignore fine-grained details during the enhancement process. To solve these problems, we propose a novel wavelet–pixel domain progressive fusion network (WPFNet) to improve details while enhancing illumination and reducing color deviation for degraded underwater images. Unlike the UIE method, we propose a wavelet domain module (WDM) to obtain different scale frequency features with fine-grained details. Concretely, we develop a residual attention block (RAB) that optimizes low-frequency sub-images to maintain rich details, and enhance color and illumination information for low-frequency features. Transformer block (T_Block) is proposed to convert three high-frequency sub-images into high-frequency features operated by self-attention mechanism. Frequency features are obtained by adding low- and high-frequency features from the same scale, suffering from insufficient color and illumination information. Thus, the pixel domain module (PDM) is introduced to extract spatial features of different scales with rich color and illumination information in the pixel domain. Reconstruction module (REM) is proposed to fuse frequency and spatial features from small to large scale to restructure clear underwater images. The dual-domain fusion block (DFB) is proposed as a key element of REM, which exploits the constraint signal to fuse superior information of frequency and spatial features. Finally, our WPFNet obtains high-quality underwater images in four benchmark datasets and achieves superior performance compared to state-of-the-art UIE methods. The code is available at: https://github.com/LiuShiBen/WPFNet.

A semantic-driven coupled network for infrared and visible image fusion

In order to be adapted to high-level vision tasks, several infrared and visible image fusion methods cascade with the downstream network to enhance the semantic information of fusion results. However, due to the feature-level heterogeneities between fusion and downstream tasks, these methods suffer from the loss of pixel-level information and incomplete reconstruction of semantic-level information. To further improve the performance of fusion images in high-level vision tasks, we propose a semantic-driven coupled network for infrared and visible image fusion, terms as SDCFusion. Firstly, to address feature heterogeneity, we couple the segmentation and fusion networks into a joint framework such that both networks share the multi-level cross-modality coupled features. Based on the joint optimization of dual tasks, a joint action between fusion and downstream tasks is formed to force the cross-modality coupled features modeled on both pixel domain and semantic domain. Subsequently, to guide the semantic information reconstruction, we cascade two networks to form the semantic-based driven action, which continuously optimizes the fusion image to achieve semantic representation capacity. In addition, we introduce an adaptive training strategy to reduce the complexity of dual-task training. Specifically, an mIoU-based semantic measurement weight is designed to balance the joint action and driven action throughout the training process. We evaluate our method at both pixel information and semantic information levels, respectively. The qualitative and quantitative experiments verify the superiority of SDCFusion in terms of visual effects and metrics. The object detection and semantic segmentation experiments demonstrate that SDCFusion achieves superior performance in high-level vision tasks. The source code is available at https://github.com/XiaoW-Liu/SDCFusion.

Level0 to Level1B processor for MethaneAIR

Abstract. This work presents the development of the MethaneAIR Level0–Level1B processor, which converts raw L0 data to calibrated and georeferenced L1B data. MethaneAIR is the airborne simulator for MethaneSAT, a new satellite under development by MethaneSAT LLC, a subsidiary of the Environmental Defense Fund (EDF). MethaneSAT's goals are to precisely map over 80 % of the production sources of methane from oil and gas fields across the globe to an accuracy of 2–4 ppb on a 2 km2 scale. Efficient algorithms have been developed to perform dark corrections, estimate the noise, radiometrically calibrate data, and correct stray light. A forward model integrated into the L0–L1B processor is demonstrated to retrieve wavelength shifts during flight accurately. It is also shown to characterize the instrument spectral response function (ISRF) changes occurring at each sampled spatial footprint. We demonstrate fast and accurate orthorectification of MethaneAIR data in a three-step process: (i) initial orthorectification of all observations using aircraft avionics, a simple camera model, and a medium-resolution digital elevation map; (ii) registration of oxygen (O2) channel grayscale images to reference Multispectral Instrument (MSI) band 11 imagery via Accelerated-KAZE (A-KAZE) feature extraction and linear transformation, with similar co-registration of methane (CH4) channel grayscale images to the registered O2 channel images; and finally (iii) optimization of the aircraft position and attitude to the registered imagery and calculation of viewing geometry. This co-registration technique accurately orthorectifies each channel to the referenced MSI imagery. However, in the pixel domain, radiance data for each channel are offset by almost 150–200 across-track pixels (rows) and need to be aligned for the full-physics or proxy retrievals where both channels are simultaneously used. We leveraged our orthorectification tool to identify tie points with similar geographic locations in both CH4 and O2 images in order to produce shift parameters in the across-track and along-track dimensions. These algorithms described in this article will be implemented into the MethaneSAT L0–L1B processor.

JPEG Quantized Coefficient Recovery via DCT Domain Spatial-Frequential Transformer.

JPEG compression adopts the quantization of Discrete Cosine Transform (DCT) coefficients for effective bit-rate reduction, whilst the quantization could lead to a significant loss of important image details. Recovering compressed JPEG images in the frequency domain has recently garnered increasing interest, complementing the multitude of restoration techniques established in the pixel domain. However, existing DCT domain methods typically suffer from limited effectiveness in handling a wide range of compression quality factors or fall short in recovering sparse quantized coefficients and the components across different colorspaces. To address these challenges, we propose a DCT domain spatial-frequential Transformer, namely DCTransformer, for JPEG quantized coefficient recovery. Specifically, a dual-branch architecture is designed to capture both spatial and frequential correlations within the collocated DCT coefficients. Moreover, we incorporate the operation of quantization matrix embedding, which effectively allows our single model to handle a wide range of quality factors, and a luminance-chrominance alignment head that produces a unified feature map to align different-sized luminance and chrominance components. Our proposed DCTransformer outperforms the current state-of-the-art JPEG artifact removal techniques, as demonstrated by our extensive experiments.

Video2Haptics: Converting Video Motion to Dynamic Haptic Feedback with Bio-Inspired Event Processing.

In cinematic VR applications, haptic feedback can significantly enhance the sense of reality and immersion for users. The increasing availability of emerging haptic devices opens up possibilities for future cinematic VR applications that allow users to receive haptic feedback while they are watching videos. However, automatically rendering haptic cues from real-time video content, particularly from video motion, is a technically challenging task. In this paper, we propose a novel framework called "Video2Haptics" that leverages the emerging bio-inspired event camera to capture event signals as a lightweight representation of video motion. We then propose efficient event-based visual processing methods to estimate force or intensity from video motion in the event domain, rather than the pixel domain. To demonstrate the application of Video2Haptics, we convert the estimated force or intensity to dynamic vibrotactile feedback on emerging haptic gloves, synchronized with the corresponding video motion. As a result, Video2Haptics allows users not only to view the video but also to perceive the video motion concurrently. Our experimental results show that the proposed event-based processing methods for force and intensity estimation are one to two orders of magnitude faster than conventional methods. Our user study results confirm that the proposed Video2Haptics framework can considerably enhance the users' video experience.

Frequency-guidance Collaborative Triple-branch Network for single image dehazing

Image dehazing is a crucial pre-processing step in many high-level vision applications, e.g., crowd surveillance, automatic driving, and remote sensing. Recently, learning-based methods have achieved promising performance by designing various convolutional neural networks (CNNs). However, most existing CNNs are based on a single feature branch and cannot fully utilize the low-frequency and high-frequency features of haze, which inevitably affects the quality of the restored image. To this end, we develop an effective Frequency-guidance Collaborative Triple-branch Network (FCT-Net) for image dehazing. It casts dehazing as a dual-frequency component restoration task and deals with pertinent features based on a triple-branch-like architecture. Specifically, our FCT-Net can be decomposed into four components: low-frequency branch (LFB), high-frequency branch (HFB), progressive fusion branch (PFB), and image reconstruction tail (IRT). The LFB aims to mine more contextual information with dual-path feature modulation block (DFMB), while the HFB aims to recover the missing structural detail via mixed attention residual block (MARB). PFB is a fusion branch built between LFB and HFB, where different feature streams (low- and high-frequency information) can be integrated together through continuous selective progressive fusion (SPF) modules. Finally, our IRT exploits all these derived features to generate the clear haze-free image. Furthermore, we utilize a multi-strategy loss to optimize our FCT-Net from the perspectives of pixel domain and feature domain, which can achieve better image recovery quality. Extensive experiments on different benchmark datasets show that our FCT-Net is able to get comparable results with other state-of-the-art methods.

Quantitatively sensitive and resolution-enhancing strategies in optical signal conversion of SPR images using extensive angle-scanning and image-reconstruction

Surface plasmon resonance imaging (SPRi) is a technology developed from SPR that has emerged in response to the growing need for high-throughput and for discovering diverse applications across multiple domains. Despite its potential, SPRi has encountered limitations in sensitivity when handling low-molecular-mass materials, which emphasizes the need for the quantitative processing of SPRi signals. In this study, an SPRi signal conversion process was developed to enhance the contrast of SPR images and convert them into quantitatively analyzed signals. At first, an optimum incident angle of light was determined as an angle that maximized the difference between sums of the brightness % of functionalized regions and the brightness % of background region in the SPR image, which enabled to obtain a distinct SPR image. In addition, the enhancement of contrast and reduction of noise in the SPR image was achieved using Contrast Limited Adaptive Histogram Equalization (CLAHE) method. Then various pixel domains of the SPR images were simultaneously quantified and visualized the multi-pixel processing technique. For validating the process, an SiO2/Au thin film was modified with four different functionalities and applied to the detection of gas molecules. The SPR images processed using the proposed process exhibited sensitive and quantitative information about the interactions of the functionalities with the target gas in the distinct multiple regions, thus, were applicable to the quantitative monitoring of various phenomena occurring on the plasmonic interface via the advantages of SPRi.

The polarization quaternion and its applications: a joint representationof the Q/U Stokes parameters and E/B mode polarizations

In this work, we point out that the Q/U Stokes parameters and E/B modepolarizations are the four components of a unique quaternion, which describes atthe same time the directions and the parity states of spherical linearpolarizations. We then point out that, with this polarization quaternion, themathematical form of all Q/U and E/B transforms are greatly simplified, to anextent that requires only one quaternion multiplication for each transform. Apreliminary application of the polarization quaternion is shown as an example todetect peculiar pixel domain patterns within the E- and B-families, which arethe former and latter halves of the polarization quaternion.

Image processing‐based Athletes' injury prevention mechanism

AbstractThis study proposes a compressed domain video action recognition to prevent athlete injuries. The main idea is to use the motion vector and residual of the compressed video to construct new spatiotemporal features while extracting compressed code stream information. The new spatiotemporal features have the spatiotemporal relationship between motion vectors and residuals and the characteristic of precise object edges. Through verification on mainstream action recognition datasets (HMDB‐51, UCF‐101), the proposed method has a more negligible computational overhead than action recognition based on the traditional pixel domain and higher recognition accuracy than action recognition based on the video compression domain. Experiments have shown that the new spatiotemporal features based on the compressed domain have advantages such as solid spatiotemporal relationships and high information density, which can make action recognition results with high accuracy, detect abnormalities or patterns associated with injuries, and analyze an athlete's movements and provide personalized training recommendations.

Compressing of Medium- to Low-Rate Transform Residuals With Semi-Extreme Sparse Coding as an Alternate Transform in Video Coding

Modern hybrid video codecs split their video content into blocks. These blocks are predicted and the difference between original and predicted block is calculated. This residual block is transformed by a discrete cosine transform (DCT) from the pixel domain into the frequency domain and quantized by a dead-zone quantizer (DZQ), which removes high frequency signals, depending on the quantization parameter (QP). DZQ has an extended zone around zero, which acts as a noise-gate, removing noise as intended but also removing useful signals. DCT and DZQ are not the optimal solution, they create distortions and side effects at lower bitrates. In contrast, sparse coding does not have these problems at lower bitrates, but it compresses inefficient at higher bit rates. We propose a method to add sparse coding as an alternate transform, which is controlled by a rate-distortion optimization (RDO) decision, in a semi-extreme sparse coding (SESC) setup and uses a semi-extreme dictionary training (SEDT) process. It is integrated into the High Efficiency Video Coding (HEVC) test model HM-16.18 and screen content coding (HEVCSCC) test model HM-16.18+SCM-8.7. Experimental results demonstrate that the proposed method achieves a Bjontegaard rate difference (BD-rate) of up to 5.5% compared to the standard.

Unified Architecture Adaptation for Compressed Domain Semantic Inference.

Advances in both lossy image compression and semantic content understanding have been greatly fueled by deep learning techniques, yet these two tasks have been developed separately for the past decades. In this work, we address the problem of directly executing semantic inference from quantized latent features in the deep compressed domain without pixel reconstruction. Although different methods have been proposed for this problem setting, they either are restrictive to a specific architecture, or are sub-optimal in terms of compressed domain task accuracy. In contrast, we propose a lightweight, plug-and-play solution which is generally compliant with popular learned image coders and deep vision models, making it attractive to vast applications. Our method adapts prevalent pixel domain neural models that are deployed for various vision tasks to directly accept quantized latent features (other than pixels). We further suggest training the compressed domain model by transferring knowledge from its corresponding pixel domain counterpart. Experiments show that our method is compliant with popular learned image coders and vision task models. Under fair comparison, our approach outperforms a baseline method by a) more than 3% top-1 accuracy for compressed domain classification, and b) more than 7% mIoU for compressed domain semantic segmentation, at various data rates.

Model-independent search for anisotropies in stochastic gravitational-wave backgrounds and application to LIGO-Virgo’s first three observing runs

A stochastic gravitational-wave (GW) background consists of a large number of weak, independent and uncorrelated events of astrophysical or cosmological origin. The GW power on the sky is assumed to contain anisotropies on top of an isotropic component, i.e., the angular monopole. Complementary to the LIGO-Virgo-KAGRA (LVK) searches, we develop an efficient analysis pipeline to compute the maximum-likelihood anisotropic sky maps in stochastic backgrounds directly in the sky pixel domain using data folded over one sidereal day. We invert the full pixel-pixel correlation matrix in map-making of the GW sky, up to an optimal eigenmode cutoff decided systematically using simulations. In addition to modeled mapping, we implement a model-independent method to probe spectral shapes of stochastic backgrounds. Using data from LIGO-Virgo's first three observing runs, we obtain upper limits on anisotropies as well as the isotropic monopole as a limiting case, consistent with the LVK results. We also set constraints on the spectral shape of the stochastic background using this novel model-independent method.

AiWave: Volumetric Image Compression with 3-D Trained Affine Wavelet-like Transform.

Volumetric image compression has become an urgent task to effectively transmit and store images produced in biological research and clinical practice. At present, the most commonly used volumetric image compression methods are based on wavelet transform, such as JP3D. However, JP3D employs an ideal, separable, global, and fixed wavelet basis to convert input images from pixel domain to frequency domain, which seriously limits its performance. In this paper, we first design a 3-D trained wavelet-like transform to enable signal-dependent and non-separable transform. Then, an affine wavelet basis is introduced to capture the various local correlations in different regions of volumetric images. Furthermore, we embed the proposed wavelet-like transform to an end-to-end compression framework called aiWave to enable an adaptive compression scheme for various datasets. Last but not least, we introduce the weight sharing strategies of the affine wavelet-like transform according to the volumetric data characteristics in the axial direction to reduce the number of parameters. The experimental results show that: 1) when cooperating our trained 3-D affine wavelet-like transform with a simple factorized entropy coding module, aiWave performs better than JP3D and is comparable in terms of encoding and decoding complexities; 2) when adding a context module to remove signal redundancy further, aiWave can achieve a much better performance than HEVC.

Ray-Space Motion Compensation for Lenslet Plenoptic Video Coding.

Plenoptic images and videos bearing rich information demand a tremendous amount of data storage and high transmission cost. While there has been much study on plenoptic image coding, investigations into plenoptic video coding have been very limited. We investigate the motion compensation (or so-called temporal prediction) for plenoptic video coding from a slightly different perspective by looking at the problem in the ray-space domain instead of in the conventional pixel domain. Here, we develop a novel motion compensation scheme for lenslet video under two sub-cases of ray-space motion, that is, integer ray-space motion and fractional ray-space motion. The proposed new scheme of light field motion-compensated prediction is designed such that it can be easily integrated into well-known video coding techniques such as HEVC. Experimental results compared to relevant existing methods have shown remarkable compression efficiency with an average gain of 20.03% and 21.76% respectively under "Low delayed B " and "Random Access" configurations of HEVC.

Optimal Transport-Based Patch Matching for Image Style Transfer

State-of-the-art image style transfer methods have achieved impressive results by using neural networks. However, neural style transfer (NST) methods either ignore the local details of the style image by using the global statistics for style modeling or cannot fully use shallow features of neural networks, leading to the synthesized image having fewer details. In this study, we proposed a new patch-based style transfer method that directly operates in the image pixel domain without using any neural networks, achieving fascinating style transfer results with rich image details. The proposed method was derived from classic texture synthesis methods. Most previous methods rely on nearest neighbor search (NNS) for patch matching. However, this greedy strategy cannot guarantee the similarity of patch distributions between the synthesized image and the style image, which limits the expressiveness of textures. We solved this problem by proposing an optimal patch matching algorithm formed on the Optimal Transport (OT) theory, which theoretically guarantees the similarity of the patch distributions and gives a flexible style modeling method. Various qualitative and quantitative experiments demonstrated that the proposed method achieves better synthesized results than state-of-the-art style transfer methods, including NST and classic methods based on texture synthesis.