Rate-distortion Performance Research Articles

Corner-to-Center long-range context model for efficient learned image compression

In the framework of learned image compression, the context model plays a pivotal role in capturing the dependencies among latent representations. To reduce the decoding time resulting from the serial autoregressive context model, the parallel context model has been proposed as an alternative that necessitates only two passes during the decoding phase, thus facilitating efficient image compression in real-world scenarios. However, performance degradation occurs due to its incomplete casual context. To tackle this issue, we conduct an in-depth analysis of the performance degradation observed in existing parallel context models, focusing on two aspects: the Quantity and Quality of information utilized for context prediction and decoding. Based on such analysis, we propose the Corner-to-Center transformer-based Context Model (C3M) designed to enhance context and latent predictions and improve rate–distortion performance. Specifically, we leverage the logarithmic-based prediction order to predict more context features from corner to center progressively. In addition, to enlarge the receptive field in the analysis and synthesis transformation, we use the Long-range Crossing Attention Module (LCAM) in the encoder/decoder to capture the long-range semantic information by assigning the different window shapes in different channels. Extensive experimental evaluations show that the proposed method is effective and outperforms the state-of-the-art parallel methods. Finally, according to the subjective analysis, we suggest that improving the detailed representation in transformer-based image compression is a promising direction to be explored.

Medical Image Compression Using Block-to-Row Principal Component Analysis (BTRPCA)

Principal Component Analysis (PCA) is capable of completely decorrelating input data in the transform domain. However, PCA is limited in image compression because there is a need to transmit or store the eigenvectors of the input data over a communication link and thereby affects the rate-distortion performance. In an effort to improve rate-distortion performance, this work proposed a block-to-row PCA (BTRPCA) algorithm that employs the eigenvectors from the model image of the same image modality coupled with a row vectorization approach. It is found from this work that the proposed method achieves PSNR improvements of up to 10 dB compared to its PCA counterparts at compression ratio of 64:1. At the same compression ratio, the proposed BTRPCA managed to achieved PSNR of 40 dB while the comparing algorithms scores well below 40 dB at the same configuration. This approach successfully improves the rate-distortion performance by reducing the overwhelming side information and computation overhead associated with PCA.

Belief Propagation Optimization for Lossy Compression Based on Gaussian Source

In the Internet of Things, sensor nodes collect environmental information and utilize lossy compression for saving storage space. To achieve this objective, high-efficiency compression of the continuous source should be studied. Different from existing schemes, lossy source coding is implemented based on the duality principle in this work. Referring to the duality principle between the lossy source coding and the channel decoding, the belief propagation (BP) algorithm is introduced to realize lossy compression based on a Gaussian source. In the BP algorithm, the log-likelihood ratios (LLRs) are iterated, and their iteration paths follow the connecting relation between the check nodes and the variable nodes in the protograph low-density parity-check (P-LDPC) code. During LLR iterations, the trapping set is the main factor that influences compression performance. We propose the optimized BP algorithms to weaken the impact of trapping sets. The simulation results indicate that the optimized BP algorithms obtain better distortion–rate performance.

A new paradigm for high-capacity reversible data hiding with pixel repetition and adaptive embedding

Reversible data hiding (RDH) offers a useful attribute of retaining the visual semantics with security properties, for the effective management and protection of digital images with their metadata. This paper introduces a Reversible Data Hiding (RDH) scheme with Pixel Repetition (PR) to selectively embed additional secret data. bits utilizing the image redundancy created with PR. A generalized framework of PR is first developed and the rate–distortion criterion is extended for adaptive embedding. An input image is defined to be scaled-up by replacing each original pixel (called seed) with a block of shape s×s containing the seed, where s is the scaling factor. For better utilizing this PR-based image redundancy, we then propose an RDH scheme with two-phase embedding for s=2. The first phase of the scheme embeds 4-bit data in each 2 × 2 image block using a mapping function, which is designed to modify the repeated pixels, keeping the seed intact in each block. Being invertible, the mapping function also allows another phase of embedding in the preserved seeds using an adaptive selection of their least significant bits. Compared to the prominent PR-based and other related schemes, an impressive rate–distortion performance of the proposed scheme is recorded with a greater potential for the applications that require a high and varying embedding rate with better image quality.

Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding

The emerging field semantic communication is driving the research of end-to-end data transmission. By utilizing the powerful representation ability of deep learning models, learned data transmission schemes have exhibited superior performance than the established source and channel coding methods. While, so far, research efforts mainly concentrated on architecture and model improvements toward a static target domain. Despite their successes, such learned models are still suboptimal due to the limitations in model capacity and imperfect optimization and generalization, particularly when the testing data distribution or channel response is different from that adopted for model training, as is likely to be the case in real-world. To tackle this, in this paper, we propose a novel online learned joint source and channel coding approach that leverages the deep learning model’s overfitting property. Specifically, we update the off-the-shelf pre-trained models after deployment in a lightweight online fashion to adapt to the distribution shifts in source data and environment domain. We take the overfitting concept to the extreme, proposing a series of implementation-friendly methods to adapt the codec model or representations to an individual data or channel state instance, which can further lead to substantial gains in terms of the end-to-end rate-distortion performance. Accordingly, the streaming ingredients include both the semantic representations of source data and the online updated decoder model parameters. The system design is formulated as a joint optimization problem whose goal is to minimize the loss function, a tripartite trade-off among the data stream bandwidth cost, model stream bandwidth cost, and end-to-end distortion. The proposed methods enable the communication-efficient adaptation for all parameters in the network without sacrificing decoding speed. Extensive experiments, including user study, on continually changing target source data and wireless channel environments, demonstrate the effectiveness and efficiency of our approach, on which we outperform existing state-of-the-art engineered transmission scheme (VVC combined with 5G LDPC coded transmission).

Block-Adaptive Point Cloud Attribute Coding With Region-Aware Optimized Transform

Block-based compression scheme shows remarkable success in image and video coding. However, existing tree-type block partition methods usually divide point clouds into clusters with few or disjoint points due to the irregular sampling, which is adverse for the subsequent transform to exploit the local region correlation. Moreover, the widely-used optimal transform, e.g., Discrete Cosine Transform (DCT), is deduced with the assumption of the specified probability model. Thus these transforms cannot adequately conform to diverse statistical characteristics of point cloud attributes. To address the above two problems, we propose a block-adaptive codec with the optimized transform for point cloud attribute compression, including Progressive Clustering (PC) for block partition and Region-Aware Signal Modeling (RASM) for transform. PC is designed via the split-and-merge strategy. In the splitting phase, over-segmented clusters are obtained by iteratively searching the nearest neighbors to maintain the spatial continuity of points in one cluster. In the merging step, blocks are determined by merging over-segmented clusters under the guidance of minimizing texture complexity, which adjusts block sizes to adapt to texture variation. RASM adaptively captures the color correlations to respond to diverse statistical characteristics by optimizing the overall Rate-Distortion (RD) cost. Then, the optimal transform bases are obtained via the eigen-decomposition of the color correlation representation with respect to the Gaussian Markov Random Field, where input colors are obtained considering the linear relationship between geometry and color variations. The linear coefficients also need to be encoded to combine geometry to reconstruct the transform bases at the decoder in the same way as the encoder. Particularly, we accelerate the RASM by constraining the independent relationship between signals in the local region, which hardly influences RD performance. Extensive experiments not only indicate the effectiveness of PC and RASM but also show the significant RD gains achieved by our approach compared with several state-of-the-art platforms.

Asymmetric Learned Image Compression With Multi-Scale Residual Block, Importance Scaling, and Post-Quantization Filtering

Recently, deep learning-based image compression has made significant progresses, and has achieved better rate-distortion (R-D) performance than the latest traditional method, H.266/VVC, in both MS-SSIM metric and the more challenging PSNR metric. However, a major problem is that the complexities of many leading learned schemes are too high. In this paper, we propose an efficient and effective image coding framework, which achieves similar R-D performance with lower complexity than the state of the art. First, we develop an improved multi-scale residual block (MSRB) that can expand the receptive field and capture global information more efficiently, which further reduces the spatial correlation of the latent representations. Second, an importance scaling network is introduced to directly scale the latents to achieve content-adaptive bit allocation without sending side information, which is more flexible than previous importance map methods. Third, we apply a post-quantization filter (PQF) to reduce the quantization error, motivated by the Sample Adaptive Offset (SAO) filter in video coding. Moreover, our experiments show that the performance of the system is less sensitive to the complexity of the decoder. Therefore, we design an asymmetric paradigm, in which the encoder employs three stages of MSRBs to improve the learning capacity, whereas the decoder only uses one stage of MSRB, which reduces the decoder complexity and still yields satisfactory performance. Experimental results show that compared to the state-of-the-art method, the encoding and decoding time of the proposed method are about 17 times faster, and the R-D performance is only reduced by about 1% on both Kodak and Tecnick-40 datasets, which is still better than H.266/VVC(4:4:4) and other leading learning-based methods. Our source code is publicly available at https://github.com/fengyurenpingsheng.

Rethinking Semantic Image Compression: Scalable Representation With Cross-Modality Transfer

This article proposes the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable cross-modality compression</i> (SCMC) paradigm, in which the image compression problem is further cast into a representation task by hierarchically sketching the image with different modalities. Herein, we adopt the conceptual organization philosophy to model the overwhelmingly complicated visual patterns, based upon the semantic, structure, and signal level representation accounting for different tasks. The SCMC paradigm that incorporates the representation at different granularities supports diverse application scenarios, such as high-level semantic communication and low-level image reconstruction. The decoder, which enables the recovery of the visual information, benefits from the scalable coding based upon the semantic, structure, and signal layers. Qualitative and quantitative results demonstrate that the SCMC can convey accurate semantic and perceptual information of images, especially at low bitrates, and promising rate-distortion performance has been achieved compared to state-of-the-art methods. The code will be available online https://github.com/ppingzhang/SCMC.

Rate distortion optimization with adaptive content modeling for random-access versatile video coding

In this paper, we study the capability of improving the rate-distortion (RD) performance based on the adaptive content modeling in the Versatile Video Coding (VVC) standard. In particular, the frame-level dependent relationships and inherent RD relationship are explored. As such, the rate dependency, distortion dependency and inherent RD characteristics are fully utilized in the global rate distortion optimization (RDO) process, and the quantization parameter (QP) for each frame could be adaptively solved. To facilitate the adaptive QP calculation, a two-pass coding strategy is proposed. In the first-pass coding, the video statistics are sufficiently collected with the proposed Dual Motion Compensation and Residual Coding (DMCRC) method to generate the parameters for the content-aware models. During the second-pass coding, the optimal QP at the frame level is obtained by optimizing the global RD performance with the dependent and inherent models. The proposed algorithm is implemented on VVC test model (VTM-4.0) and achieves significant performance gain for test sequences with constant and varying scenes.

Variable Rate Point Cloud Geometry Compression Method

With the development of 3D sensors technology, 3D point cloud is widely used in industrial scenes due to their high accuracy, which promotes the development of point cloud compression technology. Learned point cloud compression has attracted much attention for its excellent rate distortion performance. However, there is a one-to-one correspondence between the model and the compression rate in these methods. To achieve compression at different rates, a large number of models need to be trained, which increases the training time and storage space. To address this problem, a variable rate point cloud compression method is proposed, which enables the adjustment of the compression rate by the hyperparameter in a single model. To address the narrow rate range problem that occurs when the traditional rate distortion loss is jointly optimized for variable rate models, a rate expansion method based on contrastive learning is proposed to expands the bit rate range of the model. To improve the visualization effect of the reconstructed point cloud, a boundary learning method is introduced to improve the classification ability of the boundary points through boundary optimization and enhance the overall model performance. The experimental results show that the proposed method achieves variable rate compression with a large bit rate range while ensuring the model performance. The proposed method outperforms G-PCC, achieving more than 70% BD-Rate against G-PCC, and performs about, as well as the learned methods at high bit rates.

Learned Progressive Image Compression With Dead-Zone Quantizers

Progressive coding is essential to the practical deployment of learned image compression over heterogeneous networks and clients. Existing methods for learned progressive image compression require complex and empirical design to achieve near-optimal rate-distortion performance over a wide range of bit-rates. However, these methods are limited by the implicit learned mechanism based on neural networks and introduction of uniform quantizers. In this paper, we propose generalized learned progressive image compression with analytic rate-distortion optimization using dead-zone quantizers on the latent representation. Specifically, we reveal that dead-zone quantizers, as a general case of uniform quantizers, are equivalent to uniform quantizers in fixed-rate nonlinear transform coding and can prevent extra redundancy in embedded quantization for progressive coding. Consequently, we propose rate-distortion optimized learned progressive coding by approximating the optimal quantizer in the source spaces using dead-zone quantizers in an analytic manner on the Laplacian source. To our best knowledge, this paper is the first to achieve general learned progressive coding from the perspective of optimal quantizers. The proposed method achieves theoretically sound and practically efficient embedded quantization and learned progressive coding of latent representations with improved rate-distortion performance. It can also enable embedded quantization with diverse assignments of truncation points and support flexible configuration of quality layers of varying numbers and at varying target bit-rates. Furthermore, we successfully incorporate the proposed method into existing pre-trained fixed-rate models to realize progressive learned image compression without re-training. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performance in learned progressive image compression compared with traditional codecs and recent learned methods.

Multiple hypotheses based motion compensation for learned video compression

Recently, learned video compression has attracted copious research attention. However, among the existing methods, the motion used for alignment is limited to one hypothesis only, leading to inaccurate motion estimation, especially for the complicated scenes with complex movements. Motivated by multiple hypotheses philosophy in traditional video compression, we develop the multiple hypotheses based motion compensation for the learned video compression, in an effort to enhance the motion compensation efficiency by providing diverse hypotheses with efficient temporal information fusion. In particular, the multiple hypotheses module which produces multiple motions and warped features for mining sufficient temporal information, is proposed to provide various hypotheses inferences from the reference frame. To utilize these hypotheses more copiously, the hypotheses attention module is adopted by introducing the channel-wised squeeze-and-excitation layer and the multi-scale network. In addition, the context combination is employed to fuse the weighted hypotheses to generate effective contexts with powerful temporal priors. Finally, the valid contexts are used for promoting the compression efficiency by merging weighted warped features. Extensive experiments show that the proposed method can significantly improve the rate-distortion performance of learned video compression. Compared with the state-of-the-art method for end-to-end video compression, over 13% bit rate reductions on average in terms of PSNR and MS-SSIM can be achieved.

Impact of Image Compression on In Vitro Cell Migration Analysis

Collective cell movement is an indication of phenomena such as wound healing, embryonic morphogenesis, cancer invasion, and metastasis. Wound healing is a complicated cellular and biochemical procedure in which skin cells migrate from the wound boundaries into the wound area to reconstruct the injured skin layer(s). In vitro analysis of cell migration is an effective assay for measuring changes in cell migratory complement in response to experimental inspections. Open-source segmentation software (e.g., an ImageJ plug-in) is available to analyze images of in vitro scratch wound healing assays; however, often, these tools are error-prone when applied to, e.g., low-contrast, out-of-focus, and noisy images, and require manual tuning of various parameters, which is imprecise, tedious, and time-consuming. We propose two algorithmic methods (namely log gradient segmentation and entropy filter segmentation) for cell segmentation and the subsequent measurement of the collective cell migration in the corresponding microscopic imagery. We further investigate the effects of image compression on the algorithms’ measurement accuracy, applying lossy compression algorithms (the current ISO standards JPEG2000, JPEG, JPEG-XL and AV1, BPG, and WEBP). We aim to identify the most suitable compression algorithm that can be used for this purpose, relating rate–distortion performance as measured in terms of peak signal-to-noise ratio (PSNR) and the multiscale structural similarity index (MS-SSIM) to the segmentation accuracy obtained by the segmentation algorithms. The experimental results show that the log gradient segmentationalgorithm provides robust performance for segmenting the wound area, whereas the entropy filter segmentation algorithm is unstable for this purpose under certain circumstances. Additionally, the best-suited compression strategy is observed to be dependent on (i) the segmentation algorithm used and (ii) the actual data sequence being processed.

Nonlinear Transforms in Learned Image Compression From a Communication Perspective

Recently, remarkable progress has been made in learned image compression (LIC), in which nonlinear transforms (NTs) play a crucial role. Although there are many NT methods for improving the rate distortion performance, all the existing methods sacrifice the computational complexity and the number of parameters of the transformation. This paper provides a fundamental novel viewpoint on nonlinear transforms from a communication perspective, and shows how this idea can be extended to design efficient NT methods. In particular, the nonlinear transforms are inferred as signal modulation modules. Under this extrapolation, the current NTs are generalized as amplitude modulation that only varies the amplitude of the carrier wave. Therefore, a nonlinear modulation-like transform (NMLT) which varies the phase angle of the carrier is proposed. Moreover, this concept is extended by introducing In-phase/Quadrature (IQ) modulation, which is a boosting technique in communication field, in order to enhance NMLT. Furthermore, the Bit-interleaved technique in communication is used to guide the optimization of NTML with IQ. The experimental results on different datasets and backbone architectures verify the efficiency and robustness of the proposed methods. For example, when backbone architecture is hyperprior model, our method achieves 19.37% BD-rate reduction over GDN on the Kodak dataset. In addition, our method with channel wise autoregressive model leads to the state-of-the-art rate-distortion performance.

Advanced quantum image representation and compression using a DCT-EFRQI approach

In recent years, quantum image computing draws a lot of attention due to storing and processing image data faster compared to classical computers. A number of approaches have been proposed to represent the quantum image inside a quantum computer. Representing and compressing medium and big-size images inside the quantum computer is still challenging. To address this issue, we have proposed a block-wise DCT-EFRQI (Direct Cosine Transform Efficient Flexible Representation of Quantum Image) approach to represent and compress the gray-scale image efficiently to save computational time and reduce the quantum bits (qubits) for the state preparation. In this work, we have demonstrated the capability of block-wise DCT and DWT transformation inside the quantum domain to investigate their relative performances. The Quirk simulation tool is used to design the corresponding quantum image circuit. In the proposed DCT-EFRQI approach, a total of 17 qubits are used to represent the coefficients, the connection between coefficients and state (i.e., auxiliary), and their position for representing and compressing grayscale images inside a quantum computer. Among those, 8 qubits are used to map the coefficient values and the rest are used to generate the corresponding coefficient XY-coordinate position including one auxiliary qubit. Theoretical analysis and experimental results show that the proposed DCT-EFRQI scheme provides better representation and compression compared to DCT-GQIR, DWT-GQIR, and DWT-EFRQI in terms of rate-distortion performance.

Remote sensing image compression with long-range convolution and improved non-local attention model

It is a challenge to achieve high compression rates for remote sensing images because they have rich information and complex backgrounds. Long-range context information can help the model to identify spatial redundant features. Therefore, we propose a long-range convolution compression network (LRCompNet) for remote sensing images. In order to capture long-range context information, we first propose a long-range convolution, and a light-weight compression model is designed. Additionally, an improved non-local attention model is proposed to reduce the computation complexity in order to accommodate remote sensing image compression. We also collect two remote sensing datasets (GoogleMap dataset and GF7 dataset) to test the proposed method and existing codecs, such as JPEG, JPEG2000 and learned compression models. We assess the overall performance of our method in terms of rate-distortion curves and execution time, respectively. Results on both datasets show that the proposed compression network achieves improved rate-distortion performance. Furthermore, the proposed method significantly reduces the time complexity when compared to other methods.

Learning Context-Based Nonlocal Entropy Modeling for Image Compression

The entropy of the codes usually serves as the rate loss in the recent learned lossy image compression methods. Precise estimation of the probabilistic distribution of the codes plays a vital role in reducing the entropy and boosting the joint rate-distortion performance. However, existing deep learning based entropy models generally assume the latent codes are statistically independent or depend on some side information or local context, which fails to take the global similarity within the context into account and thus hinders the accurate entropy estimation. To address this issue, we propose a special nonlocal operation for context modeling by employing the global similarity within the context. Specifically, due to the constraint of context, nonlocal operation is incalculable in context modeling. We exploit the relationship between the code maps produced by deep neural networks and introduce the proxy similarity functions as a workaround. Then, we combine the local and the global context via a nonlocal attention block and employ it in masked convolutional networks for entropy modeling. Taking the consideration that the width of the transforms is essential in training low distortion models, we finally produce a U-net block in the transforms to increase the width with manageable memory consumption and time complexity. Experiments on Kodak and Tecnick datasets demonstrate the priority of the proposed context-based nonlocal attention block in entropy modeling and the U-net block in low distortion situations. On the whole, our model performs favorably against the existing image compression standards and recent deep image compression models.

Improving distributed video coding with deep learning

In the current state-of-the-art distributed video coding (DVC) solutions, side information (SI) frames are created by motion-compensated interpolation. This is the same technique used by frame rate upconversion operations. Recently, these operations are accomplished using deep learning (DL), with significantly improved outcomes. DL is also used to improve the decoded images of numerous compression methods. Consequently, it is natural to wonder what are the impacts of these enhancements on DVC. In this regard, we employed DL networks to create SI and enhance the quality of the DVC decoded frames. We show how these two operations were applied to DVC and their effects on rate-distortion (RD) performance. The comparison with state-of-the-art DVC systems shows an improvement in the quality of the SI frames and that of the decoded frames, and thus an improvement of the RD performance of DVC.

Lossy P-LDPC Codes for Compressing General Sources Using Neural Networks

It is challenging to design an efficient lossy compression scheme for complicated sources based on block codes, especially to approach the theoretical distortion-rate limit. In this paper, a lossy compression scheme is proposed for Gaussian and Laplacian sources. In this scheme, a new route using "transformation-quantization" was designed to replace the conventional "quantization-compression". The proposed scheme utilizes neural networks for transformation and lossy protograph low-density parity-check codes for quantization. To ensure the system's feasibility, some problems existing in the neural networks were resolved, including parameter updating and the propagation optimization. Simulation results demonstrated good distortion-rate performance.

Low Computational Coding-Efficient Distributed Video Coding: Adding a Decision Mode to Limit Channel Coding Load

Distributed video coding (DVC) is based on distributed source coding (DSC) concepts in which video statistics are used partially or completely at the decoder rather than the encoder. The rate-distortion (RD) performance of distributed video codecs substantially lags the conventional predictive video coding. Several techniques and methods are employed in DVC to overcome this performance gap and achieve high coding efficiency while maintaining low encoder computational complexity. However, it is still challenging to achieve coding efficiency and limit the computational complexity of the encoding and decoding process. The deployment of distributed residual video coding (DRVC) improves coding efficiency, but significant enhancements are still required to reduce these gaps. This paper proposes the QUAntized Transform ResIdual Decision (QUATRID) scheme that improves the coding efficiency by deploying the Quantized Transform Decision Mode (QUAM) at the encoder. The proposed QUATRID scheme's main contribution is a design and integration of a novel QUAM method into DRVC that effectively skips the zero quantized transform (QT) blocks, thus limiting the number of input bit planes to be channel encoded and consequently reducing both the channel encoding and decoding computational complexity. Moreover, an online correlation noise model (CNM) is specifically designed for the QUATRID scheme and implemented at its decoder. This online CNM improves the channel decoding process and contributes to the bit rate reduction. Finally, a methodology for the reconstruction of the residual frame (R^) is developed that utilizes the decision mode information passed by the encoder, decoded quantized bin, and transformed estimated residual frame. The Bjøntegaard delta analysis of experimental results shows that the QUATRID achieves better performance over the DISCOVER by attaining the PSNR between 0.06 dB and 0.32 dB and coding efficiency, which varies from 5.4 to 10.48 percent. In addition to this, results determine that for all types of motion videos, the proposed QUATRID scheme outperforms the DISCOVER in terms of reducing the number of input bit-planes to be channel encoded and the entire encoder's computational complexity. The number of bit plane reduction exceeds 97%, while the entire Wyner-Ziv encoder and channel coding computational complexity reduce more than nine-fold and 34-fold, respectively.