Common Test Conditions Research Articles

Amino acid-based nanoparticles-incorporated thin-film nanocomposite forward osmosis membranes for efficient desalination and heavy metal ions rejection

The current study explores potential applications of state-of-the-art thin-film nanocomposite forward osmosis (TFN-FO) membranes, modified with histidine-functionalized graphene quantum dots (His-GQDs) and MIP-202(Zr) nanoparticles (NPs), for sustainable desalination and heavy metal ions rejection. The porous and layered structure of the applied NPs, along with various hydrophilic functional groups on their surface, contribute to improving the fabricated membranes' ion/water separation performance. The successful preparation and incorporation of desired NPs into the polyamide layer was investigated using typical analytical methods. Under the common FO test conditions, the best-performing TFN-MQ2 membrane displayed a water flux of 21.8 LMH, which was over 1.5 times greater than the water flux of blank TFC. Simultaneously, the selectivity was found to be approximately 1.7 times greater than that of the unmodified TFC membrane. Moreover, the optimal TFN-MQ2 membrane exhibited superior rejection rates for Cu2+ ions (98.5 %) and Pb2+ ions (98.1 %), surpassing all other samples in heavy metal ion rejection. The findings of this study suggest that carefully choosing cost-efficient and eco-friendly nanofillers (such as amino acid-based NPs) can enhance the desalination performance of TFN-FO membranes and bolster their resistance to fouling and rejection of heavy metal ions. Not to mention, the overall costs of membrane production will be reduced.

GRNet:Geometry Restoration for G-PCC Compressed Point Clouds Using Auxiliary Density Signaling.

The lossy Geometry-based Point Cloud Compression (G-PCC) inevitably impairs the geometry information of point clouds, which deteriorates the quality of experience (QoE) in reconstruction and/or misleads decisions in tasks such as classification. To tackle it, this work proposes GRNet for the geometry restoration of G-PCC compressed large-scale point clouds. By analyzing the content characteristics of original and G-PCC compressed point clouds, we attribute the G-PCC distortion to two key factors: point vanishing and point displacement. Visible impairments on a point cloud are usually dominated by an individual factor or superimposed by both factors, which are determined by the density of the original point cloud. To this end, we employ two different models for coordinate reconstruction, termed Coordinate Expansion and Coordinate Refinement, to attack the point vanishing and displacement, respectively. In addition, 4-byte auxiliary density information is signaled in the bitstream to assist the selection of Coordinate Expansion, Coordinate Refinement, or their combination. Before being fed into the coordinate reconstruction module, the G-PCC compressed point cloud is first processed by a Feature Analysis Module for multiscale information fusion, in which kNN-based Transformer is leveraged at each scale to adaptively characterize neighborhood geometric dynamics for effective restoration. Following the common test conditions recommended in the MPEG standardization committee, GRNet significantly improves the G-PCC anchor and remarkably outperforms state-of-the-art methods on a great variety of point clouds (e.g., solid, dense, and sparse samples) both quantitatively and qualitatively. Meanwhile, GRNet runs fairly fast and uses a smaller-size model when compared with existing learning-based approaches, making it attractive to industry practitioners.

Leveraging occupancy map to accelerate video-based point cloud compression

Video-based Point Cloud Compression enables point cloud streaming over the internet by converting dynamic 3D point clouds to 2D geometry and attribute videos, which are then compressed using 2D video codecs like H.266/VVC. However, the complex encoding process of H.266/VVC, such as the quadtree with nested multi-type tree (QTMT) partition, greatly hinders the practical application of V-PCC. To address this issue, we propose a fast CU partition method dedicated to V-PCC to accelerate the coding process. Specifically, we classify coding units (CUs) of projected images into three categories based on the occupancy map of a point cloud: unoccupied, partially occupied, and fully occupied. Subsequently, we employ either statistic-based rules or machine-learning models to manage the partition of each category. For unoccupied CUs, we terminate the partition directly; for partially occupied CUs with explicit directions, we selectively skip certain partition candidates; for the remaining CUs (partially occupied CUs with complex directions and fully occupied CUs), we train an edge-driven LightGBM model to predict the partition probability of each partition candidate automatically. Only partitions with high probabilities are retained for further Rate–Distortion (R–D) decisions. Comprehensive experiments demonstrate the superior performance of our proposed method: under the V-PCC common test conditions, our method reduces encoding time by 52% and 44% in geometry and attribute, respectively, while incurring only 0.68% (0.66%) BD-Rate loss in D1 (D2) measurements and 0.79% (luma) BD-Rate loss in attribute, significantly surpassing state-of-the-art works.

NN-VVC: A Hybrid Learned-Conventional Video Codec Targeting Humans and Machines

Advancements in artificial intelligence have significantly increased the use of images and videos in machine analysis algorithms, predominantly neural networks. However, the traditional methods of compressing, storing and transmitting media have been optimized for human viewers rather than machines. Current research in coding images and videos for machine analysis has evolved in two distinct paths. The first is characterized by End-to-End (E2E) learned codes, which show promising results in image coding but have yet to match the performance of leading Conventional Video Codecs (CVC) and suffer from a lack of interoperability. The second path optimizes CVC, such as the Versatile Video Coding (VVC) standard, for machine-oriented reconstruction. Although CVC-based approaches enjoy widespread hardware and software compatibility and interoperability, they often fall short in machine task performance, especially at lower bitrates. This paper proposes a novel hybrid codec for machines named NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to − 43.20% and − 26.8% Bjøntegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks according to the common test conditions designed by the VCM study group in MPEG standardization activities. Furthermore, to improve reconstruction quality, we introduce a human-focused branch into our codec, enhancing the visual appeal of reconstructions intended for human supervision of the machine-oriented main branch.

Complexity analysis of the VVenC versus VVC encoder

The joint video experts team (JVET) has recently finalized a next-generation open-source video codec, called versatile video coding (VVC). The new standard presents a higher gain in the run time and rate compressions. Based on the reference software (VTM, VVC Test Model) of VVC, an optimized encoder was developed these last years resulting in a fast and efficient encoder, called Fraunhofer versatile video encoder (VVenC). Based on the Bjontegaard methodology and the GPROF profiling tool, this paper presents a technical complexity analysis and comparison of both VVC and VVenC. The appropriate comparisons cover the percentage taken by each block in terms of processing time and the resulting whole encoding time. The peak signal-to-noise ratio (PSNR) and bit rate between VVC and VVenC encoders based on the common test conditions manual are also analyzed and compared. The profiling results show that the VVenC encoder presents a maximum gain of runtime and bit rate of 90% and 20% respectively in classes A and D test sequences, compared to the VVC encoder.

A convolutional neural network-based rate control algorithm for VVC intra coding

The Versatile Video Coding (VVC) has shown significant improvements in Rate-Distortion (R-D) performance compared to its predecessor, High Efficiency Video Coding (HEVC). However, it still encounters several challenges. One of these challenges is the efficient allocation of bits among all Coding Tree Units (CTUs). Additionally, there is a lack of prior information for intra-frame coding, particularly for the first frame. After CTU-level bit allocation, only fixed parameters can be used to determine the λ for CTUs, which does not result in optimal rate–distortion performance. To tackle above challenges, we propose a rate control solution based on Convolutional Neural Network (CNN). This approach utilizes CNN to predict the key parameters α and β in the R-D model, addressing the problem of lacking prior information in intra-frame coding. Subsequently, the predicted α and β values are used to adaptively allocate bits for each CTU. Our proposed algorithm is implemented in VTM-16.0 under Common Test Conditions (CTC). Experimental results show that, compared to the default rate control algorithm in VTM-16.0, our proposed algorithm enhances R-D performance by 0.96% while maintaining rate control accuracy.

Spatio-Temporal Convolutional Neural Network for Enhanced Inter Prediction in Video Coding.

This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

Transform-Based Feature Map Compression Method for Video Coding for Machines (VCM)

The burgeoning field of machine vision has led to the development by the Moving Picture Experts Group (MPEG) of a new type of compression technology called video coding for machines (VCM), to enhance machine recognition through video information compression. This research proposes a principal component analysis (PCA)-based compression methodology for multi-level feature maps extracted from the feature pyramid network (FPN) structure. Unlike current PCA-based studies that independently carry out PCA for each feature map, our approach employs a generalized basis matrix and mean vector derived from channel correlations by a generalized PCA process to eliminate the need for a PCA process. Further compression is achieved by amalgamating high-dimensional feature maps, capitalizing on the spatial redundancy within these multi-level feature maps. As a result, the proposed VCM encoder forgoes the PCA process, and the generalized data do not incur any compression loss. It only requires compressing the coefficients for each feature map using versatile video coding (VVC). Experimental results demonstrate superior performance by our method over all feature anchors for each machine vision task, as specified by the MPEG-VCM common test conditions, outperforming previous PCA-based feature map compression methods. Notably, it achieved an 89.3% BD-rate reduction for instance segmentation tasks.

Overview of Versatile Video Coding (H.266/VVC) and Its Coding Performance Analysis

Versatile Video Coding (H.266/VVC) is the newest video coding standard jointly developed by the Joint Video Experts Team (JVET), which is organized by the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). H.266/VVC provides about 40% bitrate reduction compared with High Efficiency Video Coding (H.265/HEVC) for the same visual quality. This paper introduces in detail the core structure of H.266/VVC by highlighting its features within block partitioning structure, intra/inter prediction, transform, quantization, and in-loop filtering, compared to its predecessor (H.265/HEVC). H.266/VVC yields significantly improved the coding performance, but it increased the computational complexity, particularly for the encoder side, which remains a problem to be tackled for successful commercialization in the future. This paper examined the statistical performance of H.266/VVC coding tools from the bitstreams encoded by the VVC test model (VTM12.0) encoder through rate-distortion optimization under the JVET common test conditions. In addition, the complexity and performance analyses are conducted on the block partitioning structure and group of picture structure. It is expected that an optimized H.266/VVC encoder can be designed and developed by minimizing the coding loss based on the analysis data.

Learned Image Compression Using Cross-Component Attention Mechanism.

Learned image compression methods have achieved satisfactory results in recent years. However, existing methods are typically designed for RGB format, which are not suitable for YUV420 format due to the variance of different formats. In this paper, we propose an information-guided compression framework using cross-component attention mechanism, which can achieve efficient image compression in YUV420 format. Specifically, we design a dual-branch advanced information-preserving module (AIPM) based on the information-guided unit (IGU) and attention mechanism. On the one hand, the dual-branch architecture can prevent changes in original data distribution and avoid information disturbance between different components. The feature attention block (FAB) can preserve the important information. On the other hand, IGU can efficiently utilize the correlations between Y and UV components, which can further preserve the information of UV by the guidance of Y. Furthermore, we design an adaptive cross-channel enhancement module (ACEM) to reconstruct the details by utilizing the relations from different components, which makes use of the reconstructed Y as the textural and structural guidance for UV components. Extensive experiments show that the proposed framework can achieve the state-of-the-art performance in image compression for YUV420 format. More importantly, the proposed framework outperforms Versatile Video Coding (VVC) with 8.37% BD-rate reduction on common test conditions (CTC) sequences on average. In addition, we propose a quantization scheme for context model without model retraining, which can overcome the cross-platform decoding error caused by the floating-point operations in context model and provide a reference approach for the application of neural codec on different platforms.

JPEG Column: 97th JPEG Meeting

The 97th JPEG meeting was held online from 24 to 28 October 2022. JPEG received responses to the Call for Proposals (CfP) on JPEG Fake Media, the first multimedia international standard designed to facilitate the secure and reliable annotation of media assets creation and modifications. In total six responses were received addressing different requirements in the scope of this standardization initiative. Moreover, relevant advances were made on the standardization of learning-based coding, notably the learning-based coding of images, JPEG AI, and JPEG Pleno point cloud coding. Furthermore, the explorations on quality assessment of images, JPEG AIC, and of JPEG Pleno light field had relevant advances with the definition of their Calls for Contributions and Common Test Conditions.

Equal Value String and Copy Above String Based String Prediction for SCC in AVS3

String prediction (SP) is a very efficient screen content coding (SCC) tool which has been adopted in the third generation of Audio Video Standard (AVS3). It is observed that two special types of strings occur frequently. To further improve the coding efficiency for SCC on top of the original SP, a new variation of SP named Equal-value-string and Copy-above-string based SP (ECSP) is proposed. An ECSP coding unit uses only three types of strings: Equal-value-string, Copy-above-string, and Unpredictable-pixel-string. Compared with the AVS3 reference software HPM9.0 with ECSP disabled, using AVS3 SCC Common Test Condition and YUV 4:2:0 test sequences, the proposed technique achieves an average Y BD-rate reduction of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">5.54</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3.01%</i> for All Intra and Low Delay configurations, respectively, with low additional encoding and decoding complexity. The proposed ECSP has been adopted in the AVS3 standard.

Inverse Transform Using Linearity for Video Coding

In hybrid block-based video coding, transform plays an important role in energy compaction. Transform coding converts residual data in the spatial domain into frequency domain data, thereby concentrating energy in a lower frequency band. In VVC (versatile video coding), the primary transform is performed using DCT-II (discrete cosine transform type 2), DST-VII (discrete sine transform type 7), and DCT-VIII (discrete cosine transform type 8). Considering that DCT-II, DST-VII, and DCT-VIII are all linear transforms, inverse transform is proposed to reduce the number of computations by using the linearity of transform. When the proposed inverse transform using linearity is applied to the VVC encoder and decoder, run-time savings can be achieved without decreasing the coding performance relative to the VVC decoder. It is shown that, under VVC common-test conditions (CTC), average decoding time savings values of 4% and 10% are achieved for all intra (AI) and random access (RA) configurations, respectively.

Enhanced Intra String Copy for Screen Content Coding in AVS3

Driven by growing applications that use computer screens as interfaces for daily remote interactions, almost all current video coding standards have included screen content coding (SCC) tools. Recently, an efficient SCC tool called intra string copy (ISC) has been adopted in the third-generation of audio video coding standard in China (AVS3). ISC has two coding unit (CU) level sub-modes: fully-matching-string and partially-matching-string based string prediction (FPSP) sub-mode and equal-value-string, unit-basis-vector-string, and unmatched-pixel-string based string prediction (EUSP) sub-mode. To further improve the coding efficiency of SCC, this paper proposes four enhancement techniques of ISC (EISC), including CU partition improvements, point vector (PV) relocation and reactivation, line-based overlapping string prediction, and an optimized coding method for string length in the EUSP sub-mode. Compared with the latest AVS3 reference software HPM with EISC disabled, using AVS3 SCC common test condition and YUV test sequences in text and graphics with motion and mixed content categories, the proposed technique achieves an average Y BD-rate reduction of 2.39% and 1.49% for all intra (AI) and low-delay B (LDB) configurations, respectively, with low additional encoding complexity and almost no additional decoding complexity. All proposed ISC enhancement techniques have been adopted in AVS3.

Enhanced pruning algorithm for improving visual quality in MPEG immersive video

The moving picture experts group (MPEG) immersive video (MIV) technology has been actively developed and standardized to efficiently deliver immersive video to viewers in order for them to experience immersion and realism in various realistic and virtual environments. Such services are provided by MIV technology, which uses multiview videos as input. The pruning process, which is an important component of MIV technology, reduces interview redundancy in multiviews videos. The primary aim of the pruning process is to reduce the amount of data that available video codec must handle. In this study, two approaches are presented to improve the existing pruning algorithm. The first method determines the order in which images are pruned. The amount of overlapping region between the source views is then used to determine the pruning order. The second method considers global region-wise color similarity to minimize matching ambiguity when determining the pruning area. The proposed methods are evaluated under common test condition of MIV, and the results show that incorporating the proposed methods can improve both objective and subjective quality.

An Analytic Transform Kernel Derivation Method for Video Codecs

In the standardization of versatile video coding (VVC), discrete cosine transform (DCT)-2, discrete sine transform (DST)-7, and DCT-8 are regarded as the primary transform kernels. However, DST-4 and DCT-4 can also be considered as the transform kernels instead of using DST-7 and DCT-8 owing to their effectiveness in smaller resolution test sequences. To implement these different block size transform kernels, a considerable amount of memory has to be allocated. Moreover, memory consumption to store different block size transform kernels is regarded as a major issue in video coding standardization. To address this problem, a common sparse unified matrix concept is introduced in this study, where any block size transform kernel matrix can be obtained after some mathematical operations. The proposed common sparse unified matrix saves approximately 80% of the static memory by storing only a few transform kernel elements for DCT-2, DST-7, and DCT-8. Full-required transform kernels are derived using the stored transform kernels and generated unit-element matrices and a permutation matrix. The static memory required is only for 1648 elements instead of 8180 elements, each with 8-bit precision. The defined common sparse unified matrix is composed of two parts: a unified DST-3 matrix and a grouped DST-7 matrix. The unified DST-3 matrix is used to derive different points of DCT-2 transform kernels, and the grouped DST-7 matrix is used to derive different points of DST-7 and DCT-8 transform kernels. The new technique of grouping concept is introduced, which shows the relationship between different rows of DST-7 transform kernels with various block sizes. The proposed grouping concept supports the fast algorithm of DST-7 by implementing the proposed method of the “one group one feature” principle. The simulation was conducted using the VTM-3.0 reference software under common test conditions. The simulation result of the all intra (AI) configuration is Y = 0.00%, U = −0.02%, V = 0.00% with an encoding time of 100%, and a decoding time of 100%. Similarly, the simulation results of random access (RA) configuration are Y = −0.01%, U = 0.09%, V = 0.06%, and the encoding and decoding times are 101% and 100%, respectively. The simulation result of the low delay B (LDB) configuration is Y = 0.01%, U = 0.08%, and V = −0.27%, for encoding and decoding times of 101% and 100%, respectively.

VVC Complexity and Software Implementation Analysis

A steady increase in available processing power continues to drive advances in video compression technology. The recently completed Versatile Video Coding (VVC) standard aims to double the compression efficiency of HEVC and deliver a same quality of video at half the bitrate. To achieve this goal, VVC includes several new methods that improve coding efficiency at the cost of increased complexity. This paper provides a complexity analysis of VVC and its VTM reference software. Whereas VVC is more complex than HEVC, it remains readily implementable in software on current generation processors. Performance of practical decoders are reported, showing that real-time decoding of 8K content is feasible. An encoder is also presented, showing that most of the compression gains of VVC over HEVC can be obtained at a small fraction of the resources needed by the VTM encoder under common test conditions.

Subblock-Based Motion Derivation and Inter Prediction Refinement in the Versatile Video Coding Standard

Efficient representation and coding of fine-granular motion information is one of the key research areas for exploiting inter-frame correlation in video coding. Representative techniques towards this direction are affine motion compensation (AMC), decoder-side motion vector refinement (DMVR), and subblock-based temporal motion vector prediction (SbTMVP). Fine-granular motion information is derived at subblock level for all the three coding tools. In addition, the obtained inter prediction can be further refined by two optical flow-based coding tools, the bi-directional optical flow (BDOF) for bi-directional inter prediction and the prediction refinement with optical flow (PROF) exclusively used in combination with AMC. The aforementioned five coding tools have been extensively studied and finally adopted in the Versatile Video Coding (VVC) standard. This paper presents technical details of each tool and highlights the design elements with the consideration of typical hardware implementations. Following the common test conditions defined by Joint Video Experts Team (JVET) for the development of VVC, 5.7% bitrate reduction on average is achieved by the five tools. For test sequences characterized by large and complex motion, up to 13.4% bitrate reduction is observed. Additionally, visual quality improvement is demonstrated and analyzed.

ViSTRA2: Video coding using spatial resolution and effective bit depth adaptation

We present a new video compression framework (ViSTRA2) which exploits adaptation of spatial resolution and effective bit depth, down-sampling these parameters at the encoder based on perceptual criteria, and up-sampling at the decoder using a deep convolution neural network. ViSTRA2 has been integrated with the reference software of both the HEVC (HM 16.20) and VVC (VTM 4.0.1), and evaluated under the Joint Video Exploration Team Common Test Conditions using the Random Access configuration. Our results show consistent and significant compression gains against HM and VVC based on Bjønegaard Delta measurements, with average BD-rate savings of 12.6% (PSNR) and 19.5% (VMAF) over HM and 5.5% (PSNR) and 8.6% (VMAF) over VTM.

JPEG Column: 91st JPEG Meeting

The 91st JPEG meeting was held online from 19 to 23 April 2021. This meeting saw several activities relating to holographic coding, notably the release of the JPEG Pleno Holography Call for Proposals, consolidated with the definition of the use cases and requirements for holographic coding and common test conditions that will assure the evaluation of the future proposals.