MPEG-4 Standard Research Articles

GRNet:Geometry Restoration for G-PCC Compressed Point Clouds Using Auxiliary Density Signaling.

The lossy Geometry-based Point Cloud Compression (G-PCC) inevitably impairs the geometry information of point clouds, which deteriorates the quality of experience (QoE) in reconstruction and/or misleads decisions in tasks such as classification. To tackle it, this work proposes GRNet for the geometry restoration of G-PCC compressed large-scale point clouds. By analyzing the content characteristics of original and G-PCC compressed point clouds, we attribute the G-PCC distortion to two key factors: point vanishing and point displacement. Visible impairments on a point cloud are usually dominated by an individual factor or superimposed by both factors, which are determined by the density of the original point cloud. To this end, we employ two different models for coordinate reconstruction, termed Coordinate Expansion and Coordinate Refinement, to attack the point vanishing and displacement, respectively. In addition, 4-byte auxiliary density information is signaled in the bitstream to assist the selection of Coordinate Expansion, Coordinate Refinement, or their combination. Before being fed into the coordinate reconstruction module, the G-PCC compressed point cloud is first processed by a Feature Analysis Module for multiscale information fusion, in which kNN-based Transformer is leveraged at each scale to adaptively characterize neighborhood geometric dynamics for effective restoration. Following the common test conditions recommended in the MPEG standardization committee, GRNet significantly improves the G-PCC anchor and remarkably outperforms state-of-the-art methods on a great variety of point clouds (e.g., solid, dense, and sparse samples) both quantitatively and qualitatively. Meanwhile, GRNet runs fairly fast and uses a smaller-size model when compared with existing learning-based approaches, making it attractive to industry practitioners.

NN-VVC: A Hybrid Learned-Conventional Video Codec Targeting Humans and Machines

Advancements in artificial intelligence have significantly increased the use of images and videos in machine analysis algorithms, predominantly neural networks. However, the traditional methods of compressing, storing and transmitting media have been optimized for human viewers rather than machines. Current research in coding images and videos for machine analysis has evolved in two distinct paths. The first is characterized by End-to-End (E2E) learned codes, which show promising results in image coding but have yet to match the performance of leading Conventional Video Codecs (CVC) and suffer from a lack of interoperability. The second path optimizes CVC, such as the Versatile Video Coding (VVC) standard, for machine-oriented reconstruction. Although CVC-based approaches enjoy widespread hardware and software compatibility and interoperability, they often fall short in machine task performance, especially at lower bitrates. This paper proposes a novel hybrid codec for machines named NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to − 43.20% and − 26.8% Bjøntegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks according to the common test conditions designed by the VCM study group in MPEG standardization activities. Furthermore, to improve reconstruction quality, we introduce a human-focused branch into our codec, enhancing the visual appeal of reconstructions intended for human supervision of the machine-oriented main branch.

Remote expert viewing, laboratory tests or objective metrics: which one(s) to trust?

We present a study on the validity of quality assessment in the context of the development of visual media coding schemes. The work is motivated by the need for reliable means for decision-taking in standardization efforts of MPEG and JVET, i.e., the adoption or rejection of coding tools during the development process of the coding standard. The study includes results considering three means: objective quality metrics, remote expert viewing, which is a method designed in the context of MPEG standardization, and formal laboratory visual evaluation. The focus of this work is on the comparison of pairs of coded video sequences, e.g., a proposed change and an anchor scheme at a given rate point. An aggregation of performance measurements across multiple rate points, such as the Bjøntegaard Delta rate, is out of the scope of this paper. The paper details the test setup for the subjective assessment methods and the objective quality metrics under consideration. The results of the three approaches are reviewed, analyzed, and compared with respect to their suitability for the decision-taking task. The study indicates that, subject to the chosen test content and test protocols, the results of remote expert viewing using a forced-choice scale can be considered more discriminatory than the results of naïve viewers in the laboratory tests. The results further that, in general, the well-established quality metrics, such as PSNR, SSIM, or MS-SSIM, exhibit a high rate of correct decision-making when their results are compared with both types of viewing tests. Among the learning-based metrics, VMAF and AVQT appear to be most robust. For the development process of a coding standard, the selection of the most suitable means must be guided by the context, where a small number of carefully selected objective metrics, in combination with viewing tests for unclear cases, appears recommendable.

Subjective performance evaluation of bitrate allocation strategies for MPEG and JPEG Pleno point cloud compression

The recent rise in interest in point clouds as an imaging modality has motivated standardization groups such as JPEG and MPEG to launch activities aiming at developing compression standards for point clouds. Lossy compression usually introduces visual artifacts that negatively impact the perceived quality of media, which can only be reliably measured through subjective visual quality assessment experiments. While MPEG standards have been subjectively evaluated in previous studies on multiple occasions, no work has yet assessed the performance of the recent JPEG Pleno standard in comparison to them. In this study, a comprehensive performance evaluation of JPEG and MPEG standards for point cloud compression is conducted. The impact of different configuration parameters on the performance of the codecs is first analyzed with the help of objective quality metrics. The results from this analysis are used to define three rate allocation strategies for each codec, which are employed to compress a set of point clouds at four target rates. The set of distorted point clouds is then subjectively evaluated following two subjective quality assessment protocols. Finally, the obtained results are used to compare the performance of these compression standards and draw insights about best coding practices.

Variable-Bit-Rate Video Frame-Size Prediction by the Extended Kalman Filter Using Levenberg–Marquardt Algorithm

It is crucial to dynamically predict the future frame-sizes (bit-rates) for multimedia networking. All of the conventional bit-rate predictors are based on the assumption that instantaneous bit-rates are known precisely all the time (in the absence of uncertainty) which is surely not realistic in practice. In this work, we propose a new expectation-maximization (EM) based extended Kalman filter (EKF) to predict the bit-rates, where the EKF state-transition models will be optimized by the Levenberg–Marquardt algorithm (LMA). The main advantages of our proposed novel EKF-based bit-rate prediction approach are given as follows. First, our proposed EKF-based predictor can optimally estimate the bit-rates in the presence of uncertainty and/or noise. Second, our proposed novel EKF-based bit-rate prediction approach does not require a separate classifier to determine the individual frame-types as the conventional approach so our approach would be more robust than the conventional approach. Numerical evaluation of bit-rate (frame-size) prediction is also conducted over three movies encoded by the MPEG-4 standard. Compared to the existing Kalman-filter based bit-rate prediction methods, our proposed new LMA-EKF predictor can achieve much better performance in terms of the normalized mean square error (NMSE) and the inverse of signal-to-noise-ratio (SNR).

Local Geometry-Based Intra Prediction for Octree-Structured Geometry Coding of Point Clouds

Point cloud compression (PCC) is crucial for efficient and flexible storage as well as feasible transmission of point clouds in practice. For geometry compression, one popular approach is the octree-based solution. The intra prediction mechanism utilizes the spatial correlation in the static point cloud to predict the occupancy bit of the octree node for entropy coding, reducing the spatial redundancy. In this study, two local geometry-based prediction methods are proposed following statistical and theoretical analyses: binary prediction, which outputs the binary state (i.e., occupied or unoccupied), and ternary prediction, which provides a third option other than occupied or unoccupied (i.e., not predicted). In comparison to the state-of-the-art, the proposed binary prediction offers the Bjontegaard delta rate (BD-rate) of −0.8% for lossy compression and the bits per input point (bpip) of 100.09% for lossless compression in average, respectively. The binary prediction reduces the computational complexity in terms of more than 20% decrease in decoding time. In particular, it also provides noticeable reduction of the memory usage during entropy coding. The proposed ternary prediction provides −1.2% BD-rate for lossy compression and 97.19% bpip for lossless compression in average, respectively, in comparison to the state-of-the-art. While achieving performance gain, it is considerably more computational efficient by saving about 18% decoding time. Due to these advantages, part of the proposed ternary prediction has been adopted by the ongoing MPEG standard of geometry-based point cloud compression (G-PCC).

A Super-Resolution-Based Feature Map Compression for Machine-Oriented Video Coding

Recently, video and image compression methods using neural networks have received much attention. In MPEG standardization, Video Coding for Machine (VCM) is a newly arising topic which attempts to compress features/images for the purpose of machine vision tasks. Especially, compressing features has advantages in terms of privacy protection and computation off-loading. In this paper, we propose an effective feature compression method equipped with a super-resolution (SR) module for features. Our main motivation comes from the observation that features are somewhat robust to spatial distortions (e.g., AWGN, blur, quantization distortions, coding artifacts), which leads us to integrating an SR module into the compression framework. We also further explore the best training strategy of the proposed method, i.e., finding the best combination of various losses and proper input feature shapes. Our comprehensive experiments show that the proposed method outperforms the baseline in the original VCM anchor scenario on various QP values with Versatile Video Coding (VVC). Specifically, the proposed framework achieved up to 50% BD-rate reduction compared to the conventional P-layer feature map compression method for the object detection task on the OpenImage dataset.

The Next Frontier For MPEG-5 LCEVC: From HDR and Immersive Video to the Metaverse

In 2021, the newest MPEG standard was published as MPEG-5 low complexity enhancement video coding (LCEVC). Contrary to typical video codecs, LCEVC is an enhancement codec, meaning it works in combination with other codecs, to produce a more efficiently compressed video. Thanks to its simplified architecture, it is designed to be deployed as a software enhancer, which uses hardware blocks more efficiently. Despite being relatively new, it has already been adopted for a major next-gen television system (TV 3.0 in Brazil) and is being deployed across a full spectrum of applications, from broadcast to broadband. In this article we are focusing on future applications of LCEVC, from high dynamic range, 8K, and immersive video to metaverse, explaining how this new standard can make a positive impact on these applications.

Random projection and hashing based privacy preserving for image retrieval paradigm using invariant and clustered feature

In recent years, exploration of local characteristics have been an effective feature extraction technique for managing multimedia repository for image retrieval. These class of approaches accelerate the image retrieval. However, with the increase of online intrusion, it is also essential to provide confidentiality for the private online data with effective functionality and significant retrieval accuracy. In this paper, we have solved the protuberant bottleneck problem of privacy-preserving in CBIR using random projection technique for feature encryption with effective retrieval. For this, we formulated highly discriminative interest points based features that are rotation and translation invariant and are constructed using k-means clustering and silhouette concept. Moreover, we have devised locally likely arrangement hashing technique that can incorporate fast, accurate, and secure matching over encrypted features. Based on the experiment, it substantially outperforms the recent state-of-the-art techniques in terms of retrieval accuracy and time overhead on various standard benchmarks, namely, Corel, Olivia, ALOI, and MPEG-7. The proposed method on average provided retrieval accuracy of 83.10%, 81.30%, 81.62%, 73.01% for Corel, Olivia, ALOI and MPEG-7 plaintexted feature, respectively while retrieval accuracy of 79.90% for Corel, 79.10% for Olivia, 79.97% for ALOI, and 67.34% for MPEG-7 using the encrypted feature.

Enhancement of 3D Point Cloud Contents Using 2D Image Super Resolution Network

Media technology has been developed to give users a sense of immersion. Recent media using 3D spatial data, such as augmented reality and virtual reality, has attracted attention. A point cloud is a data format that consists of a number of points, and thus can express 3D media using coordinates and color information for each point. Since a point cloud has a larger capacity than 2D images, a technology to compress the point cloud is required, i.e., standardized in the international standard organization MPEG as a video-based point cloud compression (V-PCC). V-PCC decomposes 3D point cloud data into 2D patches along orthogonal directions, and those patches are placed into a 2D image sequence, and then compressed using existing 2D video codecs. However, data loss may occur while converting a 3D point cloud into a 2D image sequence and encoding this sequence using a legacy video codec. This data loss can cause deterioration in the quality of a reconstructed point cloud. This paper proposed a method of enhancing a reconstructed point cloud by applying a super resolution network to the 2D patch image sequence of a 3D point cloud.

Video-Based Point Cloud Compression Artifact Removal

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

Lightweight Hardware Transform Design for the Versatile Video Coding 4K ASIC Decoders

Versatile Video Coding (VVC) is the next generation video coding standard finalized in July 2020. VVC introduces new coding tools enhancing the coding efficiency compared to its predecessor High Efficiency Video Coding (HEVC). These new tools have a significant impact on the VVC software decoder complexity estimated to 2 times HEVC decoder complexity. In particular, the transform module includes in VVC separable and non-separable transforms named Multiple Transform Selection (MTS) and Low Frequency Non-Separable Transform (LFNST) tools, respectively. In this paper, we present an area-efficient hardware architecture of the inverse transform module for a VVC decoder. The proposed design uses a total of 64 regular multipliers in a pipelined architecture targeting Application-Specific Integrated Circuit (ASIC) platforms. It consists in a multi-standard architecture that supports the transform modules of recent MPEG standards including Advanced Video Coding (AVC), HEVC and VVC. The architecture leverages all primary and secondary transforms’ optimisations including butterfly decomposition, coefficients zeroing and the inherent linear relationship between the transforms. The synthesized results show that the proposed method sustains a constant throughput of 1 sample per cycle and a constant latency for all block sizes. The proposed hardware inverse transform module operates at 600 MHz frequency enabling to decode in real-time 4K video at 30 frames per second in 4:2:2 chroma sub-sampling format. The proposed module has been integrated in an ASIC UHD decoder targeting energy-aware decoding of VVC videos on consumer devices.

MPEG Standards for Compressed Representation of Immersive Audio

The term “immersive audio” is frequently used to describe an audio experience that provides the listener the sensation of being fully immersed or “present” in a sound scene. This can be achieved via different presentation modes, such as surround sound (several loudspeakers horizontally arranged around the listener), 3D audio (with loudspeakers at, above, and below listener ear level), and binaural audio to headphones. This article provides an overview of two recent standards that support the bitrate-efficient carriage of high-quality immersive sound. The first is MPEG-H 3D audio, which is a versatile standard that supports multiple immersive sound signal formats (channels, objects, and higher order ambisonics) and is now being adopted in broadcast and streaming applications. The second is MPEG-I immersive audio, an extension of 3D audio, currently under development, which is targeted for virtual and augmented reality applications. This will support rendering of fully user-interactive immersive sound for three degrees of user movement [three degrees of freedom (3DoF)], i.e., yaw, pitch, and roll head movement, and for six degrees of user movement [six degrees of freedom (6DoF)], i.e., 3DoF plus translational x, y, and z user position movements.

FiM’s DE - the communication package for the creative pipeline

The FotoInMotion (FiM) project is building a novel media creation platform, leveraging the use of semi-automated analysis and editing tools to empower creators to easily transform static visual acquisitions of real-world events into rich, animated and engaging objects, distributable through common channels. FiM transforms the content creative chain into an integrated pipeline across which media and metadata seamlessly flow and are exploited to produce more complex media objects. One of the addressed challenges consists the need for a seamless and efficient communication across such pipeline and on how to preserve, in a structured manner, all of the involved media and metadata. Existing standardized metadata tools and content wrappers are limited in expressivity and scope and incapable of fully supporting the needs of the content creative pipeline. This paper describes FiM’s new structured data object, i.e. the Digital Event (DE), which acts as a universal vehicle for media and metadata. It builds on well-established and emergent MPEG standards (MPEG-21, MPEG-V, MPEG-7 and MPEG HEIF), to support data diversity, interoperability, packaging and sharing, within complex, Machine Learning enhanced, creative pipelines. Our solution has been validated by creative professionals (photojournalism, fashion marketing and festivals), who have conducted experiments within the context of different creative workflows in real world scenarios. DE’s employment revealed to be advantageous, particularly in the homogenization of the media and metadata representation and packaging and in the normalization of the interaction between different pipeline components.

Влияние алгоритмов компрессии контент-сервера системы DRM на качество передаваемых звуковых программ

This work studied the influence of MPEG-4 HE-AAC v.2 and MPEG-4 xHE-AAC compression algorithms on the reproduction performance of high-quality audio signals. The first part of the article is based on the analysis of earlier publications, studied quality of the compression algorithms for MPEG-4 ISO/IEC 14493-3 (AAC, AAC+SBR, PS, AAC+SBR+PS) and MPEG D Surround standards applied in MPEG-4 HE-AAC v.2 and MPEG-4 xHE-AAC codecs of DAB and DRM systems for digital audio broadcasting, respectively. It has been demonstrated that their use for processing of high-quality audio signals in the frequency range of 40…15000 Hz, allowed to reduce the digital-data rate at the output to 24…30 кbit/s, without observing noticeable phenomena. This conclusion was confirmed in the second part of the article, which described the results of relative comparison of the quality of reproduction of audio signals broadcasted via analogues FM or digital DRM channels. The equipment employed for their realization was described, as well as the methodology to obtain test recordings in order to perform the statistical data analysis. Further, the specific data scales were shown to evaluate the signal pairs being compared. Results of the experiments and their analysis confirmed that, for a digital data rate of 30 кbit/s obtained at the output of the MPEG-4 xHE-AAC coder, the difference in quality was rather weak for audio signals ranging from 40 to 15000 Hz, broadcasted via both analogues and digital channels. In addition, only a marginal difference was noticed for the signal pairs received from the digital output channels of the transmitter processor and DRM receiver. It has been demonstrated that the MPEG-4 xHE-AAC codec, compared to the previous MPEG-4 HE-AAC v.2 version, exhibited clear advantages for practical applications in terms of sound quality. Based on the gained experience, common criteria of deterioration of the data quality were formulated and generalized for all audio compression MPEG standards.

Notice of Violation of IEEE Publication Principles: An assessment of MPEG-7 visual descriptors for images of maize plagues and diseases

Notice of Violation of IEEE Publication Principles <br><br>“An Assessment of MPEG-7 Visual Descriptors for Images of Maize Plagues and Diseases” <br> by J. Manjarrez-Sanchez <br> in the IEEE Latin America Transactions, Vol. 18, No. 8, August 2020 <br><br> After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE’s Publication Principles. <br><br> This paper contains significant portions of original content from the paper cited below. The original content was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission. <br><br> “Análisis de descriptores visuales MPEG-7 para enfermedades y plagas en plantas de maíz” <br> by Luis Angel Pinedo Sánchez <br> in his Bachelor’s Thesis, Instituto Technologico Superior de Jerez <br><br> <br/> Feature description is a fundamental process in the analysis of images for content-based retrieval and classification, among other tasks in which the image feature descriptor should have enough discriminative power to differentiate similar from dissimilar images according with a distance measure. Although several descriptors have been proposed for a variety of images, the challenge is their suitability to solve these tasks efficiently. One proposal is the set of standard MPEG-7 visual descriptors. We address their suitability to efficiently describe plagues and diseases in images of maize plants. The importance of this crop is its worldwide relevance for human and animal consumption. Experiments for similarity search queries using a set of distance measures, show that the Color Structure Descriptor with the Bray-Curtis distance is the most efficient and provides around 68% precision in most cases.

Video Compression for Surveillance Application using Deep Neural Network

We present a new approach to video compression for video surveillance by refining the shortcomings of conventional approach and substitute each traditional component with their neural network counterpart. Our proposed work consists of motion estimation, compression and compensation and residue compression, learned end-to-end to minimize the rate-distortion trade off. The whole model is jointly optimized using a single loss function. Our work is based on a standard method to exploit the spatio-temporal redundancy in video frames to reduce the bit rate along with the minimization of distortions in decoded frames. We implement a neural network version of conventional video compression approach and encode the redundant frames with lower number of bits. Although, our approach is more concerned toward surveillance, it can be extended easily to general purpose videos too. Experiments show that our technique is efficient and outperforms standard MPEG encoding at comparable bitrates while preserving the visual quality.

Point cloud coding: A privileged view driven by a classification taxonomy

Humans mainly communicate among them and with the world around them using light and vision, thus implying that visual representation technologies play a central role in human societies. While visual representation has been based on the 2D representation paradigm for many decades, multiple developments are nowadays pressing towards the adoption of more realistic and immersive 3D visual representation models. Point clouds are one of these emerging representation models. However, the huge amount of data involved asks for highly efficient coding solutions, some of which have recently started to be developed by the MPEG and JPEG standardization groups. In this hectic context, this paper proposes a privileged view over the current point cloud coding technologies, driven by a novel, appropriate classification taxonomy. For this purpose, some of the most representative point cloud coding solutions available in the literature will be reviewed to exercise the most relevant classification paths in the proposed taxonomy. It is expected that this type of classification taxonomy and privileged view may help better understanding the point cloud coding landscape for further solid and consistent advancements in this emerging technical area.

An Analysis of the Current Situation of College Students’ Learning: Based on Survey Research Concerning the Need of Wuyi University to Increase College Students’ Obligations

3D point cloud is a simple data structure representing both static and dynamic 3D objects. Though providing close-to-reality visualization, point clouds with high density raise massive demand for storage while sparse point clouds used for navigation require high precision. The emergency of compression technologies is mandatory in the form of standardization. A walkthrough for the technical approaches exploited in the two frameworks (V-PCC and G-PCC) under the MPEG standardization process is presented. Though with certain limitations, the achieved promising compression performances indicate a foreseeable evolution of the standard and a bright future for point cloud compression technologies.

Mitigation Strategies for Wet Market Food Safety Hazards to Prevent Coronavirus or Other Pathogen Outbreaks

3D point cloud is a simple data structure representing both static and dynamic 3D objects. Though providing close-to-reality visualization, point clouds with high density raise massive demand for storage while sparse point clouds used for navigation require high precision. The emergency of compression technologies is mandatory in the form of standardization. A walkthrough for the technical approaches exploited in the two frameworks (V-PCC and G-PCC) under the MPEG standardization process is presented. Though with certain limitations, the achieved promising compression performances indicate a foreseeable evolution of the standard and a bright future for point cloud compression technologies.