Multi-view Video Research Articles

GAN-based multi-view video coding with spatio-temporal EPI reconstruction

The introduction of multiple viewpoints in video scenes inevitably increases the bitrates required for storage and transmission. To reduce bitrates, researchers have developed methods to skip intermediate viewpoints during compression and delivery, and ultimately reconstruct them using Side Information (SInfo). Typically, depth maps are used to construct SInfo. However, these methods suffer from reconstruction inaccuracies and inherently high bitrates. In this paper, we propose a novel multi-view video coding method that leverages the image generation capabilities of Generative Adversarial Network (GAN) to improve the reconstruction accuracy of SInfo. Additionally, we consider incorporating information from adjacent temporal and spatial viewpoints to further reduce SInfo redundancy. At the encoder, we construct a spatio-temporal Epipolar Plane Image (EPI) and further utilize a convolutional network to extract the latent code of a GAN as SInfo. At the decoder, we combine the SInfo and adjacent viewpoints to reconstruct intermediate views using the GAN generator. Specifically, we establish a joint encoder constraint for reconstruction cost and SInfo entropy to achieve an optimal trade-off between reconstruction quality and bitrate overhead. Experiments demonstrate the significant improvement in Rate–Distortion (RD) performance compared to state-of-the-art methods.

Segmentation of multi-view video speed extraction method that can cross lane dividing lines

Aiming at the problem that the existing video traffic parameter extraction methods are overly dependent on manual annotation and a single perspective cannot effectively correct the dynamic driving deviation of vehicles on site, a multi-perspective video traffic parameter extraction method that can segment and cross lane dividing lines is proposed. The method consists of an automatic annotation point generation module and a multi-perspective correction module. The automatic annotation point generation module realizes the process of automatically generating annotation points by constructing a reference block based on equal-length lane dividing lines. The multi-perspective correction module proposes multiple mapping methods between vehicles and lane dividing lines and a correction speed measurement method based on the average speed probability density function to correct the two types of deviations generated by vehicles during dynamic driving. Experimental results on public data sets and measured data sets show that the vehicle speed extraction accuracy of the proposed method is better than other speed measurement methods, and it has a certain degree of universality.

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

Hierarchical QAM and Inter-Layer FEC for Multi-View Video Plus Depth Format in Two-Way Relay Channels

Hierarchical QAM and Inter-Layer FEC for Multi-View Video Plus Depth Format in Two-Way Relay Channels

Annealing Temporal–Spatial Contrastive Learning for multi-view Online Action Detection

In open-world scenarios, the analysis of action events from multiple viewpoints is crucial for achieving a holistic understanding, a concept that resonates with human perception. However, achieving a synthesis of information from multiple viewpoints presents a challenge, as the data can induce inter-view regularization, complicating the learning process. This paper is the first to delve into Online Action Detection (OAD) through a multi-view lens, underscoring the value of cross-observation in enriching view-level information. By harnessing the spatiotemporal dynamics inherent in multi-view video sequences, an Annealing Temporal–Spatial Contrastive Learning (ATSCL) consisting of Annealing Temporal Contrastive Learning (ATCL) and Spatial Contrastive Learning (SCL) is proposed, optimized for compatibility with RNN-based models. ATCL employs an annealing temporal loss to uncover the intrinsic video structures via a temporal annealing sampling mechanism. Concurrently, SCL utilizes a spatial loss to draw representations from various viewpoints closer together, mitigating the regularization effects. The ATSCL liberates training multi-view OAD from the stringent requirements of synchronized training videos, enabling the execution of OAD tasks asynchronously. Experiments demonstrate that the RNN-based models realize an average improvement of 5.92% on the DAHLIA dataset, 3.36% on the IKEA ASM dataset, 2.92% on the BREAKFAST dataset and 0.6% (mAP) on the THUMOS’14 dataset following the integration of the ATSCL framework, underscoring the ATSCL’s efficacy across different RNN structures.

Online Multicast Traffic Engineering for Multi-View Videos With View Synthesis in SDN

Online Multicast Traffic Engineering for Multi-View Videos With View Synthesis in SDN

A self-supervised spatio-temporal attention network for video-based 3D infant pose estimation

General movement and pose assessment of infants is crucial for the early detection of cerebral palsy (CP). Nevertheless, most human pose estimation methods, in 2D or 3D, focus on adults due to the lack of large datasets and pose annotations on infants. To solve these problems, here we present a model known as YOLO-infantPose, which has been fine-tuned, for infant pose estimation in 2D. We further propose a self-supervised model called STAPose3D for 3D infant pose estimation based on videos. We employ multi-view video data during the training process as a strategy to address the challenge posed by the absence of 3D pose annotations. STAPose3D combines temporal convolution, temporal attention, and graph attention to jointly learn spatio-temporal features of infant pose. Our methods are summarized into two stages: applying YOLO-infantPose on input videos, followed by lifting these 2D poses along with respective confidences for every joint to 3D. The employment of the best-performing 2D detector in the first stage significantly improves the precision of 3D pose estimation. We reveal that fine-tuned YOLO-infantPose outperforms other models tested on our clinical dataset as well as two public datasets MINI-RGBD and YouTube-Infant dataset. Results from our infant movement video dataset demonstrate that STAPose3D effectively comprehends the spatio-temporal features among different views and significantly improves the performance of 3D infant pose estimation in videos. Finally, we explore the clinical application of our method for general movement assessment (GMA) in a clinical dataset annotated as normal writhing movements or abnormal monotonic movements according to the GMA standards. We show that the 3D pose estimation results produced by our STAPose3D model significantly boost the GMA prediction performance than 2D pose estimation. Our code is available at github.com/wwYinYin/STAPose3D.

DATRA-MIV: Decoder-Adaptive Tiling and Rate Allocation for MPEG Immersive Video

The emerging immersive video coding standard moving picture experts group (MPEG) immersive video (MIV), which is ongoing standardization by MPEG-Immersive (MPEG-I) group, enables six degrees of freedom in a virtual reality environment that represents both natural and computer-generated scenes using multi-view video compression. The MIV eliminates the redundancy between multi-view videos and merges the residuals into multiple pictures, called an atlas. Thus, bitstreams with encoded atlases are generated and corresponding number of decoders are needed, which is challenging for the lightweight device with a single decoder. This article proposes a decoder-adaptive tiling and rate allocation method for MIV to overcome the challenge. First, the proposed method divides atlases into subpictures considering two aspects: (i) subpicture bitstream extracting and merging into one bitstream to use a single decoder and (ii) separation of each source view from the atlases for rate allocation. Second, the atlases are encoded by versatile video coding (VVC), using an extractable subpicture to divide the atlases into subpictures. Third, each subpicture bitstream is extracted, and asymmetric quality allocation for each subpictures is conducted by considering the residuals in the subpicture. Fourth, mixed-quality subpictures were merged by using the proposed bitstream merger. Fifth, the merged bitstream is decoded by using a single decoder. Finally, the viewing area of the user is synthesized by using the reconstructed atlases. Experimental results with the VVC test model (VTM) show that the proposed method achieves a 21.37% Bjøntegaard delta rate saving for immersive video peak signal-to-noise ratio and a 26.76% decoding runtime saving compared to the VTM anchor configuration. Moreover, it supports bitstreams for multiple decoders and single decoder without re-encoding, transcoding, or a substantial increase of the server-side storage.

Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos

Recent advancements in 4D scene reconstruction using neural radiance fields (NeRF) have demonstrated the ability to represent dynamic scenes from multi-view videos. However, they fail to reconstruct the dynamic scenes and struggle to fit even the training views in unsynchronized settings. It happens because they employ a single latent embedding for a frame while the multi-view images at the same frame were actually captured at different moments. To address this limitation, we introduce time offsets for individual unsynchronized videos and jointly optimize the offsets with NeRF. By design, our method is applicable for various baselines and improves them with large margins. Furthermore, finding the offsets always works as synchronizing the videos without manual effort. Experiments are conducted on the common Plenoptic Video Dataset and a newly built Unsynchronized Dynamic Blender Dataset to verify the performance of our method. Project page: https://seoha-kim.github.io/sync-nerf

Assessment of valve regurgitation severity via contrastive learning and multi-view video integration

Objective. This paper presents a novel approach for addressing the intricate task of diagnosing aortic valve regurgitation (AR), a valvular disease characterized by blood leakage due to incompetence of the valve closure. Conventional diagnostic techniques require detailed evaluations of multi-modal clinical data, frequently resulting in labor-intensive and time-consuming procedures that are vulnerable to varying subjective assessment of regurgitation severity. Approach. In our research, we introduce the multi-view video contrastive network, designed to leverage multiple color Doppler imaging inputs for multi-view video processing. We leverage supervised contrastive learning as a strategic approach to tackle class imbalance and enhance the effectiveness of our feature representation learning. Specifically, we introduce a contrastive learning framework to enhance representation learning within the embedding space through inter-patient and intra-patient contrastive loss terms. Main results. We conducted extensive experiments using an in-house dataset comprising 250 echocardiography video series. Our results exhibit a substantial improvement in diagnostic accuracy for AR compared to state-of-the-art methods in terms of accuracy by 9.60%, precision by 8.67%, recall by 9.01%, and F 1-score by 8.92%. These results emphasize the capacity of our approach to provide a more precise and efficient method for evaluating the severity of AR. Significance. The proposed model could quickly and accurately make decisions about the severity of AR, potentially serving as a useful prescreening tool.

Performance analysis of multiview video compression based on MIV and VVC multilayer

AbstractTo represent immersive media providing six degree‐of‐freedom experience, moving picture experts group (MPEG) immersive video (MIV) was developed to compress multiview videos. Meanwhile, the state‐of‐the‐art versatile video coding (VVC) also supports multilayer (ML) functionality, enabling the coding of multiview videos. In this study, we designed experimental conditions to assess the performance of these two state‐of‐the‐art standards in terms of objective and subjective quality. We observe that their performances are highly dependent on the conditions of the input source, such as the camera arrangement and the ratio of input views to all views. VVC‐ML is efficient when the input source is captured by a planar camera arrangement and many input views are used. Conversely, MIV outperforms VVC‐ML when the camera arrangement is non‐planar and the ratio of input views to all views is low. In terms of the subjective quality of the synthesized view, VVC‐ML causes severe rendering artifacts such as holes when occluded regions exist among the input views, whereas MIV reconstructs the occluded regions correctly but induces rendering artifacts with rectangular shapes at low bitrates.

PADELVIC: Multicamera videos and motion capture data of an amateur padel match

Recent advances in computer vision and deep learning techniques have opened new possibilities regarding the automatic labeling of sport videos. However, an essential requirement for supervised techniques is the availability of accurately labeled training datasets. In this paper we present PadelVic, an annotated dataset of an amateur padel match which consists of multi-view video streams, accurate motion capture data of one of the players, as well as synthetic videos specifically designed to serve as training sets for convolutional neural networks estimating positional data from videos. As a demonstration of one of the applications of the dataset, we present a system for the accurate prediction of the center-of-mass of the players projected onto the court plane, from a single-view video of the match.

Cross-Layer Video Synthesizing and Antenna Allocation Scheme for Multi-View Video Provisioning Under Massive MIMO Networks

Due to the growing need for bandwidth starving Multi-View Videos (MVV) in virtual reality, TV, and education, effectively allocating the resources of next-generation wireless technologies for MVV streams becomes increasingly crucial. To achieve high utility for MVV users, this paper proposes a cross-layer resource allocation mechanism to leverage video synthesizing schemes (such as Depth-Image-Based Rendering (DIBR) for efficient MVV streaming with massive MIMO). First, we formulate a new problem, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">antenna allocation with video synthesis</i> (AAVS), and prove its NP-hardness. Then, we design an approximation algorithm named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Utility-based Multi-View Synthesis</i> (UMVS) with the analytical performance provided, and dynamic scenarios are addressed by augmenting UMVS with deep reinforcement learning. Data-driven simulation results show that UMVSoutperforms existing antenna allocation schemes by at least 10%, and the DRL extension provides an additional 6% improvement in system utility under congested scenarios.

Animatable Implicit Neural Representations for Creating Realistic Avatars From Videos.

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. To improve the quality of human modeling, we further represent the human geometry as a signed distance field in the canonical space. Additionally, a neural point displacement field is introduced to enhance the capability of the blend weight field on modeling detailed human motions. Experiments show that our approach significantly outperforms recent human modeling methods.

MPS-NeRF: Generalizable 3D Human Rendering From Multiview Images.

There has been rapid progress recently on 3D human rendering, including novel view synthesis and pose animation, based on the advances of neural radiance fields (NeRF). However, most existing methods focus on person-specific training and their training typically requires multi-view videos. This paper deals with a new challenging task - rendering novel views and novel poses for a person unseen in training, using only multiview still images as input without videos. For this task, we propose a simple yet surprisingly effective method to train a generalizable NeRF with multiview images as conditional input. The key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Using a canonical space enables our method to learn shared properties of human and easily generalize to different people. Volume deformation is used to connect the canonical space with input and target images and query image features for radiance and density prediction. We leverage the parametric 3D human model fitted on the input images to derive the deformation, which works quite well in practice when combined with our canonical NeRF. The experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.

Multi-View Video Quality Enhancement Method Based on Multi-Scale Fusion Convolutional Neural Network and Visual Saliency

Multi-View Video Quality Enhancement Method Based on Multi-Scale Fusion Convolutional Neural Network and Visual Saliency

Multi-View Time-Series Hypergraph Neural Network for Action Recognition.

Recently, action recognition has attracted considerable attention in the field of computer vision. In dynamic circumstances and complicated backgrounds, there are some problems, such as object occlusion, insufficient light, and weak correlation of human body joints, resulting in skeleton-based human action recognition accuracy being very low. To address this issue, we propose a Multi-View Time-Series Hypergraph Neural Network (MV-TSHGNN) method. The framework is composed of two main parts: the construction of a multi-view time-series hypergraph structure and the learning process of multi-view time-series hypergraph convolutions. Specifically, given the multi-view video sequence frames, we first extract the joint features of actions from different views. Then, limb components and adjacent joints spatial hypergraphs based on the joints of different views at the same time are constructed respectively, temporal hypergraphs are constructed joints of the same view at continuous times, which are established high-order semantic relationships and cooperatively generate complementary action features. After that, we design a multi-view time-series hypergraph neural network to efficiently learn the features of spatial and temporal hypergraphs, and effectively improve the accuracy of skeleton-based action recognition. To evaluate the effectiveness and efficiency of MV-TSHGNN, we conduct experiments on NTU RGB+D, NTU RGB+D 120 and imitating traffic police gestures datasets. The experimental results indicate that our proposed method model achieves the new state-of-the-art performance.

Feature extraction using multi-view video analytics for dairy cattle body weight estimation

Accurate body weight (BW) estimation of livestock provides valuable information about animal health and welfare and can be used to assess changes in nutrient status of individuals or groups of animals. To obtain BW information regularly, automated low-cost methods that do not intrude upon normal farm operation are essential. Measurements of BW from 196 lactating, Holstein dairy cattle were collected across 5 days (655 +/- 77.1 kg; range 458 - 876 kg). This population of animals was used to develop a BW estimation system for dairy cattle using a synchronized two-camera system. The system applies deep-learning models and domain knowledge to both camera views to estimate BW of dairy cattle despite challenges such as occluding fences and crowded backgrounds. Videos from side and front-view were used to extract features about the anatomical locations and shape of the individual cows, and a regression model was applied to the features for BW prediction. We demonstrate that each view provides valuable information for BW estimation, where combining two views outperforms either view separately. This is despite the side view having distractors such as occluding fences and crowded backgrounds. The experimental results, applied to videos captured within the constraints of an operating dairy farm, demonstrate a root-mean-squared prediction error of 34 kg and a mean absolute percentage error of 4.1%, when a Random Forest is trained on four days of data and tested on the fifth unseen day. This model enabled 57% of cattle BW to be estimated within 25 kg of their measured BW. In addition, we demonstrated that linear regression generalized well to unseen data.

View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume.

Image view synthesis has seen great success in reconstructing photorealistic visuals, thanks to deep learning and various novel representations. The next key step in immersive virtual experiences is view synthesis of dynamic scenes. However, several challenges exist due to the lack of high-quality training datasets, and the additional time dimension for videos of dynamic scenes. To address this issue, we introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes. We develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable view extrapolation from binocular videos of dynamic scenes, captured by static cameras. Our algorithm addresses the temporal inconsistency of disocclusions by identifying the error-prone areas with a 3D mask volume, and replaces them with static background observed throughout the video. Our method enables manipulation in 3D space as opposed to simple 2D masks, We demonstrate better temporal stability than frame-by-frame static view synthesis methods, or those that use 2D masks. The resulting view synthesis videos show minimal flickering artifacts and allow for larger translational movements.

Joint Solution for Temporal-Spatial Synchronization of Multi-View Videos and Pedestrian Matching in Crowd Scenes

Joint Solution for Temporal-Spatial Synchronization of Multi-View Videos and Pedestrian Matching in Crowd Scenes