Video Partitioning Research Articles

Efficient inter partitioning of versatile video coding based on supervised contrastive learning

Nowadays, Versatile Video Coding (VVC) has achieved a superior performance than previous video coding standard (High Efficiency Video Coding). The Quadtree with Nested Multi-Type Tree (QTMT) coding block structure can enhance the coding performance. Nevertheless, this technique also leads to the significantly increasing complexity of VVC inter coding. Therefore, complexity optimization is an urgent problem to be optimized in the market application of VVC. To solve this issue, we propose a Supervised-Contrastive-Learning-based Inter Partitioning (SCLIP) method in this paper. Firstly, we define the above complexity optimization problem as a supervised classification task. Next, we develop a SCLIP Estimation Network (SCLIPEst-Net) with a supervised contrastive learning module and a classification module. After training on a newly established dataset, the SCLIPEst-Net can reasonably predict the mode partitioning. Finally, we propose an overall SCLIP algorithm that effectively determines the inter partitions of VVC with a low computational overhead. Experimental results indicate that our method achieves 45.14% average Time Saving (TS) with a 2.40% Bjøntegaard Delta Bit Rate (BDBR) in Random Access (RA), outperforming the benchmarks.

TVM: A Tile-based Video Management Framework

With the exponential growth of video data, there is a pressing need for efficient video analysis technology. Modern query frameworks aim to accelerate queries by reducing the frequency of calls to expensive deep neural networks, which often overlook the overhead associated with video decoding and retrieval. Furthermore, video storage frameworks optimize video retrieval through video partition or caching, often relying on prior information about the query workload. To further accelerate queries, this study introduces a novel tile-based video management framework, called TVM, which leverages the semantic information embedded in videos, without being dependent on specific query workloads. By constructing a tile-based semantic index for newly ingested videos, TVM effectively reduces the size of decoded and processed video data. To achieve this, TVM introduces an optimal index construction algorithm that utilizes cost function and pseudo-labels. Additionally, the framework proposes a query-driven tile parallel decoding algorithm and resource caching algorithms, which further expedite the retrieval of video frames. Experimental results demonstrate that TVM can significantly enhance the throughput of various query tasks, achieving a notable speedup of more than 5.6×.

A Survey on Deep Learning Technique for Video Segmentation.

Video segmentation-partitioning video frames into multiple segments or objects-plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to creating virtual background in video conferencing. Recently, with the renaissance of connectionism in computer vision, there has been an influx of deep learning based approaches for video segmentation that have delivered compelling performance. In this survey, we comprehensively review two basic lines of research - generic object segmentation (of unknown categories) in videos, and video semantic segmentation - by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. We also offer a detailed overview of representative literature on both methods and datasets. We further benchmark the reviewed methods on several well-known datasets. Finally, we point out open issues in this field, and suggest opportunities for further research. We also provide a public website to continuously track developments in this fast advancing field: https://github.com/tfzhou/VS-Survey.

Video2Flink: real-time video partitioning in Apache Flink and the cloud

Video2Flink is a distributed highly scalable video processing system for bounded (i.e., stored) or unbounded (i.e., continuous) and real-time video streams with the same efficiency. It shows how complicated video processing tasks can be expressed and executed as pipelined data flows on Apache Flink, an open-source stream processing platform. Video2Flink uses Apache Kafka to facilitate the machine-to-machine (m2m) communication between the video production and the video processing system that runs on Apache Flink. Features that make the combination of Apache Kafka and Apache Flink a desirable solution to the problem of video processing are the ease of customization, portability, scalability, and fault tolerance. The application is deployed on a Flink cluster of worker machines that run on Kubernetes in the Google Cloud Platform. The experimental results support our claims of speed showing excellent speed-up results for all tested video resolutions. The highest (i.e., more than seven times) speed-up was observed with the videos of the highest resolutions and in real time.

High-fidelity video reversible data hiding using joint spatial and temporal prediction

An efficient predictor is essential in reversible data hiding methods. This paper proposes a video reversible data hiding method, where the correlations between pixels in both spatial and temporal domains are fully considered to improve prediction accuracy. In more detail, for each pixel, both intra prediction and inter prediction are implemented and the better prediction mode is selected according to intra or inter local complexity. A double-layered video partition scheme is applied so that the motion vectors generated during inter prediction are not necessarily embedded as additional information while reversibility is guaranteed. In addition, a cover pixel selection principle based on the probability distribution of the prediction errors is proposed, which prevents the pixels with large prediction errors from being modified. Experimental results demonstrate that the joint spatial and temporal prediction scheme offers satisfactory accuracy, and the proposed video reversible data hiding method can obtain high-quality stego videos.

Video2flink: Real-Time Video Partitioning in Apache Flink and the Cloud

Video2flink: Real-Time Video Partitioning in Apache Flink and the Cloud

Key frame extraction based on global motion statistics for team-sport videos

Key frame extraction is an important manner of video summarization. It can be used to interpret video content quickly. Existing approaches first partition the entire video into video clips by shot boundary detection, and then, extract key frames by frame clustering. However, in most team-sport videos, a video clip usually includes many events, and it is difficult to extract the key frames related to all of these events accurately, because different events of a game shot can have features of similar appearance. As is well known, most events in team-sport videos are attack and defense conversions, which are related to global translation. Therefore, by using fine-grained partition based on the global motion, a shot could be further partitioned into more video clips, from which more key frames could be extracted and they are related to the events. In this study, global horizontal motion is introduced to further partition video clips into fine-grained video clips. Furthermore, global motion statistics are utilized to extract candidate key frames. Finally, the representative key frames are extracted based on the spatial–temporal consistence and hierarchical clustering, and the redundant frames are removed. A dataset called SportKF is built, which includes 25 videos of 197,878 frames in 112 min and 764 key frames from four types of sports (basketball, football, American football and field hockey). The experimental results demonstrate that the proposed scheme achieves state-of-the-art performance by introducing global motion statistics.

Shot boundary detection based on orthogonal polynomial

Shot boundary detection (SBD) is a substantial step in video content analysis, indexing, retrieval, and summarization. SBD is the process of automatically partitioning video into its basic units, known as shots, through detecting transitions between shots. The design of SBD algorithms developed from simple feature comparison to rigorous probabilistic and using of complex models. Nevertheless, accelerate the detection of transitions with higher accuracy need to be improved. Extensive research has employed orthogonal polynomial (OP) and their moments in computer vision and signal processing owing to their powerful performance in analyzing signals. A new SBD algorithm based on OP has been proposed in this paper. The Features are derived from orthogonal transform domain (moments) to detect the hard transitions in video sequences. Moments are used because of their ability to represent signal (video frame) without information redundancy. These features are the moments of smoothed and gradients of video frames. The moments are computed using a developed OP which is squared Krawtchouk-Tchebichef polynomial. These moments (smoothed and gradients) are fused to form a feature vector. Finally, the support vector machine is utilized to detect hard transitions. In addition, a comparison between the proposed algorithm and other state-of-the-art algorithms is performed to reinforce the capability of the proposed work. The proposed algorithm is examined using three well-known datasets which are TRECVID2005, TRECVID2006, and TRECVID2007. The outcomes of the comparative analysis show the superior performance of the proposed algorithm against other existing algorithms.

Supervoxel Segmentation with Voxel-Related Gaussian Mixture Model.

Extended from superpixel segmentation by adding an additional constraint on temporal consistency, supervoxel segmentation is to partition video frames into atomic segments. In this work, we propose a novel scheme for supervoxel segmentation to address the problem of new and moving objects, where the segmentation is performed on every two consecutive frames and thus each internal frame has two valid superpixel segmentations. This scheme provides coarse-grained parallel ability, and subsequent algorithms can validate their result using two segmentations that will further improve robustness. To implement this scheme, a voxel-related Gaussian mixture model (GMM) is proposed, in which each supervoxel is assumed to be distributed in a local region and represented by two Gaussian distributions that share the same color parameters to capture temporal consistency. Our algorithm has a lower complexity with respect to frame size than the traditional GMM. According to our experiments, it also outperforms the state-of-the-art in accuracy.

Action Recognition Using Hierarchical STIP Saliency and Mixed Neighborhood Features

In video action recognition, the Dollar detector has been widely used to extract Spatio-Temporal Interest Points (STIPs) from action video sequence. It generates two kinds of information: STIP position and the respond value. However, in many cases, the detector respond, which measures the strength of local motion changes, is ignored. By utilizing such information, we propose to build a Hierarchical STIP Saliency (HSS) framework to provide different types of motion information. A novel local feature named Mixed Neighborhood Feature (MNF), which integrates the similarity and position relationship between local features, is put forward, and encoded by localityconstrained linear coding. Then, by partitioning video sequence along temporal direction, a group of sub-STVs are produced, and their corresponding descriptors are obtained with a max-pooling-on-absolute-value technique. In classification stage, Locality-constrained Group Sparse Representation (LGSR) is adopted as classifier to utilize the intrinsic group information of these sub-STV features. The experiments on the KTH and UCF Sports datasets show that in contrast to the classical recognition systems published recently, our recognition system based on the HSS and MNF achieves good performance.

Video Partitioning and Secured Keyframe Extraction of MPEG Video

Video partitioning and keyframe extraction (KFE) are the key foundations of video analysis and Content based video retrieval. The use of keyframes reduces the amount of data that is necessary in video indexing and provides the outline for dealing with the video content. In the last few years, many algorithms of keyframe extraction concentrated on the original compressed video stream. This can increase computational complexity when decompression is required before video processing. Keyframe is the frame, which can be a prototype of the salient content and information of the shot. The keyframes extracted must summarize the significant features of the video in time sequence. Therefore, there is a commensurate need of an efficient and secured keyframe selection technique in an efficient CBVR system. We propose an algorithm for keyframe extraction of compressed video shots using adaptive threshold method. Extensive computation on 200 plus video clips is performed and results are accurate and satisfactory.

Empowered or Objectified? Personal Narrative and Audiovisual Aesthetics in Beyoncé’s Partition

This article examines the feminist argument that Beyoncé presents through her personal narrative in relation to the audiovisual aesthetics of the music video Partition (2013), which arguably objectifies the female in accordance with normalized perceptions of gender and sexuality. The video is characterized by striptease aesthetics and hyperembodied display, firmly grounded within the tradition of contemporary pop videos that stylize the body both visually and sonically through gloss and excess. Situated within the field of popular musicology, I approach my study through a model of audiovisual analysis. With a primary focus on the strategies of production that fetishize the body through manipulation, I provide a close reading of this video by elucidating gender politics within a mainstream pop context. I conclude that the performative power of Beyoncé’s erotic display is contingent upon both the dissemination of her persona in terms of agency and the fetishizing aspects of audiovisual production.

Fast Highlight Detection and Scoring for Broadcast Soccer Video Summarization using On-Demand Feature Extraction and Fuzzy Inference

In this paper, a fast highlight detection and scoring method is proposed using an ondemand feature extraction and a fuzzy inference system. The proposed method partitions video to highlights and analyzes their content using an on-demand feature extraction approach. Then, a score is assigned to each highlight using a Fuzzy Inference System (FIS) according to the analyzed content. The assigned score determines importance of the events occurred in the highlight. This method is useful for flexible video summarization. The proposed method for on-demand feature extraction is a heuristic model of attention control that reduces computational complexity of the algorithm greatly. Additionally, FIS offers a simple and robust solution for content analysis. Experimental results illustrate that the proposed method is fast and processes about 130 frames per second on a personal computer. In addition, objective and subjective evaluations show that the proposed method generates high quality results for highlight detection, scoring and video summarization.

Empirical Analysis of Video Partitioning Methods for Distributed HEVC Encoding

As cloud computing has emerged as a promising technique in mainstream application domains, significant attention has been paid to distributed video encoding, in which resource-intensive encoding tasks are distributed across unlimited computational resources available in the cloud environment. For distributed video encoding, the input video must be partitioned into several segments. This approach decreases the total encoding time but may suffer from quality degradation associated with a lack of information, such as the coding complexity of the previous video segment. In this paper, two well-known video partitioning methods are explored from different performance perspectives, including encoding time, bitrates, and peak signal-to-noise ratio (PSNR).

Unsupervised video categorization based on multivariate information bottleneck method

The integration of multiple features is important for action categorization and object recognition in videos, because single feature based representation hardly captures imaging variations and individual attributes. In this paper, a novel formulation named Multivariate video Information Bottleneck (MvIB) is defined. It is an extensional type of multivariate information bottleneck and can discover categories from a collection of unlabeled videos automatically. Differing from the original multivariate information bottleneck, the novel approach extracts the video categories from multiple features simultaneously, such as local static and dynamic feature, each type of feature is treated as a relevant variable. Specifically, by preserving the relevant information with respect to these feature variables maximally, the MvIB method is able to integrate various aspects of semantic information into the final video partitioning results, and thus captures the complementary information resided in multiple feature variables. Extensive experimental results on five challenging video data sets show that the proposed approach can consistently and significantly outperform other state-of-the-art unsupervised learning methods.

Temporal segmentation and keyframe selection methods for user-generated video search-based annotation

In this paper we propose a temporal segmentation and a keyframe selection method for User-Generated Video (UGV). Since UGV is rarely structured in shots and usually user’s interest are revealed through camera movements, a UGV temporal segmentation system has been proposed that generates a video partition based on a camera motion classification. Motion-related mid-level features have been suggested to feed a Hierarchical Hidden Markov Model (HHMM) that produces a user-meaningful UGV temporal segmentation. Moreover, a keyframe selection method has been proposed that picks a keyframe for fixed-content camera motion patterns such as zoom, still, or shake and a set of keyframes for varying-content translation patterns.The proposed video segmentation approach has been compared to a state-of-the-art algorithm, achieving 8% performance improvement in a segmentation-based evaluation. Furthermore, a complete search-based UGV annotation system has been developed to assess the influence of the proposed algorithms on an end-user task. To that purpose, two UGV datasets have been developed and made available online. Specifically, the relevance of the considered camera motion types has been analyzed for these two datasets, and some guidelines are given to achieve the desired performance-complexity tradeoff. The keyframe selection algorithm for varying-content translation patterns has also been assessed, revealing a notable contribution to the performance of the global UGV annotation system. Finally, it has been shown that the UGV segmentation algorithm also produces improved annotation results with respect to a fixed-rate keyframe selection baseline or a traditional method relying on frame-level visual features.

Rate Allocation Algorithm with Successive Refinement in Peer-to-Peer Networks

We introduce a new across-peer rate allocation algorithm with successive refinement to improve the video transmission performance in P2P networks, based on the combination of multiple description coding and network coding. Successive refinement is implemented through layered multiple description codes. The algorithm is developed to maximize the expected video quality at the receivers by partitioning video bitstream into different descriptions depending on different bandwidth conditions of each peer. Adaptive rate partition adjustment is applied to ensure the real reflection of the packet drop rate in the network. Also the granularity is changed to the scale of atomic blocks instead of stream rates in prior works. Through simulation results we show that the algorithm outperforms prior algorithms in terms of video playback quality at the peer ends, and helps the system more adjustable to the peer dynamics.

Fuzzy segmentation of video shots using hybrid color spaces and motion information

Video segmentation can be defined as the process of partitioning video into spatio-temporal objects that are homogeneous in some feature space, with the choice of features being very important to the success of the segmentation process. Fuzzy segmentation is a semi-automatic region-growing segmentation algorithm that assigns to each element in an image a grade of membership in an object. In this paper, we propose an extension of the multi-object fuzzy segmentation algorithm to segment pre-acquired color video shots. The color features are selected from channels belonging to different color models using two different heuristics: one that uses the correlations between the color channels to maximize the amount of information used in the segmentation process, and one that chooses the color channels based on the separation of the clusters formed by the seed spels for all possible color spaces. Motion information is also incorporated into the segmentation process by making use of dense optical flow maps. We performed experiments on synthetic videos, with and without noise, as well as on some real videos. The experiments show promising results, with the segmentations of real videos produced using hybrid color spaces being more accurate than the ones produced using three other color models. We also show that our method compares favorably to a state-of-the art video segmentation algorithm.

Inter-Layer FEC Aided Unequal Error Protection for Multilayer Video Transmission in Mobile TV

Layered video coding creates multiple layers of unequal importance that enables us to progressively refine the reconstructed video quality. When the base layer (BL) is corrupted or lost during transmission, the enhancement layers (ELs) must be dropped, regardless of whether they are perfectly decoded or not, which implies that the transmission power assigned to the ELs is wasted. In this treatise, we propose an inter-layer forward error correction (FEC) coded video transmission scheme for mobile TV. At the transmitter, the proposed interlayer (IL) coding technique implants the systematic information of the BL into the ELs by using exclusive-OR operations. At the receiver, the implanted bits of the ELs may be utilized for assisting in decoding the BL. Furthermore, the data partition mode of H.264 video coding is utilized as the source encoder, where the type B and type C partitions will assist in protecting the type A partition. The IL coded bitstream will then be modulated and transmitted over a multifunctional multiple input multiple output (MF-MIMO) scheme for the sake of improving the system's performance in mobile environments. The proposed system may be readily combined with the traditional unequal error protection (UEP) technique, where extrinsic mutual information (MI) measurements are used for characterizing the performance of our proposed technique. Finally, our simulation results show that the proposed system model outperforms the traditional UEP aided system by about 2.5 dB of E <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">b</sub> /N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> or 3.4 dB of peak signal-to-noise ratio (PSNR) at the cost of a 21% complexity increase, when employing a recursive systematic convolutional code. Furthermore, unlike the traditional UEP strategies, where typically stronger FEC-protection is assigned to the more important layer, employing our proposed IL coding technique requires weaker FEC to the more important layer. For example, the system relying on channel coding rates of 0.85, 0.44, and 0.44 for the type A, type B, and type C H.264 video partitions, respectively, achieves the best system performance when employing a recursive systematic convolutional (RSC) code.

Highly-Demanded 비디오 서비스를 위한 낮은 복잡도의 혼합 멀티캐스트/브로드캐스트 알고리즘

광대역 네트워크 기술의 발전함에 따라 점점 더 많은 클라이언트들이 여러 가지 VoD (Video on Demand) 서비스를 사용할 수 있게 되었다. 많은 클라이언트들을 서포트하기 위하여, VoD 전송 방식을 설계할 때 아래의 몇가지 요소들을 고려하여야 한다. 즉 사용자 대기 시간 (Viewer's Waiting Time), 매 클라이언트에서 요구하는 최대 버퍼 요구량 (Buffer Requirement at Each Client ), 비디오 전송에 요구되는 채널개수(number of channel required for video delivery)와 비디오 분할 복잡도(video segmentation complexity) 등이다. 최근의 여러 가지 VoD 서비스들 중에서 Polyharmonic과 Staircase전송 방식이 사용자 대기 시간과 매 클라이언트에서 요구하는 최대 버퍼 요구량에서 제일 좋은 성능을 보여주고 있다. 그러나 이런 방식들은 하나의 비디오를 너무 많은 세그먼트들로 나누어야 하는데 이런 방식은 동시에 관리하고 사용하는 태널의 개수가 많아지게 한다. 이런 문제들을 해결하기 위하여 이 논문에서는 시스템 복잡도를 낮추면서 사용자 대기시간과 최대 버퍼 요구량의 성능을 동시에 향상시키기 위한 방법으로 Polyharmonic과 Staircase 모델을 헤드 파트에 적용하고 기존의 VoD 전송 방식들 가운데 가장 간단한 모델인 Staggered을 뒤 파트에 적용하여 Polyharmonic-Staircase-Staggered (PSS) 방식을 제안하고 있다. 이 방식은 간단하고 효율적이다. 수학적분석을 통해 사용자 대기시간은 기존의 Harmonic Broadcasting 방식과 비교했을 때 대역폭을 조금만 더 크게 차지하면 거의 동일한 성능을 나타내고 있고 최대 버퍼 요구량은 비디오 분할 계수의 조절에 의해 Harmonic Broadcasting보다 60% 우수한 성능을 나타냄을 알 수 있다. 제일 중요한 것은 제안된 방식은 제안된 방식은 실제 용용에서 중요한 요소인 비디오 서브세그먼트의 수, 동시에 관리하는 채널의 수, 동시에 사용하는 채널의 수도 크게 감소시켰다는 것이다. 또한 다양한 환경에 따라 비디오 분할 계수를 어떻게 적절히 조정하는지도 서술하고 있다. With the deployment of broadband networking technology, many clients are enabled to receive various Video on Demand (VoD) services. To support many clients, the network should be designed by considering the following factors: viewer's waiting time, buffer requirement at each client, number of channel required for video delivery, and video segmentation complexity. Among the currently available VoD service approaches, the Polyharmonic and Staircase broadcasting approaches show best performance with respect to each viewer's waiting time and buffer requirement, respectively. However, these approaches have the problem of dividing a video into too many segments, which causes very many channels to be managed and used at a time. To overcome this problem, we propose Polyharmonic-Staircase-Staggered (PSS) broadcasting approach that uses the Polyharmonic and Staircase approaches for the head part transmission and the Staggered approach for the tail part transmission. It is simple and bandwidth efficient. The numerical results demonstrate that our approach shows viewer's waiting time is comparable to that in the Harmonic approach with a slight increase in the bandwidth requirement, and saves the buffer requirement by about 60\% compared to the Harmonic broadcasting approach by simply adjusting the video partitioning coefficient factor. More importantly, our approach shows the best performance in terms of the number of segments and the number of channels managed and used simultaneously, which is a critical factor in real operation of VoD services. Lastly, we present how to configure the system adaptively according to the video partitioning coefficient.