Resolution Video Research Articles

LENS: Bandwidth-efficient video analytics with adaptive super resolution

Video streaming analytic pipelines are deployed widely nowadays, where in many cases videos are transmitted continuously from cameras to servers. However, video analytic accuracy highly depends on video resolution, and when bandwidth suffers only low-resolution videos can be transmitted which leads to low accuracy. The rapid development of super resolution DNNs sparks the possibility of reconstructing high-resolution videos reliably from low resolution videos, and thus boosts video analytic accuracy. However, pretrained SR models may fail to cope with dynamic video content, while introducing online learning can be efficient under this condition. In this paper, we propose a practical system, LENS, which integrates an online learning period into analytic pipelines. The experimental results show that LENS is capable of responding to video dynamics and retraining SR models for changes quickly. In summary, LENS can save bandwidth consumption by up to 87%, or achieve a higher accuracy by up to 9.6% compared with state-of-the-art methods with slight additional bandwidth.

AV1 Arithmetic Encoder Design on Open-Source EDA

With the increasing demand for video transmission through the Internet, video coding has become a key technology to allow this market's growth at a reduced cost. Moreover, with the inception of higher video resolutions (e.g., 4K, 8K) and their impact on video size, new video coding standards must tackle this issue to reduce video traffic demand on the global internet infrastructure. The AV1, a recently released royalties-free video coding format created by the Alliance for Open Media (AOMedia), reaches great compression rates but cannot accomplish real-time execution on software-only implementations due to its high complexity. This paper presents and analyzes AE-AV1, a high-performance 4-stage pipelined architecture to accelerate the AV1 arithmetic encoding process (part of the entropy encoder block) and make it capable of real-time execution. For the analysis, this work aims to rely on fully open-source Electronic Design Automation (EDA) tools and Package Design Kits (PDKs).

Measurement of idlers rotation speed in belt conveyors based on image data analysis for diagnostic purposes

It is a challenge for mining enterprises to diagnose the belt conveyors, which length can be tens of kilometres and the number of idlers reaches several thousand. All of them require inspection, which is infeasible to perform using monitoring sensors, such as accelerometers. The authors proposed a novel method for the rotational speed measurement of idlers based on image data analysis. The relation is assumed of the rotation speed decrease due to internal defects. The procedure of visual data processing is verified on the real conveyor idlers. The remote sensing method can be applied in mobile inspection robots such as UAVs or UGVs as well as for manual camera recordings. The accuracy of the proposed method is 0.13-0.67% error depending on the speed range, which is provided by the standard sampling rates (30-60 FPS) and video resolution (1280 x 720 px). Recommendations are formulated for method implementation in practice.

MPEG-5 LCEVC for 3.0 Next Generation Digital TV in Brazil

TV 3.0 is the next generation digital broadcasting system developed in Brazil by the SBTVD Forum. The ambition of TV 3.0 is significantly higher than that of previous generations as it targets the delivery of IP based signals for applications, such as 8K, HDR, virtual and augmented reality, video enhancement and scalability. To deliver such services, more advanced and flexible compression technologies are required. MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) is a new video coding standard which works in combination with a separate video standard (e.g., H.264/AVC [H.264/AVC], H.265/HEVC [H.265/HEVC], H.266/VVC [H.266/VVC], AV1 [AV1]) to enhance the quality of a video. In the typical scenario, the enhanced quality is provided in terms of a higher resolution video obtained by adding details coded through an enhancement layer to a lower resolution version of the same video coded through a base layer. The LCEVC format also provides the ability to signal the bit-depth of the base layer independently from that of the enhancement layer and allowing up to 14-bit depth HDR. MPEG-5 LCEVC has been selected by the SBTVD committee as part of the TV 3.0 in December 2021. In this paper we describe the proposal submitted for LCEVC in response to the SBTVD Call for Proposals (CfP) for TV 3.0.

A compact and wideband MIMO antenna for high-data-rate biomedical ingestible capsules

Due to recent advancements in complementary metal-oxide-semiconductor (CMOS) cameras, transferring high resolution images and videos are possible in wireless capsule endoscopy. High-data-rates communication is required for such data, which is possible using multiple-input-multiple-output (MIMO) antennas. In this paper, a low-sized, compact, high-data-rate, highly isolated two-element MIMO antenna with a large bandwidth has been proposed at 2.45 GHz for wireless capsule endoscopy. The geometry of the antenna (5times 4.2 times 0.12,hbox {mm}^{3}) is kept small using meandered geometry, defected ground structure, and high permittivity of the substrate. A wider bandwidth of 620 MHz (2.15–2.77 GHz) is achieved by exciting dual-modes of the antenna using defected ground structure. Furthermore, a lower mutual coupling between the antennas (30.1 dB at 2.45 GHz) is realized, despite the small edge-to-edge gap of 0.5 mm, using combination of defected ground structure and I-shaped stub. Keeping in mind of system level configuration, this antenna is simulated and measured inside a capsule device by considering effects of the other components and the device itself. The practical measurements are performed by inserting the capsule device (containing the MIMO antenna) inside minced meat. To check the safety and effectiveness of the proposed MIMO antenna, it’s specific absorption rate (SAR) and link budget are calculated and validated. In addition, the 2times 2 channel specifications are verified which shows satisfactory performance. This antenna has high channel capacity (approx 8.2 ,hbox {bps/Hz} at hbox {SNR} = 20 ,hbox {dB}) than single-input-single-output (SISO) antennas, thus, is a suitable choice for high-data-rate capsule endoscopic devices. To the best of the authors’ knowledge, this is the first implantable MIMO antenna reported so far with such lower dimension and wider bandwidth.

A new curb lane monitoring and illegal parking impact estimation approach based on queueing theory and computer vision for cameras with low resolution and low frame rate

The rapid development of the internet of things (IoT), sensing technologies, and machine learning and deep learning techniques, along with the growing variety and volume of data, have yielded new perspectives on how novel technologies can be applied to obtain new sources of curb data to achieve cost-effective curb management. This study presents a new computer vision–based data acquisition and analytics approach for curb lane monitoring and illegal parking impact assessment. The proposed “rank, detect, and quantify impacts” system consists of three main modules: 1) hotspot identification based on rankings generated by local spatial autocorrelation analysis, 2) curb lane occupancy estimation leveraging traffic cameras and computer vision techniques, and 3) illegal parking traffic impact quantification using an M/M/∞ queueing model. To demonstrate the feasibility and validity of the proposed approach, it was tested and empirically validated using field data collected from three case study sites in Midtown Manhattan, New York City (NYC)—one of the most complex urban transportation networks in the world. Different types of curb lane occupancy, including parking and bus lanes, and different frequencies of illegal parking (high, moderate, low) were investigated. The proposed method was proven to be effective even for low resolution and discontinuous video feeds obtained from publicly available traffic cameras. All three case study sites achieved good detection accuracy (86% to 96%) for parking and bus lane occupancy, and acceptable precision and recall in detecting illegal parking events. The queueing model was also proven to effectively quantify link travel time with the appearance of illegal parking events of different frequencies. The proposed “rank, detect, and quantify impacts” system is friendly for large-scale real-time implementation and is highly scalable to help evaluate the impact of other modes such as bike or mobility-on-demand (MOD) services. It can also be easily adopted by other cities to provide transportation agencies with effective data collection and innovative curb space management strategies.

Efficient high-end video data privacy preservation with integrity verification in cloud storage

Cloud data protection intends to secure data stored in the cloud. Security and integrity are still major concerns regarding the outsourced data to the cloud. This work focuses to preserve the privacy and integrity of videos stored in the cloud. A stream cipher suite involving key generation, encryption, and authentication phases is designed. The Elliptic Curve Diffie Hellman approach is used to generate asymmetric keys. HECV compressed videos are fast processed by Apache Flink and are encrypted by XSalsa20 stream cipher. An ASCON-based SHA-3 authentication mechanism is proposed to generate an authentication code for the uploaded videos to guarantee tamper-proof. The proposed method scrambles the Spatio-temporal resolution of videos with better encryption time compared to the traditional approaches. Experimental result shows that the HEVC videos are authenticated and encrypted using the proposed cipher suite is fast, safe, and resists several forms of attacks resulting in improved information sensitivity and cost benefits.

V - Channel magnification enabled by hybrid optimization algorithm: Enhancement of video super resolution

Although being a really active area of research, television super-resolution remains a difficult problem. Additionally, it is noted that the blur motion and computational crisis hinder the enhancement. As a result, the goal of this research is to present a brand-new smart SR framework for the camera shot. To create High Resolution (HR) videos, first frames in RGB format are converted to HSV and then the V-channel is enhanced. In order to create enriched video frames, a high - dimension grid with enhanced pixel intensity is then created. This paper introduces a particular progression to enable this: Motion estimation, Cubic Spline Interpolation, and Deblurring or Sharpening are the three methods. By carefully adjusting the parameters, the cubic spline interpolation is improved during operation. A brand-new hybrid technique dubbed Lion with Particle Swarm Velocity Update (LPSO-VU), which combines the principles of the Lion Algorithm (LA) and Particle Swarm Optimization (PSO) algorithms, is presented for this optimal tuning purpose. Finally, using the BRISQUE, SDME, and ESSIM metrics, the adequacy of the method is contrasted to other traditional models, and its superiority is demonstrated. Accordingly, the analysis shows that the suggested LPSO-VU model for video frame 1 is 16.6%, 25.56%, 26.2%, 26.2%, and 27.2% superior to the previous systems like PSO, GWO, WOA, ROA, MF-ROA, and LA, respectively, in terms of BRISQUE.

An Efficient Real-Time FPGA-Based ORB Feature Extraction for an UHD Video Stream for Embedded Visual SLAM

The detection and description of feature points are important components of many computer vision systems. For example, in the field of autonomous unmanned aerial vehicles (UAV), these methods form the basis of so-called Visual Odometry (VO) and Simultaneous Localisation and Mapping (SLAM) algorithms. In this paper, we present a hardware feature points detection system able to process a 4K video stream in real-time. We use the ORB algorithm—Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)—to detect and describe feature points in the images. We make numerous modifications to the original ORB algorithm (among others, we use the RS-BRIEF instead of classic R-BRIEF) to adapt it to the high video resolution, make it computationally efficient, reduce the resource utilisation and achieve lower power consumption. Our hardware implementation supports a 4 ppc (pixels per clock) format (with simple adaptation to 2 ppc, 8 ppc, and more) and real-time processing of a 4K video stream (UHD—Ultra High Definition, 3840×2160 pixels) @ 60 frames per second (150 MHz clock). We verify our system using simulations in the Vivado IDE and implement it in hardware on the ZCU 104 evaluation board with the AMD Xilinx Zynq UltraScale+ MPSoC device. The proposed design consumes only 5 watts.

Real-Time CLAHE Algorithm Implementation in SoC FPGA Device for 4K UHD Video Stream

One of the problems encountered in the field of computer vision and video data analysis is the extraction of information from low-contrast images. This problem can be addressed in several ways, including the use of histogram equalisation algorithms. In this work, a method designed for this purpose—the Contrast-Limited Adaptive Histogram Equalization (CLAHE) algorithm—is implemented in hardware. An FPGA platform is used for this purpose due to the ability to run parallel computations and very low power consumption. To enable the processing of a 4K resolution (UHD, 3840 × 2160 pixels) video stream at 60 fps (frames per second) by using the CLAHE method, it is necessary to use a vector data format and process multiple pixels simultaneously. The algorithm realised in this work can be a component of a larger vision system, such as in autonomous vehicles or drones, but it can also support the analysis of underwater, thermal, or medical images both by humans and in an automated system.

Deep camera obscura: an image restoration pipeline for pinhole photography.

Modern machine learning has enhanced the image quality for consumer and mobile photography through low-light denoising, high dynamic range (HDR) imaging, and improved demosaicing among other applications. While most of these advances have been made for normal lens-based cameras, there has been an emerging body of research for improved photography for lensless cameras using thin optics such as amplitude or phase masks, diffraction gratings, or diffusion layers. These lensless cameras are suited for size and cost-constrained applications such as tiny robotics and microscopy that prohibit the use of a large lens. However, the earliest and simplest camera design, the camera obscura or pinhole camera, has been relatively overlooked for machine learning pipelines with minimal research on enhancing pinhole camera images for everyday photography applications. In this paper, we develop an image restoration pipeline of the pinhole system to enhance the pinhole image quality through joint denoising and deblurring. Our pipeline integrates optics-based filtering and reblur losses for reconstructing high resolution still images (2600 × 1952) as well as temporal consistency for video reconstruction to enable practical exposure times (30 FPS) for high resolution video (1920 × 1080). We demonstrate high 2D image quality on real pinhole images that is on-par or slightly improved compared to other lensless cameras. This work opens up the potential of pinhole cameras to be used for photography in size-limited devices such as smartphones in the future.

Remote Photoplethysmography and Heart Rate Estimation by Dynamic Region of Interest Tracking.

Remote photoplethysmography (PPG) estimates vital signs by measuring changes in the reflected light from the human skin. Compared to traditional PPG techniques, remote PPG enables contactless measurement at a reduced cost. In this paper, we propose a novel method to extract remote PPG signals and heart rate from videos. We propose an algorithm to dynamically track regions of interest (ROIs) and combine the signals from all ROIs based on signal qualities. To maintain a stable frame rate and accuracy, we propose a dynamic down-sampling approach, which makes our system robust to the different video resolutions and user-camera distances. We also propose the strategy of adaptive measurement time to estimate HR, which can achieve comparable accuracy in HR estimation while reducing the average measurement time. To test the accuracy of the proposed system, we have collected data from 30 subjects with facial masks. Experimental results show that the proposed system can achieve 3.0 bpm mean absolute error in HR estimation.

Visual Art Design of Digital Works Guided by Big Data

With the rapid development of digital technology, the development speed of digital media is also relatively fast. Digital media technology has a great impact on people’s lifestyles and aesthetic concepts, and it also has a greater impact on visual art, creative thinking communication methods, and expression methods. In this study, the quality enhancement of digital images has been intensively studied based on the guidance of big data of eye-movement gaze points. A large amount of visual data are collected from public social resources, and the optimization research of image sensory quality is carried out in-depth using the acquired big data. Next, the region of interest (ROI) is obtained by combining the data with a two-dimensional Gaussian distribution model-fitting method, and the obtained data clustered and improved based on the K-means clustering algorithm to obtain ROI fixation points. Finally, discontinuities in the choice of sharpness in graphics and video playback are pointed out, and the final fixation data analysis is utilized. Results show that targeted optimization is very effective in improving the quality of digital images and saving space, enabling users to enjoy higher-quality visual digital images. The proposed method can be used to improve the dynamic resolution of images and videos.

Parallel multi-view HEVC for heterogeneously embedded cluster system

In this paper, we present a computer cluster with heterogeneous computing components intended to provide concurrency and parallelism with embedded processors to achieve a real-time Multi-View High-Efficiency Video Coding (MV-HEVC) encoder/decoder with a maximum resolution of 1088p. The latest MV-HEVC standard represents a significant improvement over the previous video coding standard (MVC). However, the MV-HEVC standard also has higher computational complexity. To this point, research using the MV-HEVC has had to use the Central Processing Unit (CPU) on a Personal Computer (PC) or workstation for decompression, because MV-HEVC is much more complex than High-Efficiency Video Coding (HEVC), and because decompressors need higher parallelism to decompress in real time. It is particularly difficult to encode/decode in an embedded device. Therefore, we propose a novel framework for an MV-HEVC encoder/decoder that is based on a heterogeneously distributed embedded system. To this end, we use a parallel computing method to divide the video into multiple blocks and then code the blocks independently in each sub-work node with a group of pictures and a coding tree unit level. To appropriately assign the tasks to each work node, we propose a new allocation method that makes the operation of the entire heterogeneously distributed system more efficient. Our experimental results show that, compared to the single device (3D-HTM single threading), the proposed distributed MV-HEVC decoder and encoder performance increased approximately (20.39 and 68.7) times under 20 devices (multithreading) with the CTU level of a 1088p resolution video, respectively. Further, at the proposed GOP level, the decoder and encoder performance with 20 devices (multithreading) respectively increased approximately (20.78 and 77) times for a 1088p resolution video with heterogeneously distributed computing compared to the single device (3D-HTM single threading).

Threshold adaptation and XOR accumulation algorithm for objects detection

<span>Object detection, tracking and video analysis are vital and energetic tasks for intelligent video surveillance systems and computer vision applications. Object detection based on background modelling is a major technique used in dynamically objects extraction over video streams. This paper presents the threshold adaptation and XOR accumulation (TAXA) algorithm in three systematic stages throughout video sequences. First, the continuous calculation, updating and elimination of noisy background details with hybrid statistical techniques. Second, thresholds are calculated with an effective mean and gaussian for the detection of the pixels of the objects. The third is a novel step in making decisions by using XOR-accumulation to extract pixels of the objects from the thresholds accurately. Each stage was presented with practical representations and theoretical explanations. On high resolution video which has difficult scenes and lighting conditions, the proposed algorithm was used and tested. As a result, with a precision average of 0.90% memory uses of 6.56% and the use of CPU 20% as well as time performance, the result excellent overall superior to all the major used foreground object extraction algorithms. As a conclusion, in comparison to other popular OpenCV methods the proposed TAXA algorithm has excellent detection ability.</span>

Adaptive bitrate streaming in multi-user downlink NOMA edge caching systems with imperfect SIC

Recently, mobile networks have witnessed an increasing dominance of video traffic streams to simultaneously distribute real-time contents to massive number of user devices. In this context, fifth-generation (5G) technologies and beyond expose their advanced new radio access interfaces to facilitate multi-user associations with successive interference cancellation (SIC)-based non-orthogonal multiple access (NOMA) mechanisms. However, the imperfectness of the SIC in reality may result in significant performance decrease in NOMA owing to inter-channel interference existence. Motivated by this observation, this paper studies adaptive bitrate streaming services given such a challenging transmission environment assumed. In this context, our objectives are to maximize the video bitrate (accordingly the video resolution) of the online streams while retaining the playback smoothness. The problem is transformed into a drift-plus-penalty balancing optimization, which is then resolved by an approximation algorithm. Numerical results highlight the outperformance of the proposed approach compared to existing algorithms in terms of video quality and smoothness in various system scenarios.

Implementation of Omni-D Tele-Presence Robot Using Kalman Filter and Tricon Ultrasonic Sensors

The tele-presence robot is designed to set forth an economic solution to facilitate day-to-day normal activities in almost every field. There are several solutions to design tele-presence robots, e.g., Skype and team viewer, but it is pretty inappropriate to use Skype and extra hardware. Therefore, in this article, we have presented a robust implementation of the tele-presence robot. Our proposed omnidirectional tele-presence robot consists of (i) Tricon ultrasonic sensors, (ii) Kalman filter implementation and control, and (iii) integration of our developed WebRTC-based application with the omnidirectional tele-presence robot for video transmission. We present a new algorithm to encounter the sensor noise with the least number of sensors for the estimation of Kalman filter. We have simulated the complete model of robot in Simulink and Matlab for the tough paths and critical hurdles. The robot successfully prevents the collision and reaches the destination. The mean errors for the estimation of position and velocity are 5.77% and 2.04%. To achieve efficient and reliable video transmission, the quality factors such as resolution, encoding, average delay and throughput are resolved using the WebRTC along with the integration of the communication protocols. To protect the data transmission, we have implemented the SSL protocol and installed it on the server. We tested three different cases of video resolutions (i.e., , and ) for the performance evaluation of the video transmission. For the highest resolution, our TPR takes 3.5 ms for the encoding, and the average delay is 2.70 ms with 900 × 590 pixels.

Pedestrian Reidentification Algorithm Based on Local Feature Fusion Mechanism

In the application of pedestrian reidentification, misjudgment is often caused by low video resolution, illumination variation, and background interference. In order to solve these problems, this study proposes a pedestrian reidentification algorithm based on local feature fusion. Taking advantage of the inherent structure of the human body, we pay attention to pedestrian parts with prominent features and ignore other parts with interference information. Feature extraction is carried out for detected pedestrian parts with significant features, and new fusion features are generated. By calculating distance measurement between image features, pedestrians are classified and recognized. Experimental results show that the accuracy of the proposed algorithm is superior to that of other comparison algorithms on the datasets of Market1501, Duke, and CUHK03. It is proved that the proposed algorithm has a good pedestrian reidentification effect.

Infant AFAR: Automated facial action recognition in infants

Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.

Low-Complexity Intra Coding in Versatile Video Coding

Versatile Video Coding (VVC) was finalized in 2020 and offered promising coding efficiency with a bitrate reduction of about 50% for same video quality as High Efficiency Video Coding. However, its high encoding complexity is a heavy burden on real-time applications. In particular, the very high complexity in intra coding can be a big barrier into market entry. This paper presents an efficient low-complexity intra coding scheme, which employs downsampling and upsampling processes. The downsampling is simply performed by reducing the resolution of an original video in both horizontal and vertical directions. In the upsampling, convolutional neural network based super-resolution is used to increase the resolution of the reconstructed video. In addition, this paper thoroughly analyzes the performance and complexity of all intra coding tools in VVC. Experimental results demonstrate that the significantly high reduction of the encoding complexity can be achieved with acceptable video quality.