Real-time Video Processing Applications Research Articles

A Fast Gradient Iterative Affine Motion Estimation Algorithm Based on Edge Detection for Versatile Video Coding

In the Versatile Video Coding (VVC) standard, affine motion models have been applied to enhance the resolution of complex motion patterns. However, due to the high computational complexity involved in affine motion estimation, real-time video processing applications face significant challenges. This paper focuses on optimizing affine motion estimation algorithms in the VVC environment and proposes a fast gradient iterative algorithm based on edge detection for efficient computation. Firstly, we establish judging conditions during the construction of affine motion candidate lists to streamline the redundant judging process. Secondly, we employ the Canny edge detection method for gradient assessment in the affine motion estimation process, thereby enhancing the iteration speed of affine motion vectors. The experimentalresults show that the encoding time of the affine motion estimation algorithm is about 15–35% lower than the overall encoding time of the anchor algorithm encoder, the average encoding time of the affine motion estimation part of the inter-frame prediction part is reduced by 24.79%, and the peak signal-to-noise ratio (PSNR) is only reduced by 0.04.

Object Detection Using Adaptive Block Partition and RCNN Algorithm

Advancements in image and video processing are growing over the years for industrial robots, autonomous vehicles, cryptography, surveillance, medical imaging and computer-human interaction applications. One of the major challenges in real-time image and video processing is the execution of complex functions and high computational tasks. To overcome this issue, a hardware acceleration of different filter algorithms for both image and video processing is implemented on Xilinx Zynq®-7000 System on-Chip (SoC) device consists of Dual-core Cortex™-A9 processors which provides computing ability to perform with the help of software libraries using Vivado® High-Level Synthesis (HLS). The acceleration of object detection algorithms include Sobel-Feldman filter, posterize and threshold filter algorithms implemented with 1920 x 1080 image resolutions for real-time object detection. The implementation results exhibit effective resource utilization such as 45.6% of logic cells, 51% of Look-up tables (LUTs), 29.47% of Flipflops, 15% of Block RAMs and 23.63% of DSP slices under 100 MHz frequency on comparing with previous works. There are a few reasons why tracking is preferable over detecting objects in each frame. Tracking facilitates in identifying the identity of various items across frames when there are several objects. Object detection may fail in some instances, but tracking may still be achievable which takes into account the location and appearance of the object in the previous frame. The key hurdles in real-time image and video processing applications are object tracking and motion detection. Some tracking algorithms are extremely fast because they perform a local search rather than a global search. Tracking algorithms such as meanshift, Regional Neural Network probabilistic data association, particle filter, nearest neighbor, Kalman filter and interactive multiple model (IMM) are available to estimate and predict the state of a system.

Hardware Acceleration of Video Edge Detection with Hight Level Synthesis on the Xilinx Zynq Platform

The study conducted in the current paper consists of validating an original design flow for the rapid prototyping of real-time image and video processing applications on FPGAs. A video application for edge detection with Simulink HDL coder and Vivado High-Level Synthesis (HLS) has been designed as if the code was going to be executed on a conventional processor. The developed tools will automatically translate the code into VHDL hardware language using an advanced compilation technique. This amounts to embedding processors on Xilinx Zynq-7000 System on-Chip (SoC) device in an optimal manner. This automated hardware design flow reduces the time to create a prototype since only the high-level description is required. The design of the video edge detection system is implemented on Xilinx Zynq-7000 platform. The result of the implementation gave effective resource utilization and a good frame rate (95 FPS) under 170MHz frequency.

FPGA implementation of multi-dimensional Kalman filter for object tracking and motion detection

Object tracking and motion detection are the major challenges in the real-time image and video processing applications. There are several tracking and prediction algorithms available to estimate and predict the state of a system. Kalman filter is the most widely used prediction algorithm as it is very simple, efficient and easy to implement for linear measurements. However, these types of filter algorithms are customized on hardware platforms such as Field-Programmable Gate Arrays (FPGAs) and Graphic Processing Units (GPUs) to achieve design requirements for embedded applications. In this work, a multi-dimensional Kalman filter (MDKF) algorithm is proposed for object tracking and motion detection. The numerical analysis of proposed tracking algorithm achieves competitive tracking performance in contrast with state-of-the-art tracking algorithms trained on standard benchmarks. Furthermore, MDKF is implemented on Xilinx Zynq™-7000 System-on-a chip (SoC). The implementation of MDKF on SoC performs 2× times tracking speed than that of software approach. The experimental results provide resource utilization of about 61.43% of Block RAMs (BRAMs), 90.09% of DSPs, 83.27% of Look-up tables (LUTs) and 82.35% of logic cells operating at 140 MHz with power consumption of 780 mW which outperforms previous related methods.

Keyframe extraction using Pearson correlation coefficient and color moments

Keyframe extraction plays a significant role in wide variety of real-time video processing applications such as video summarization, video management and retrieval, etc. A keyframe captures the whole content of its shot and does not contain any redundant information. The keyframe extraction algorithms are facing challenges due to different visual characteristics in videos of different categories. Therefore, a single feature is not enough to capture visual characteristics of a variety of videos. In order to tackle this problem, we propose an approach of keyframe extraction that uses hybridization of features. In the present article, we propose a novel shot detection-based keyframe extraction algorithm based on combination of two features: one is Pearson correlation coefficient (PCC) and other is color moments (CM). The linear transformation invariance property of PCC facilitates the proposed algorithm to work well under varying lighting conditions. On the other hand, the scale and rotation invariance properties of color moments are beneficial for representation of complex objects that may be present in different poses and orientations. These sustained reasons support the combination of these two features, which brings significant benefits for keyframe extraction in the proposed method. The proposed method detects shot boundaries by employing combo feature set (PCC and CM). From each shot, the frame with highest mean and standard deviation is selected as keyframe. Furthermore, another important contribution is that we developed a new dataset by collecting the videos of different categories such as movies, news, serials, animations and personal interviews and made it available online. The proposed method is experimented on three datasets: two publicly available datasets and one dataset developed by us. The performance of the proposed method on these datasets has been evaluated on the basis of different evaluation parameters: figure of merit, detection percentage, accuracy, and missing factor. Principal advantage of proposed work lies in the fact that it is capable to detect both the abrupt and gradual shot transitions. In real-time videos, it is common to have abrupt and small transitions. The experimental results show the superior performance of the proposed method over the other state-of-the-art methods.

Optimized OpenCL™ kernels for frequency domain image high-boost filters using image vectorization technique

Image high boost filtering uses high-boost filters to enhance the quality of an image, which has also seen in remote sensing, satellite broadcasting, classroom monitoring, and many more real-time video processing applications and requires its faster implementation. OpenCL is a widely adapted parallel programming framework that provides core level data parallelism, and dedicated for heterogeneous parallel devices like from low cost DSP to high-end CPU, GPU and FPGA. In this article, we have considered mostly used Ideal, Gaussian, Butterworth, and Laplacian of Gaussian frequency domain high-boost filters and implemented channelized OpenCL kernels for their rapid execution. In addition to that, these kernels are modified using image vectorization technique to optimize their time utilization by reducing the execution time of these OpenCL kernels to half. At last, performance analysis is carried out for these two types of OpenCL kernel implementations to determine their effectiveness with regard to time consumption and accuracy. Here, different image performance evaluation metrics like entropy, standard deviation, mean absolute error, percentage fit error, SSIM, correlation, and peak signal to noise ratio are applied to measure rightness of the above high-boost filters. From the results, we have concluded that a vectorized Butterworth high-boost filter kernel is the suitable one to provide better results among those filters, which might be highly adaptable in time bound real-time applications using various embedded devices.

Fast and Resource-Efficient Hardware Implementation of Modified Line Segment Detector

Lines are significant features enclosing high-level information in an image. The line segment Detector (LSD) Algorithm with low error rate is a widely used method to extract lines in images effectively and accurately. However, the algorithm on PC performs too costly both in time and resources for the real-time video processing. This paper provides a fast and resource-efficient hardware implementation solution for a modified LSD algorithm on Field Programmable Gate Arrays (FPGA) for real-time line detection. The task-level pipeline structures are exploited fully in a stream process mapped to the hardware architecture free of frame buffer. Our proposed hardware implementation processes in a stream-in–stream-out manner with little consumption of the on-chip block RAM to store intermediate values. We first employ hardware Gaussian filter and adjust Canny edge detection to obtain an edge map at single-pixel width. Then, a novel structure of region growing model based on dynamic rooted tree is used to detect line segment regions accurately with a latency of only a few rows of pixels. The low cost in time, on-chip resources, and power consumption makes our proposed algorithm suitable for portable real-time streaming video processing applications using line segment features, such as Lane departure warning systems. It can also be applied in real-time machine vision systems that use line segments information for further recognition or stereo correspondence and many others. The proposed algorithm is synthesized and tested on XC7Z020 FPGA with high reliability, accuracy speed, and low cost in both resources and energy.

Modeling and Analysis of FPGA Accelerators for Real-Time Streaming Video Processing in the Healthcare Domain

Complex real-time video processing applications with strict throughput constraints are commonly found in a typical healthcare application. The video processing chain is implemented as Field-Programmable Gate Array (FPGA) accelerators (processing blocks) communicating through a number of First-In First-Out (FIFO) buffers. The FIFO buffers are made out of Block RAM (BRAM) and limited in availability. Therefore, a key design question is the optimal sizes of the FIFO buffers with respect to the throughput constraint. In this paper, we use model-driven analysis and detailed hardware level simulation to address the question of buffer dimensioning in an efficient way. Using a Cyclo-Static Dataflow (CSDF) model and an optimization method, we identify and optimize the FIFO buffers. The results are confirmed using a detailed hardware level simulation and validated by comparison with VHDL simulations. The technique is illustrated on a use case from Philips Healthcare Image Guided Therapy (IGT) on the imaging pipeline of an Interventional X-Ray (i XR) system.

VLSI implementation of anisotropic probabilistic neural network for real-time image scaling

This study proposes an VLSI implementation of anisotropic probabilistic neural network (APNN) for real-time video processing applications. The APNN interpolation method achieves good sharpness enhancement at edge regions and reveals the noise reduction at smooth region. For real-time applications, the APNN interpolation is further implemented with efficient pipelined very-large-scale integration (VLSI) architecture. The VLSI architecture of APNN has a five-layer structure, which is comprised of Euclidian layer, Gaussian layer, weighting layer, summation layer, and division layer. The VLSI implementation outperforms software with the low-loss quality. The experimental results indicate that the performance of VLSI implementation is competent for image interpolation. The presented VLSI implementation of APNN interpolation method can reach $$1920\times 1080$$ at 30 frames per second (FPS) with a reasonable hardware cost.

A High Speed Architecture for Lifting-based 2-D Cohen-Daubechies-Feauveau (5,3) Discrete Wavelet Transform used in JPEG2000

For real-time applications, efficient VLSI implementation of DWT is desired. In this paper, DWT architecture based on retiming for pipelining and unfolding is presented. The architecture is based on lifting one-dimensional Cohen-Daubechies-Feauveau (CDF) (5,3) wavelet filter, which is easily extended to 2-D implementation. It consists of low complexity and easily repeatable components. This paper is focused on the critical path minimization and throughput optimization at the same time. The architecture has been implemented on Virtex 6 Xilinx FPGA platform. The implementation results show that the critical path is minimized four to five times, while throughput is doubled, making the overall architecture approximately ten times faster when compared with the conventional lifting-based DWT architecture. Further with parallel implementation, the throughput has doubled without any increase in number of row buffers, implying that the architecture is memory efficient as well. The even and odd rows of the image are scanned in parallel fashion. To perform the 2-D DWT transform of an image of size 15 Megapixels, it takes 16.86 ms, which implies 59 images of that size can be processed in one second. This can be utilized for real-time video processing applications even for high resolution videos.

Algorithm and VLSI Architecture of Edge-Directed Image Upscaling for 4k Display System

High-quality and cost-efficient image upscaling design is very important for many real-time video processing applications, especially when the display panel resolution reaches ultrahigh definition. Compared with New Edge-Directed Interpolation (NEDI) based implicit edge directional upscaling, explicit methods require less computational resource and more easily reach real-time performance, especially when the required image definition and upscaling ratio are very high. Nevertheless, the investigation of applications of explicit methods in video processing systems remains largely missing arguably because it is commonly believed that explicit edge-directed interpolation tends to introduce unexpected artifacts because of inaccurate detection and hence its image quality is relatively poor. This paper proposes an explicit edge-directed adaptive interpolation method that leverages more sophisticated edge detection and orientation estimation algorithms to avoid misinterpolation, thereby providing similar or even better image quality than those with implicit methods. Targeting the real-time 4K video display system, the proposed edge-directed image upscaling algorithm is further implemented with an efficient very-large-scale integration (VLSI) architecture. The experimental results demonstrate that the proposed interpolation algorithm outperforms previous explicit and implicit edge-directed methods in both objective and subjective tests. The presented VLSI implementation further demonstrates that the maximum output video sequence of the proposed interpolation method can reach 4k × 2k@60 Hz with a reasonable hardware cost.

Semantic annotation for complex video street views based on 2D–3D multi-feature fusion and aggregated boosting decision forests

Accurate and efficient semantic annotation is an important but difficult step in large-scale video interpretation. This paper presents a novel framework based on 2D–3D multi-feature fusion and aggregated boosting decision forest (ABDF) for semantic annotation of video street views. We first integrate the 3D and 2D features to define the appearance model for characterizing the different types of superpixels and the similarities between two adjacent superpixel blocks. We then propose the ABDF algorithm to build the weak classifier by using a modified integrated splitting strategy for decision trees. And a Markov random field is then adopted to perform global superpixel block optimization to correct the minor errors and make the boundary for semantic annotation smoother. Finally, a boosting strategy is used to aggregate the different weak decision trees into one final strong classification decision tree. The superpixel block instead of the pixel is used as the basic processing unit, thus only a small amount of features are required to build an accurate and efficient model. The experimental results demonstrate the advantages of the proposed method in terms of classification accuracy and computation efficiency over those of existing semantic segmentation methods. The proposed framework can be used in real-time video processing applications.

Hardware implementation and validation of a traffic road sign detection and identification system

Reconfigurability and parallel computing capability of field programmable gate array (FPGA) devices are highly exploited in real-time digital image and video processing applications. In this field, real-time traffic road signs detection systems present a huge interest since they help to assist drivers and decrease accidents. In this paper, we propose an FPGA-based hardware implementation of road signs detection and identification system. The proposed system can achieve real-time video constraints while assuring a high-level accuracy in terms of detection rate. The performance of the system in terms of processing latency was evaluated relatively to the reaction distance, the braking distance and the vehicle speed. The evaluation results show that our system can support real-time driving conditions until the speed of 110 km/h. To prove the validity of the proposed implementation, a hardware co-simulation strategy was applied. This is based on the use of Matlab/Xilinx system generator. A comparison of the co-simulation results shows the effectiveness of the developed architecture.

Massively parallel Lucas Kanade optical flow for real-time video processing applications

This paper deals with dense optical flow estimation from the perspective of the trade-off between quality of the estimated flow and computational cost which is required by real-world applications. We propose a fast and robust local method, denoted by eFOLKI, and describe its implementation on GPU. It leads to very high performance even on large image formats such as 4 K (3,840 × 2,160) resolution. In order to assess the interest of eFOLKI, we first present a comparative study with currently available GPU codes, including local and global methods, on a large set of data with ground truth. eFOLKI appears significantly faster while providing quite accurate and highly robust estimated flows. We then show, on four real-time video processing applications based on optical flow, that eFOLKI reaches the requirements both in terms of estimated flows quality and of processing rate.

안드로이드 기반의 지능형자동차 미들웨어 오픈플랫폼 서비스 응용

최근 지능형자동차 기술은 IT와 융합되어 새로운 첨단 영상미디어 응용기술로 발전하며, 오픈소스 기반의 안드로이드를 탑제한 스마트폰 응용기술과 지능형자동차 개발이라는 새로운 패러다임으로 많은 연구개발이 이루어지고 있다. 안드로이드 기반의 지능형자동차 응용 기술은 이동수단의 한계를 넘어 다양한 멀티미디어 중심의 집합기술로 진화하고 있으며, 이러한 분산 환경에서 개발되어 있는 각각의 멀티미디어 플랫폼서비스와 애플리케이션이 개발되고 있음에 따라 다양한 서비스 기술을 통한 휴대 단말장치기술이 절대적으로 요구되고 있다. 본 논문에서는 안드로이드 기반의 지능형 자동차 전용 미들웨어 설계를 통하여 단일형 시스템 스펙에 맞춰 SVC Codec과 실시간 동영상, 그래픽처리등 지능형자동차 미들웨어 SoC 설계를 통해 응용 연구를 실험 하였으며, 오픈플랫폼에서 제공되는 각종 단말서비스 기능들을 SoC 기반으로 새롭게 설계하고 표준화된 인터페이스 분석기법을 본 연구에서 실험을 통하여 입증하였다. Intelligent automobile technology and IT convergence, the development of new imaging technology media applications based on open source Android installed on tracked, wheeled smart phone application technology and the development of intelligent vehicles as a new paradigm a lot of research and development being made. Android-based intelligent automotive applications, technology, and evolved into the center of a set of various multimedia technologies move beyond the limits of the means of each of multimedia platforms, services and applications that have been developed in such a distributed environment, has been developed according to a variety of services through technology mobile terminal device technology is an absolute requirement. In this paper, SVC Codec, real-time video and graphics processing and SoC design intelligent vehicles middleware applications with monolithic system specification through Android-based design of intelligent vehicles dedicated middleware research experiments on open platforms, and provides various terminal services functions SoC based on a newly designed and standardized interface analysis techniques in this study were verified through experiments.

Software architecture for time-constrained machine vision applications

Real-time image and video processing applications require skilled architects, and recent trends in the hardware platform make the design and implementation of these applications increasingly complex. Many frameworks and libraries have been proposed or commercialized to simplify the design and tuning of real-time image processing applications. However, they tend to lack flexibility, because they are normally oriented toward particular types of applications, or they impose specific data processing models such as the pipeline. Other issues include large memory footprints, difficulty for reuse, and inefficient execution on multicore processors. We present a novel software architecture for time-constrained machine vision applications that addresses these issues. The architecture is divided into three layers. The platform abstraction layer provides a high-level application programming interface for the rest of the architecture. The messaging layer provides a message-passing interface based on a dynamic publish/subscribe pattern. A topic-based filtering in which messages are published to topics is used to route the messages from the publishers to the subscribers interested in a particular type of message. The application layer provides a repository for reusable application modules designed for machine vision applications. These modules, which include acquisition, visualization, communication, user interface, and data processing, take advantage of the power of well-known libraries such as OpenCV, Intel IPP, or CUDA. Finally, the proposed architecture is applied to a real machine vision application: a jam detector for steel pickling lines.

To accelerate shot boundary detection by reducing detection region and scope

Video Shot Boundary Detection (SBD) is the fundamental process towards video summarization and retrieval. A fast and efficient SBD algorithm is necessary for real-time video processing applications. Extensive work has focused on accurate shot boundary detection at the expense of demanding computational costs. In this paper, we propose a fast SBD approach that reduces the computation pixel-wise and frame-wise while still giving satisfactory accuracy. The proposed approach substantially speeds up the computation through reducing both detection region and scope. Color histogram and mutual information are used together to measure the difference between frames. Corner distribution of frames is utilized to exclude most of false boundaries. We conduct extensive experiments to evaluate the proposed approach, and the results show that our approach can not only speed up SBD, but also detect shot boundaries with high accuracy in both Cut (CUT) and Gradual Transition (GT) boundaries.

Image and video processing platform for field programmable gate arrays using a high-level synthesis

In this study, an image and video processing platform (IVPP) based on field programmable gate array (FPGAs) is presented. This hardware/software co-design platform has been implemented on a Xilinx Virtex-5 FPGA using a high-level synthesis and can be used to realise and test complex algorithms for real-time image and video processing applications. The video interface blocks are done in Register Transfer Languages and can be configured using the MicroBlaze processor allowing the support of multiple video resolutions. The IVPP provides the required logic to easily plug-in the generated processing blocks without modifying the front-end (capturing video data) and the back-end (displaying processed output data). The IVPP can be a complete hardware solution for a broad range of real-time image/video processing applications including video encoding/decoding, surveillance, detection and recognition.

A novel adaptive color to grayscale conversion algorithm for digital images

A color image sent to the monochrome device must undergo the color-to-grayscale conversion process. Although the conventional color-to-grayscale conversion algorithms are widely used, their performance is questionable in certain circumstances and poor quality resultant grayscale images will be produced. This paper tackled some drawbacks of the conventional methods by introducing a novel approach namely the Adaptive Color to Grayscale (ACGS) conversion algorithm. In the proposed ACGS method, the pixels distribution analysis of input color images was performed to calculate the weight contribution of red, green, and blue components during the conversion process. The extensive experimental results demonstrated the superior qualitative and quantitative performance of the proposed ACGS method over the conventional algorithms and its feasibility in real-time video processing applications. These promising results suggest that the proposed ACGS method is suitable to be employed in the monochrome devices for pre- and post- processing of digital images. Key words: Color-to-grayscale conversion, adaptive, color image, grayscale image, image pre-processing.

Fuzzy diffusion filter with extended neighborhood

Anisotropic diffusion filters, which are motivated from heat diffusion between mediums, have become a widely used technique in the field of image processing. In the initial proposals of anisotropic diffusion filters, 4-neighborhood values with diffusivity functions are computed independently for each spatial location because of numerical approximation. However, anisotropic diffusion filters could not be used in real-time image and video processing applications because they need diffusivity parameters, which must be specified by users in every sampling period. In this study, a fuzzy adaptive diffusion filter using extended neighborhood without diffusivity functions has been developed. The fuzzy adaptive diffusion filter does not require any parameter chosen by user and therefore they could be employed in real-time applications. In the fuzzy adaptive diffusion filter, a similarity transformation by means of relation matrix and fuzzy logic is carried out. Accordingly, the similarity image, output of transformation, is directly used as a heat diffusion coefficient in the diffusion filter. Results show that the fuzzy adaptive diffusion filter is very efficient for removing noise in image while preserving edges.