Real-time Video Processing System Research Articles

Edge computing based real-time Nephrops (Nephrops norvegicus) catch estimation in demersal trawls using object detection models

In demersal trawl fisheries, the unavailability of the catch information until the end of the catching process is a drawback, leading to seabed impacts, bycatches and reducing the economic performance of the fisheries. The emergence of in-trawl cameras to observe catches in real-time can provide such information. This data needs to be processed in real-time to determine the catch compositions and rates, eventually improving sustainability and economic performance of the fisheries. In this study, a real-time underwater video processing system counting the Nephrops individuals entering the trawl has been developed using object detection and tracking methods on an edge device (NVIDIA Jetson AGX Orin). Seven state-of-the-art YOLO models were tested to discover the appropriate training settings and YOLO model. To achieve real-time processing and accurate counting simultaneously, four frame skipping ideas were evaluated. It has been shown that adaptive frame skipping approach, together with YOLOv8s model, can increase the processing speed up to 97.47 FPS while achieving correct count rate and F-score of 82.57% and 0.86, respectively. In conclusion, this system can improve the sustainability of the Nephrops directed trawl fishery by providing catch information in real-time.

Design of multi-channel real-time video processing system based on FPGA

Design of multi-channel real-time video processing system based on FPGA

Real-time video processing system in coal mine based on edge-cloud collaborative framework

Real-time video processing system in coal mine based on edge-cloud collaborative framework

Special issue on deep learning for emerging embedded real-time image and video processing systems

Special issue on deep learning for emerging embedded real-time image and video processing systems

An efficient and adaptable multimedia system for converting PAL to VGA in real-time video processing

Real-time video processing has found its range of applications from defense to consumer electronics for surveillance, video conferencing, etc. With the advent of Field Programmable Gate Arrays (FPGAs), flexible real-time video processing systems which can meet hard real-time constraints are easily realized with short development time. Most of the existing solutions have high utilization of system resources and are not quite flexible with many applications. Here we propose a hardware–software co-design for an FPGA-based real-time video processing system to convert video in standard Phase Alternating Line (PAL) 576i format to standard video of Video Graphics Array (VGA)/Super Video Graphics Array (SVGA) format with little utilization of resources. Switching between multiple video streams, character/text overlaying, and skin color detection are also incorporated with the system. The system is also adaptable for rugged applications. VHSIC Hardware Description Language (VHDL) codes for the architecture were synthesized using Altera Quartus II and targeted for Altera Stratix I FPGA. Results achieved confirm that the proposed system performs efficient conversion with very less resource utilization compared to the existing solutions. Since the proposed system is also flexible, many other applications can be incorporated in the future.

Quantified Soccer Using Positional Data: A Case Study.

Performance development in international soccer is undergoing a silent revolution fueled by the rapidly increasing availability of athlete quantification data and advanced analytics. Objective performance data from teams and individual players are increasingly being collected automatically during practices and more recently also in matches after FIFA's 2015 approval of wearables in electronic performance and tracking systems. Some clubs have even started collecting data from players outside of the sport arenas. Further algorithmic analysis of these data might provide vital insights for individual training personalization and injury prevention, and also provide a foundation for evidence-based decisions for team performance improvements. This paper presents our experiences from using a detailed radio-based wearable positioning data system in an elite soccer club. We demonstrate how such a system can detect and find anomalies, trends, and insights vital for individual athletic and soccer team performance development. As an example, during a normal microcycle (6 days) full backs only covered 26% of the sprint distance they covered in the next match. This indicates that practitioners must carefully consider to proximity size and physical work pattern in microcycles to better resemble match performance. We also compare and discuss the accuracy between radio waves and GPS in sampling tracking data. Finally, we present how we are extending the radio-based positional system with a novel soccer analytics annotation system, and a real-time video processing system using a video camera array. This provides a novel toolkit for modern forward-looking soccer coaches that we hope to integrate in future studies.

Superpixel-Based Causal Multisensor Video Fusion

Video surveillance systems have become extremely important recently. It has been observed that information extracted from a single spectrum video is often insufficient in adverse conditions like low illumination, shadowing, smoke, dust, unstable background, and camouflage. Real-time video processing systems also need to be very fast where future frames are often unavailable at the time of processing the current frame. In this paper, we propose a superpixel-based causal multisensor video fusion algorithm suitable for real-time surveillance tasks. We develop new superpixel level spatial and temporal saliency models. Novel superpixel level multiple fusion rules are also designed to obtain the fused output. Comprehensive comparisons with several existing works clearly indicate the benefit of our solution.

Toward the Implementation of an ASIC-Like System on FPGA for Real-Time Video Processing with Power Reduction

Driven by the importance of energy consumption in system-on-chip design as an evaluation factor, this paper presents a design methodology at the system level to optimize power consumption on ARM-based architecture for real-time video processing. The proposed design flow is based on the interaction between the tool and user optimizations. The tool optimizations are the options and best practices available on the integrated design environment for the Xilinx technology and the target Zynq-7000 architecture. The user methods present methods proposed by the user to optimize power consumption. We used the principles of voltage scaling and frequency scaling techniques for user methods. These two techniques allow energy to be consumed in the proportion of work to be done. The suggested flow is applied on real-time video processing system. The results show power savings for up to 60% with respect to performance and real-time constraints.

VLSI Architecture Exploration of Guided Image Filtering for 1080P@60Hz Video Processing

Guided image filtering (GIF) is a promising edge-preserving filtering technique that has been applied in a variety of applications. Nevertheless, an efficient very-large-scale integration (VLSI) architecture design of GIF is still very challenging for the real-time processing of full-high definition videos. Previously proposed architectures are somewhat inefficient in terms of either on-chip memory usage or off-chip memory bandwidth. This paper aims to improve the balance between on-chip memory usage and off-chip memory bandwidth through architecture exploration. Three critical architectural tradeoffs in the VLSI design of GIF are explored, and two efficient VLSI architectures, namely sequential line-based and parallel line-based architectures, are proposed. Experimental results demonstrate that the proposed VLSI design only consumes 34.1-K logic gates, 25.4-KB on-chip memories, and 373-MB/s off-chip memory bandwidth while achieving a real-time video processing of 1080P@60Hz at the maximum clock frequency of 297-MHz. Moreover, the proposed VLSI circuits are fully pipelined and synchronized to the pixel clock of output video, so can be seamlessly integrated into diverse real-time video processing systems.

Automatic behaviour analysis system for honeybees using computer vision

We present a fully automatic online video system, which is able to detect the behaviour of honeybees at the beehive entrance. Our monitoring system focuses on observing the honeybees as naturally as possible (i.e. without disturbing the honeybees). It is based on the Raspberry Pi that is a low-cost embedded computer with very limited computational resources as compared to an ordinary PC. The system succeeds in counting honeybees, identifying their position and measuring their in-and-out activity. Our algorithm uses background subtraction method to segment the images. After the segmentation stage, the methods are primarily based on statistical analysis and inference. The regression statistics (i.e. R2) of the comparisons of system predictions and manual counts are 0.987 for counting honeybees, and 0.953 and 0.888 for measuring in-activity and out-activity, respectively. The experimental results demonstrate that this system can be used as a tool to detect the behaviour of honeybees and assess their state in the beehive entrance. Besides, the result of the computation time show that the Raspberry Pi is a viable solution in such real-time video processing system.

A Rate Control Algorithm for Low Delay Video Coding

For the standard definition video, the inherent delay produced by the existing coding standards is above 260ms. The delay is too big to apply in real-time video processing system. In order to reduce codec system delay, this paper gives a low delay coding structure. Firstly, the input video is divided into a number of slices for reducing the acquisition delay, and then the slice is coded as a basic unit. Secondly, In order to solve the problem of bit allocation with low delay coding process, this paper presents a constant bit rate control algorithm based on slices; frame and macro block line rate allocation strategy is given. The experimental results show that, compared with the frame/field codec system, new slice coding structure and rate control algorithm can keep the quality of the reconstructed image and effectively reduce the inherent codec system delay. The delay of the proposed algorithm is about 100ms and it can satisfy the real-time requirement.

NEROvideo: a general-purpose CNN-UM video processing system

Emulations of cellular nonlinear networks (CNN) on digital reconfigurable hardware have proved to be adequate for highly-efficient computation of massive data, exceeding the accuracy and flexibility of full-custom designs. Based on a recently-proposed architecture for the emulation of a large-scale CNN universal machine, a new real-time video processing system has been developed. Due to its free programmability and massively-parallel architecture the system is very suitable for high-speed computation of complex algorithms that follow the idea of spatio-temporal computing. Implemented on a state-of-the-art Xilinx Zynq system-on-chip, the proposed setup is capable of processing a $$640\times 480$$640×480p video stream with up to 1,700 fps, depending on the respective algorithm.

Platform-Based Design Approach for Embedded Vision Applications

Implementation of an efficient, complex real-time image and video processing system has many different contrary design needs. These systems require managing and processing a large amount of data in a timely manner. The robust implementation requires design reuse and flexibility, which is very crucial for an ever-increasing application need. A combination of hardware and software which form a platform fulfills the demands of the continuously developing application domain. The platform-based design has recently gained in importance which allows to explore the trade-offs between various designs requirements for the hardware and software implementation of real-time image and video processing algorithms and its systems. In this paper, we present the exploration of platform-based design approach for implementing a real-time vision system. We have used the platform-based design approach to design a configurable real-time image and video streaming module for vision-based systems on an Virtex-5 FPGA. We have also implemented a real-time image and video acquisition module for a smart camera system. 

An efficient and expandable hardware implementation of multilayer cellular neural networks

This paper proposes a new CNN architecture conceived for hardware implementation of complex ML-CNNs on programmable devices. The architecture is completely modular and expandable, and includes advanced features such as non-linear templates, time-variant coefficients or multi-layer structure. We also present an implementation platform based on the pre-designed but user-configurable FPGA processing modules that inherit the modularity and expandability of the logical architecture. All the modules share the same, properly designed, I/O interface, so the platform can be configured to accommodate CNNs of any size or structure, composed of a number of processing blocks that can be physically distributed over several FPGA boards. Our Carthagonova architecture makes use of a temporal processing approach with a super-pipelined unfolded cell structure, leading to the maximum degree of parallelism while still keeping the most efficient use of FPGA resources. Both the CNN architecture and the hardware platform have been validated by the implementation of a real-time video processing system, showing that they conform a valuable set of tools for the development of CNN-based applications.

A Row-Parallel 8$\,\times\,$8 2-D DCT Architecture Using Algebraic Integer-Based Exact Computation

An algebraic integer (AI) based time-multiplexed row-parallel architecture and two final-reconstruction step (FRS) algorithms are proposed for the implementation of bivariate AI-encoded 2-D discrete cosine transform (DCT). The architecture directly realizes an error-free 2-D DCT without using FRSs between row-column transforms, leading to an 8$\times$8 2-D DCT which is entirely free of quantization errors in AI basis. As a result, the user-selectable accuracy for each of the coefficients in the FRS facilitates each of the 64 coefficients to have its precision set independently of others, avoiding the leakage of quantization noise between channels as is the case for published DCT designs. The proposed FRS uses two approaches based on (i) optimized Dempster-Macleod multipliers and (ii) expansion factor scaling. This architecture enables low-noise high-dynamic range applications in digital video processing that requires full control of the finite-precision computation of the 2-D DCT. The proposed architectures and FRS techniques are experimentally verified and validated using hardware implementations that are physically realized and verified on FPGA chip. Six designs, for 4- and 8-bit input word sizes, using the two proposed FRS schemes, have been designed, simulated, physically implemented and measured. The maximum clock rate and block-rate achieved among 8-bit input designs are 307.787 MHz and 38.47 MHz, respectively, implying a pixel rate of 8$\times$307.787$\approx$2.462 GHz if eventually embedded in a real-time video-processing system. The equivalent frame rate is about 1187.35 Hz for the image size of 1920$\times$1080. All implementations are functional on a Xilinx Virtex-6 XC6VLX240T FPGA device.

Real Time Defects of Beer Bottle Detection System Based on DSP

The paper analyses the problem of beer bottles detection techniques on the beer bottles production line, uses digital image processing technique on the beer bottles online defect detection. The paper puts forward the designing ideas of the hardware, developing flow of the software and the algorithm of beer bottles detection. TMSDM642 is used to set up the real-time video processing system of the hardware .The hardware system is mainly composed of three parts: the part of memory, the part of the input and the part of the output. When beer bottles are put into the work area, the video images of the bottle-mouth and bottle-bottom will be gained by the CCD camera, firstly, preprocessing is used to eliminate video image noise. Secondly, the image segmentation algorithm is used to detect defects in video images. Lastly the goal of extracting defects will be accomplished. The experimental result indicated that this system may effectively exam the flaw or the unqualified beer bottles.

Bit-Width Constrained Memory Hierarchy Optimization for Real-Time Video Systems

The great variety of pixel dynamics of real-time video-processing systems (RTVPS), ranging from color, grayscale, or binary pixels, means that a careful design and specification of bit widths is required. It is obvious that the bit-width specification will affect the total memory storage requirement. However, what is not so obvious is that the bit-width specification will also affect the design of the memory hierarchy, an impact similar for both hardware and software implementations. We have developed an integer-nonlinear-program formulation for the optimization of the memory hierarchy of RTVPS. An active surveillance video camera is introduced as a test case. We demonstrate how the optimization model can reduce the on-chip memory storage by 61% compared to a nonoptimal memory hierarchy

Automatic Generation of Spatial and Temporal Memory Architectures for Embedded Video Processing Systems

This paper presents a tool for automatic generation of the memory management implementation for spatial and temporal real-time video processing systems targeting field programmable gate arrays (FPGAs). The generator creates all the necessary memory and control functionality for a functional spatio-temporal video processing system. The required memory architecture is automatically optimized and mapped to the FPGAs' memory resources thus producing an efficient implementation in terms of used internal resources. The results in this paper show that the tool is able to efficiently and automatically generate all required memory management modules for both spatial and temporal real-time video processing systems.

RAM ALLOCATION ALGORITHM FOR VIDEO PROCESSING APPLICATIONS ON FPGA

This paper presents an algorithm for the allocation of on-chip FPGA Block RAMs for the implementation of Real-Time Video Processing Systems. The effectiveness of the algorithm is shown through the implementation of realistic image processing systems. The algorithm, which is based on a heuristic, seeks the most cost-effective way of allocating memory objects to the FPGA Block RAMs. The experimental results obtained, show that this algorithm generates results which are close to the theoretical optimum for most design cases.

VLSI architecture for real-time image and video processing systems

This paper discusses the challenges of the design of real-time image and video processing systems and reviews some practical design approaches for these systems.