Real-time Image Processing Research Articles

Smart Traffic Management System

Abstract: An efficient management system is imperative to address the frequent traffic congestion experienced at major intersections. This paper proposes the implementation of an intelligent traffic controller that utilizes real-time image processing. By employing various edge detection algorithms and object counting methods, the camera footage is analyzed to effectively manage the traffic flow. Previously, a matching method was employed, wherein cameras were installed alongside traffic lights to capture sequential image sequences. These images were then compared to a reference image depicting an empty road. However, in my paper, a filtering method is utilized instead. This method effectively filters out unwanted objects, focusing solely on vehicles, and accurately determines the number of cars present in each image.

An investigation of camera movements and capture techniques on optical flow for real-time rendering and presentation

New and interesting uses for portable devices include the creation and viewing of 3D models and 360-degree photos of real landscapes. To provide a 3D model and a 360-degree view of a scenario, these apps search for real-time rendering and presentation. This study examines the impact of real-time image processing on movements in camera view and the application of optical flow algorithms. The optical flows are affected by how the objects involved motion. The way the image is recorded and the camera movements affect the relative motion. As a consequence, camera motions are responsive to optical flow algorithms. To record an image, the camera may pan around, move in a straight path, or move randomly. We have captured datasets of videos produced in different contexts to better replicate real-world scenarios. Each dataset was captured in a variety of illumination situations, camera movements, and indoor and outdoor recording sites. Here, to determine the most effective optical flow algorithms for use in near real-time applications, such as augmented reality and virtual reality, we compare results based on quality and processing delay for each video frame. We conducted a comparison study to better understand how random camera motion affects real-time video processing. These methods can be used to handle a variety of real-world issues, such as object tracking, video segmentation, structure from motion, gesture tracking, and so on.

Real-time approximate and combined 2D convolvers for FPGA-based image processing

Convolution widely has been used as the main part of the improvement in digital image processing applications. In convolutional computations, a large number of memory accesses and a huge amount of computations challenge its performance. Many of the related proposed convolvers are based on exact computations. Although exact convolvers keep the accuracy of the convolution operation at the top level, sometimes by missing a negligible amount of accuracy, the performance can be improved. Approximate computing is a new technique for solving computation overhead problems. In this paper, approximate 2D convolvers are presented which minimize the memory access rate and computations by a special factor of multiply-and-accumulate (MAC) terms. On the other hand, to preserve the flexibility for supporting different required accuracy, the proposed approximate convolvers are combined with the exact designs with real-time pre-processing stages by exploiting innovative methods which manage the hardware overhead. In comparison with conventional convolvers, the proposed designs improve the number of active resources which causes a significant reduction in power consumption. For 3 × 3 kernel size, the evaluation results on the Xilinx Virtex-7 (XC7V2000t) FPGA device show 34% and 20% power optimization of the proposed approximate and combined convolvers, respectively, in comparison with exact convolver (EC). Also, this improvement grows by increasing the kernel size. Finally, a comparison based on RMSE and PSNR for different sample images and filters reveals that the error rate and image quality reduction are acceptable for many real-time image processing applications.

Application of real-time image processing for monitoring bacterial cellulose growth in various nitrogen sources using soybean-boiled wastewater medium during fermentation

One of the utilizations of liquid waste of soybean boiling was a combination medium of bacterial cellulose (BC) fermentation. Recent study was found the implementation of image processing approach on observation of BC growth. This work utilized the image processing approach with a soybean-boiled wastewater medium using ammonium sulphate and yeast extract as nitrogen source. This study aimed to find out the highest correlation between variables that affect the fermentation process and to examine the kinetic model of the BC thickness during fermentation stage using Real-Time Image Processing (RTIP) approach with the variation of nitrogen sources. The outcome demonstrates that the correlation between time and thickness was found as the highest correlation variable during fermentation on both mediums. The Gompertz model equation was followed in the kinetic observation of BC growth on both medium and showed that the first growth of BC on ammonium sulphate was earlier than on yeast extract as nitrogen source. Further studies are also needed to understand the ability of image processing approach to detect potential failures in BC growth.

Monitoring bacterial cellulose growth during fermentation with various carbon sources by applying real-time image processing

Nata de coco is a fermented product made from coconut water. Acetobacter xylinum is used in coconut water fermentation to produce very fine cellulose fibers. This study aims to observe the bacterial cellulose (BC) formation with various carbon sources using the Real-Time Image Processing approach. Fermentation is stopped when the BC reaches a certain thickness. A set of tools have been modified to perform fermentation, such as using Raspberry Pi, Arduino sensors, USB Web Camera, and building a BC growth monitoring database that is integrated with PHP. The results showed the highest correlation between time and thickness for glucose, sucrose and mannitol mediums. Cluster analysis showed that glucose had three clusters, while sucrose and mannitol had two groups. The kinetic observations of BC growth on glucose, sucrose, and mannitol mediums follow the Gompertz model equation. The conclusions obtained are that glucose had the fastest adaptation phase, followed by sucrose, and mannitol. Mannitol was referred to as the carbon sources that had the longest BC formation.

Automatic Inhomogeneous Background Correction for Spatial Target Detection Image Based on Partition Processing

High-resolution imaging with wide field of view (FoV) ground-based telescopes is often affected by skylight background and noise due to the detector, resulting in an inhomogeneous background. In this paper, we propose an improved method for spatial image non-uniformity correction based on partition processing. First, an evaluation metric is introduced to evaluate the partition size and automatically iterate a suitable partition value for different scenarios based on the different operating conditions of the telescope. Then, we iteratively calculate the mean and variance in each partitioned region to extract the background of each partitioned region. Finally, after applying bilinear interpolation to the background extracted from each region, the inhomogeneous background is obtained and removed from the original image. The experiments on the simulated and real images show that the proposed method can effectively remove the inhomogeneous background of spatial images and meet the requirements of the real-time processing of high-resolution images under long exposure conditions.

Real-Time Image Processing Review Paper: Methods, Techniques, and Applications

Abstract—In modern days, Image processing is now of utmost importance, especially in real-time scenarios where errors in contemporaneous image processing could have disastrous results; the research and analysis methods of real- time image processing istherefore of extraordinary importance. Real-time image processing research and analysis methodologies are consequently of the utmost significance. This article's main aim is to provide an overview of the currentstate of real-time image processing research (Applications), useful methods, and practice. Keywords— Processing images in real time, Applications of image processing, image processing techniques, and image enhancement methods.

FPGA-Based In-Vivo Calcium Image Decoding for Closed-Loop Feedback Applications.

Miniaturized calcium imaging is an emerging neural recording technique that has been widely used for monitoring neural activity on a large scale at a specific brain region of rats or mice. Most existing calcium-image analysis pipelines operate offline. This results in long processing latency, making it difficult to realize closed-loop feedback stimulation for brain research. In recent work, we have proposed an FPGA-based real-time calcium image processing pipeline for closed-loop feedback applications. It can perform real-time calcium image motion correction, enhancement, fast trace extraction, and real-time decoding from extracted traces. Here, we extend this work by proposing a variety of neural network based methods for real-time decoding and evaluate the tradeoff among these decoding methods and accelerator designs. We introduce the implementation of the neural network based decoders on the FPGA, and show their speedup against the implementation on the ARM processor. Our FPGA implementation enables the real-time calcium image decoding with sub-ms processing latency for closed-loop feedback applications.

P‐12.1: Two‐dimensional Edge Detection Implementation of Liquid Crystal Computing Metasurfaces

Computing metasurfaces have received widespread attention since they were proposed, and the ability to realize mathematical operations has enabled real time image processing with new breakthroughs, such as optical edge detection. In this paper, a new method for two-dimensional edge detection is presented, and the two-dimensional liquid crystal polarization grating is designed from the theoretical derivation. The gratings with different periods are fabricated by photoalignment technology. The one-dimensional and two-dimensional imaged edge detection are experimental realized in this work. By measuring the displacement of edge detection, the experimental results with different periods are in line with theoretical formula. The low cost and easily fabricated photoaligned liquid crystal polarization gratings achieved the real time optical computation with broadband and adjustable, which can be further applied in biological image edge and other optical display related fields.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy.

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Time-resolved determination of velocity fields on the example of a simple water channel flow, using a real-time capable FPGA-based camera platform

Field Programmable Gate Arrays (FPGAs) have become a widely available computational resource for real-time image processing and can be used in time-critical optical process measurements. In this study, we investigate the applicability of the spatial filtering technique for time-resolved velocity field determination, on the example of a simple water tunnel flow. For this, the spatial filtering technique was implemented for a custom FPGA platform and analyzed regarding the effect of spatial and temporal resolution on the accuracy of the results. It was possible to prove the algorithm as suitable for velocity field estimation and identify algorithm parameters, which allow for increasing the accuracy of the results.Graphical abstract

Effects of image-sharpening algorithm on surgical field visibility during 3D heads-up surgery for vitreoretinal diseases

We conducted clinical and experimental studies to investigate the effects of image-sharpening algorithms and color adjustments, which enabled real-time processing of live surgical images with a delay of 0.004 s. The images were processed with image-sharpening intensities of 0%, 12.5%, 25%, and 50% during cataract surgery, vitrectomy, peeling of epiretinal membrane, and peeling of internal limiting membrane (ILM) with the Ngenuity 3D visualization system. In addition, the images obtained with a yellow filter during the ILM peeling were processed with color adjustments. Five vitreoretinal surgeons scored the clarity of the images on a 10-point scale. The images of a 1951 United States Air Force grating target placed in no fluid (control), saline, and 0.1% and 1% milk solution were evaluated. The results showed that the mean visibility score increased significantly from 5.0 ± 0.6 at 0% to 6.4 ± 0.6 at 12.5%, 7.3 ± 0.7 at 25%, and 7.5 ± 0.9 at 50% (P < 0.001). The visibility scores during ILM peeling improved significantly with color adjustments (P = 0.005). In the experimental study, the contrast of the grating targets blurred by the 0.1% and 1% milk solution increased significantly by the image-sharpening procedure. We conclude that the image-sharpening algorithms and color adjustments improved the intraoperative visibility of 3D heads-up surgery.

Integrating EDICAM into the MARTe framework

At the Centre for Energy Research a process has started to integrate the Event Detection Intelligent Camera (EDICAM) and its software API into the Multithreaded Application Real-time executor (MARTe) real-time framework. This framework is developed for and widely used in the fusion community (e.g. JET, MAST, COMPASS, ISTTOK etc.) and it is compatible with EPICS, which is the control system e.g. on what ITER's own control system will be based on.Throughout this contribution we integrated the basic real-time image processing functionality into MARTe by implementing a General Application Module (GAM) for EDICAM data processing, a Data Source for the offline testing of new and improved algorithms (using HDF5) and preparations made for implementing a Data Source for the live, real-time operation (using the raw camera data), which otherwise uses the same software stack. The implementation and performance measure of the aforementioned components is discussed.

Fast underwater image enhancement based on a generative adversarial framework

Underwater image enhancement is a fundamental requirement in the field of underwater vision. Along with the development of deep learning, underwater image enhancement has made remarkable progress. However, most deep learning-based enhancement methods are computationally expensive, restricting their application in real-time large-size underwater image processing. Furthermore, GAN-based methods tend to generate spatially inconsistent styles that decrease the enhanced image quality. We propose a novel efficiency model, FSpiral-GAN, based on a generative adversarial framework for large-size underwater image enhancement to solve these problems. We design our model with equal upsampling blocks (EUBs), equal downsampling blocks (EDBs) and lightweight residual channel attention blocks (RCABs), effectively simplifying the network structure and solving the spatial inconsistency problem. Enhancement experiments on many real underwater datasets demonstrate our model's advanced performance and improved efficiency.

Butterfly network: a convolutional neural network with a new architecture for multi-scale semantic segmentation of pedestrians.

The detection of multi-scale pedestrians is one of the challenging tasks in pedestrian detection applications. Moreover, the task of small-scale pedestrian detection, i.e., accurate localization of pedestrians as low-scale target objects, can help solve the issue of occluded pedestrian detection as well. In this paper, we present a fully convolutional neural network with a new architecture and an innovative, fully detailed supervision for semantic segmentation of pedestrians. The proposed network has been named butterfly network (BF-Net) because of its architecture analogous to a butterfly. The proposed BF-Net preserves the ability of simplicity so that it can process static images with a real-time image processing rate. The sub-path blocks embedded in the architecture of the proposed BF-Net provides a higher accuracy for detecting multi-scale objective targets including the small ones. The other advantage of the proposed architecture is replacing common batch normalization with conditional one. In conclusion, the experimental results of the proposed method demonstrate that the proposed network outperform the other state-of-the-art networks such as U-Net + + , U-Net3 + , Mask-RCNN, and Deeplabv3 + for the semantic segmentation of the pedestrians.

A hardware system for real-time decoding of in vivo calcium imaging data.

Epifluorescence miniature microscopes ('miniscopes') are widely used for in vivo calcium imaging of neural population activity. Imaging data are typically collected during a behavioral task and stored for later offline analysis, but emerging techniques for online imaging can support novel closed-loop experiments in which neural population activity is decoded in real time to trigger neurostimulation or sensory feedback. To achieve short feedback latencies, online imaging systems must be optimally designed to maximize computational speed and efficiency while minimizing errors in population decoding. Here we introduce DeCalciOn, an open-source device for real-time imaging and population decoding of in vivo calcium signals that is hardware compatible with all miniscopes that use the UCLA Data Acquisition (DAQ) interface. DeCalciOn performs online motion stabilization, neural enhancement, calcium trace extraction, and decoding of up to 1024 traces per frame at latencies of <50 ms after fluorescence photons arrive at the miniscope image sensor. We show that DeCalciOn can accurately decode the position of rats (n = 12) running on a linear track from calcium fluorescence in the hippocampal CA1 layer, and can categorically classify behaviors performed by rats (n = 2) during an instrumental task from calcium fluorescence in orbitofrontal cortex. DeCalciOn achieves high decoding accuracy at short latencies using innovations such as field-programmable gate array hardware for real-time image processing and contour-free methods to efficiently extract calcium traces from sensor images. In summary, our system offers an affordable plug-and-play solution for real-time calcium imaging experiments in behaving animals.

An IoT-based smart mosquito trap system embedded with real-time mosquito image processing by neural networks for mosquito surveillance.

An essential aspect of controlling and preventing mosquito-borne diseases is to reduce mosquitoes that carry viruses. We designed a smart mosquito trap system to reduce the density of mosquito vectors and the spread of mosquito-borne diseases. This smart trap uses computer vision technology and deep learning networks to identify features of live Aedes aegypti and Culex quinquefasciatus in real-time. A unique mechanical design based on the rotation concept is also proposed and implemented to capture specific living mosquitoes into the corresponding chambers successfully. Moreover, this system is equipped with sensors to detect environmental data, such as CO2 concentration, temperature, and humidity. We successfully demonstrated the implementation of such a tool and paired it with a reliable capture mechanism for live mosquitos without destroying important morphological features. The neural network achieved 91.57% accuracy with test set images. When the trap prototype was applied in a tent, the accuracy rate in distinguishing live Ae. aegypti was 92%, with a capture rate reaching 44%. When the prototype was placed into a BG trap to produce a smart mosquito trap, it achieved a 97% recognition rate and a 67% catch rate when placed in the tent. In a simulated living room, the recognition and capture rates were 90% and 49%, respectively. This smart trap correctly differentiated between Cx. quinquefasciatus and Ae. aegypti mosquitoes, and may also help control mosquito-borne diseases and predict their possible outbreak.

Real-time imaging processing with augmented reality glasses for mobile low-vision users

Real-time imaging processing with augmented reality glasses for mobile low-vision users

A public and private key image encryption by modified approach of Vigener cipher and the chaotic maps

ABSTRACT The article presents a real-time image encryption using a modified approach of Vigenère cipher and the chaotic maps with an efficient key management. The modification is provided to Vigenère cipher through the key expansion procedure. The proposed encryption chooses four random keys of 8-tuples to expand into the initial key block. Thereafter, the Vigenère cipher encrypts the pixels blocks using the initial key block and the next blocks through key updation using the XOR operation and the Arnold transform. Later, the partially encrypted image is further scrambled through the Baker map to increase the randomness and unpredictability of the data. The chosen secret key parameters are exchanged publicly through the Diffie–Hellman key exchange agreement. Simulation results and comparison analysis of some extant systems ensure that the proposed algorithm is secure against cryptanalytic attacks and efficient in real-time image processing. Further, the average execution time for encryption and decryption of an image with size is 0.4926 which is very efficient than a typical and recent encryption scheme.

Implementation of real-time image processing on bacterial cellulose formation using soybean-boiled wastewater with the variation of carbon sources during fermentation

Bacterial cellulose (BC) is produced by aerobic bacteria through oxidative fermentation in synthetic and non-synthetic mediums. Several mediums reported to be used as BC formation mediums are coconut water and soybean-boiled wastewater. Carbon sources are needed to optimize the BC formation process. Recent study has implemented a real-time image processing approach for monitoring BC formation. This study aimed to investigate the correlation between variables that influence the fermentation and to determine the kinetic model of BC formation using an image processing approach with the variation of carbon sources during the fermentation. The results showed that the correlation between fermentation time and thickness had the highest percentage for glucose, sucrose, and mannitol mediums. The kinetic observation of BC formation in the medium using glucose, sucrose, and mannitol followed the Gompertz model equation, with the medium using sucrose having the fastest rate of increase at the 44th hour, followed by the medium using mannitol at the 112th hour, and the medium using glucose at the 149th hour.