Real-time Computer Vision Applications Research Articles

Compact optimized deep learning model for edge: a review

<p>Most real-time computer vision applications, such as pedestrian detection, augmented reality, and virtual reality, heavily rely on convolutional neural networks (CNN) for real-time decision support. In addition, edge intelligence is becoming necessary for low-latency real-time applications to process the data at the source device. Therefore, processing massive amounts of data impact memory footprint, prediction time, and energy consumption, essential performance metrics in machine learning based internet of things (IoT) edge clusters. However, deploying deeper, dense, and hefty weighted CNN models on resource-constraint embedded systems and limited edge computing resources, such as memory, and battery constraints, poses significant challenges in developing the compact optimized model. Reducing the energy consumption in edge IoT networks is possible by reducing the computation and data transmission between IoT devices and gateway devices. Hence there is a high demand for making energy-efficient deep learning models for deploying on edge devices. Furthermore, recent studies show that smaller compressed models achieve significant performance compared to larger deep-learning models. This review article focuses on state-of-the-art techniques of edge intelligence, and we propose a new research framework for designing a compact optimized deep learning (DL) model deployment on edge devices.</p>

Domain-Aware Few-Shot Learning for Optical Coherence Tomography Noise Reduction.

Speckle noise has long been an extensively studied problem in medical imaging. In recent years, there have been significant advances in leveraging deep learning methods for noise reduction. Nevertheless, adaptation of supervised learning models to unseen domains remains a challenging problem. Specifically, deep neural networks (DNNs) trained for computational imaging tasks are vulnerable to changes in the acquisition system's physical parameters, such as: sampling space, resolution, and contrast. Even within the same acquisition system, performance degrades across datasets of different biological tissues. In this work, we propose a few-shot supervised learning framework for optical coherence tomography (OCT) noise reduction, that offers high-speed training (of the order of seconds) and requires only a single image, or part of an image, and a corresponding speckle-suppressed ground truth, for training. Furthermore, we formulate the domain shift problem for OCT diverse imaging systems and prove that the output resolution of a despeckling trained model is determined by the source domain resolution. We also provide possible remedies. We propose different practical implementations of our approach, verify and compare their applicability, robustness, and computational efficiency. Our results demonstrate the potential to improve sample complexity, generalization, and time efficiency, for coherent and non-coherent noise reduction via supervised learning models, that can also be leveraged for other real-time computer vision applications.

Robust Deep Learning Models for OFDM-Based Image Communication Systems in Intelligent Transportation Systems (ITS) for Smart Cities

Internet of Things (IoT) ecosystem in smart cities demands fast, reliable, and efficient image data transmission to enable real-time Computer Vision (CV) applications. To fulfill these demands, an Orthogonal Frequency Division Multiplexing (OFDM)-based communication system has been widely utilized due to its higher spectral efficiency and data rate. When adapting such a system to achieve fast and reliable image transmission over fading channels, noise is introduced in the signal which heavily distorts the recovered image. This noise independently corrupts pixel values, however, certain intrinsic properties of the image, such as spatial information, may remain intact, which can be extracted as multidimensional features (in the convolution layers) and interpreted (in the top layers) by a Deep Learning (DL) model. Therefore, the current study analyzes the robustness of such DL models utilizing various OFDM-based image communication systems for CV applications in an Intelligent Transportation Systems (ITS) environment. Our analysis has shown that the EfficientNetV2-based model achieved a range of 70–90% accuracy across different OFDM-based image communication systems over the Rayleigh Fading channel. In addition, leveraging different data augmentation techniques further improves accuracy up to 18%.

Design of deep convolution feature extraction for multimedia information retrieval

PurposeThe analysis of multimedia content is being applied in various real-time computer vision applications. In multimedia content, digital images constitute a significant part. The representation of digital images interpreted by humans is subjective in nature and complex. Hence, searching for relevant images from the archives is difficult. Thus, electronic image analysis strategies have become effective tools in the process of image interpretation.Design/methodology/approachThe traditional approach used is text-based, i.e. searching images using textual annotations. It consumes time in the manual process of annotating images and is difficult to reduce the dependency in textual annotations if the archive consists of large number of samples. Therefore, content-based image retrieval (CBIR) is adopted in which the high-level visuals of images are represented in terms of feature vectors, which contain numerical values. It is a commonly used approach to understand the content of query images in retrieving relevant images. Still, the performance is less than optimal due to the presence of semantic gap among the image content representation and human visual understanding perspective because of the image content photometric, geometric variations and occlusions in search environments.FindingsThe authors proposed an image retrieval framework to generate semantic response through the feature extraction with convolution network and optimization of extracted features using adaptive moment estimation algorithm towards enhancing the retrieval performance.Originality/valueThe proposed framework is tested on Corel-1k and ImageNet datasets resulted in an accuracy of 98 and 96%, respectively, compared to the state-of-the-art approaches.

An efficient and contrast-enhanced video de-hazing based on transmission estimation using HSL color model

This paper proposed a fast and efficient video de-hazing system with reduced computational complexity for real-time computer vision applications. Video de-hazing is an important task and extensively researched in image/video processing and computer vision. The proposed method initially developed and verified for single images and later extended for real-time video’s. The first key aspect of the proposed method is estimating the accurate transmission map using the hue, saturation, and light color model together with red, green, and blue color space. The second relevant aspect is preserving the edges and avoiding halos and artifacts by employing the median of pixels. These aspects reduce the number of computations. It does not require the most computationally complex step of refine transmission map. The advantage of this method is evaluated with five existing classical methods in terms of the average time constant (ATC), peak signal-to-noise ratio, percentage of haze improvement, average contrast of the output image, mean square error and structural similarity index. The comparative experiment shows that the proposed method is two times faster than the existing methods. The qualitative and quantitative analysis demonstrated that the proposed method can attain better de-hazing results and can be efficiently used for real-time video de-hazing applications. Based on comparative analysis, we mapped the proposed method on Raspberry Pi3 and Jetson Nano (GPU) with 24 fps (frames per second) without noticeable delay from input to output and demonstrated for the real-time video.

An Asynchronous Real-Time Corner Extraction and Tracking Algorithm for Event Camera.

Event cameras have many advantages over conventional frame-based cameras, such as high temporal resolution, low latency and high dynamic range. However, state-of-the-art event- based algorithms either require too much computation time or have poor accuracy performance. In this paper, we propose an asynchronous real-time corner extraction and tracking algorithm for an event camera. Our primary motivation focuses on enhancing the accuracy of corner detection and tracking while ensuring computational efficiency. Firstly, according to the polarities of the events, a simple yet effective filter is applied to construct two restrictive Surface of Active Events (SAEs), named as RSAE+ and RSAE−, which can accurately represent high contrast patterns; meanwhile it filters noises and redundant events. Afterwards, a new coarse-to-fine corner extractor is proposed to extract corner events efficiently and accurately. Finally, a space, time and velocity direction constrained data association method is presented to realize corner event tracking, and we associate a new arriving corner event with the latest active corner that satisfies the velocity direction constraint in its neighborhood. The experiments are run on a standard event camera dataset, and the experimental results indicate that our method achieves excellent corner detection and tracking performance. Moreover, the proposed method can process more than 4.5 million events per second, showing promising potential in real-time computer vision applications.

RETRACTED: Simulation of swimming sports image recognition based on multi-core processor and dynamic image sampling

Abstract Human understanding of human movement is one of the most important areas of research activities. In recent years, the use of RGB-D movement recognition data has raised concerns. With artificial intelligence development, the depth of learning technology has made great success in computer vision. A variety of application scenarios and real-time constraints computer vision applications spurred on computer architecture conducted extensive research. These studies take advantage of image processing algorithms and computer vision functions to maximize and improve efficiency through the hardware's scalability. Growth, programmable computing power, and multi-core architecture's potential to accelerate parallel image processing and computer vision algorithms provide a broad prospect. Conduct a new study to discover the unique attributes of three computer vision algorithms that are widely used and determine the optimal computing architecture for each algorithm. Advantages of parallel processing hardware to provide modern programmable graphics data and the thread of instructions, which is significantly faster than the standard CPU implementation.

Improved approximated median filter algorithm for real-time computer vision applications

Median filter is one of the predominant filters that are used to suppress impulse noise. Its simplicity and ability to maintain edges has led to an extensive application in the domain of image processing and computer vision. However, challenges such as moderate to high running time of the standard median filter algorithm and relatively poorer performance when the image is highly corrupted with impulse noise, have led to the design of several variations of the algorithm. One set of variation of the algorithm concentrates on generating quality outputs, while the other set focuses on reducing running time. Among the set targeting the reduction of the running time of the median filter is the DP approximated median filter. However, DP performs poorly when images are corrupted with moderate to high levels of noise. This paper therefore proposes an Improved Approximation Median Filtering Algorithms (IAMFA-I & IAMFA-II) based on DP to generate a better output. The introduction of Mid-Value-Decision-Median in DP reduces the chances of selecting corrupted pixel for denoised image. Experimental results indicate that the IAMFA-II has better running time and equivalent output compared with DP, while IAMFA-I generates better output and has equivalent running time when compared with DP.

Fast and memory efficient de-hazing technique for real-time computer vision applications

Some features of an image may spoil due to fog or haze, smoke. These images lose their brightness due to air-light scattering. It offers troublesomeness to the people lives in hill and fog regions of the world. This paper proposed two key aspects. One is a modified dark-channel method based on the median for eliminating the refine the transmission map as well as halos and artifacts, another important aspect is a memory-efficient row-based arrangement of the pixels for real-time applications. The advantage of this method is air-light can be predicted directly from the modified dark channel and also accurate transmission map can be estimated. This method is compared with other existing four algorithms. Our proposed method analyzed in terms of Peak Signal to Noise Ratio (PSNR), Average Time cost (ATC), percentage of haze improvement (PHI), average contrast of output image (ACOI), Mean Squared Error (MSE) and Structural Similarity Index (SSIM). The quality of the output de-haze image of our algorithm over existing algorithms is more. It has taken less computation time, equal MSE, with higher SSIM and has more percentage of haze improved over existing methods.

A Light-Weight Change Detection Method Using YCbCr-Based Texture Consensus Model

Background subtraction is a prerequisite and often the very first step employed in several high-level and real-time computer vision applications. Several parametric and non-parametric change detection algorithms employing multiple feature spaces have been proposed to date but none has proven to be robust against all challenges that can possibly be posed in a complex real-time environment. Amongst the varied challenges posed, illumination variations, shadows, dynamic backgrounds, camouflaged and bootstrapping artifacts are some of the well-known problems. This paper presents a light-weight hybrid change detection algorithm that integrates a novel combination of RGB color space and conditional YCbCr-based XCS-LBP texture descriptors (YXCS-LBP) into a modified pixel-based background model. The conditional employment of light-weight YXCS-LBP texture features with the modified Visual background extractor (ViBe) aiming at reduction in false positives, produces outperforming results without incurring much memory and computational cost. The random and time-subsampled update strategy employed with the proposed classification procedure ensures the efficient suppression of shadows and bootstrapping artifacts along with the complete retention of long-term static objects in the foreground masks. Comprehensive performance analysis of the proposed technique on publicly available Change Detection dataset (2014 CDnet dataset) demonstrates the superiority of the proposed technique over different state-of-the-art-methods against varied challenges.

A Pre-processing Step for Efficient Edge Extraction

In this paper, we propose a pre-processing step for an efficient edge extraction technique that takes input as an original image to generate an edge map. Generated edge maps could be inputted for state-of-art traditional edge detection algorithms like Canny, Sobel, Prewitt, and recent edge detection algorithms gb-UCM, CED Contours, Structured forest, Sparse Code Gradients and CNN based edge detection Deep Edge, N4 to get better performance. Further, the proposed algorithm has not required any training or learning to improve the edge detection method and is not depending on any parameters. Visual experiments and quantitative evaluation results show that our proposed algorithm greatly improves the modal quality of edge/edge maps. It preserves the original shape, structure of the objects and local features, which presents in an input image. The proposed method takes very less amount of time to execute and making it more suitable for real-time image processing and computer vision applications that depend on edge like classification, object localization, object recognition, image retrieval, segmentation, shape representation.

High-Throughput and Area-Optimized Architecture for rBRIEF Feature Extraction

Feature matching is a fundamental step in many real-time computer vision applications such as simultaneous localization and mapping, motion analysis, and stereo correspondence. The performance of these applications depends on the distinctiveness of the visual feature descriptors used, and the speed at which they can be extracted from video frames. When combined with standard key-point detectors, the rotation-aware binary robust independent elementary features (rBRIEF) descriptor has been shown to outperform its counterparts. In this paper, we present a deep-pipelined stream processing architecture that is capable of extracting rBRIEF features from high-throughput video frames. To achieve high processing rate and low complexity hardware, the proposed architecture incorporates an enhanced moving summation strategy to calculate the key-points’ patch moments and employs approximate computations to achieve patch rotation. Multiplier-less circuitry is introduced throughout the architecture to avoid the use of costly multipliers. Implementation on the Altera Aria V device demonstrates that the proposed architecture leads to 53.3% reduction in hardware resources (adaptive logic modules), while achieving 50% higher accuracy (in terms of average Hamming distance) when compared to the state-of-the-art architecture. In addition, the proposed architecture is able to process high-resolution ( $1920 \times 1080$ ) images at 60 fps, while consuming only 456.15 mW power.

BALG: An alternative for fast and robust feature matching

Robust feature matcher is usually inapplicable to real time computer vision applications as its high computational complexity, while fast feature matcher is short of robustness to geometric transformations. By studying the sampling pattern of binary descriptor and local geometric statistics of features, this paper proposes a fast and robust feature matching method with binary affine invariant descriptor and local geometric consistency check (BALG). The sampling pattern is adaptively adjusted according local affine transformation that can be effectively estimated from the local intensity moments. The affine sampling pattern improves the affine invariance of binary descriptor while enable fast processing. Furthermore, candidate matches are sorted into local groups along different orientations with sequence order constraint, and then followed by local geometric consistency check. False matches are efficiently filtered out with high recall. Extensive experiments on four publicly benchmark datasets prove the proposed method to be an alternative for time critical feature matcher.

Threshold-Guided Design and Optimization for Harris Corner Detector Architecture

High-speed corner detection is an essential step in many real-time computer vision applications, e.g., object recognition, motion analysis, and stereo matching. Hardware implementation of corner detection algorithms, such as the Harris corner detector (HCD) has become a viable solution for meeting real-time requirements of the applications. A major challenge lies in the design of power, energy and area efficient architectures that can be deployed in tightly constrained embedded systems while still meeting real-time requirements. In this paper, we proposed a bit-width optimization strategy for designing hardware-efficient HCD that exploits the thresholding step in the algorithm to determine interest points from the corner responses. The proposed strategy relies on the threshold as a guide to truncate the bit-widths of the operators at various stages of the HCD pipeline with only marginal loss of accuracy. Synthesis results based on 65-nm CMOS technology show that the proposed strategy leads to power-delay reduction of 35.2%, and area reduction of 35.4% over the baseline implementation. In addition, through careful retiming, the proposed implementation achieves over 2.2 times increase in maximum frequency while achieving an area reduction of 35.1% and power-delay reduction of 35.7% over the baseline implementation. Finally, we performed repeatability tests to show that the optimized HCD architecture achieves comparable accuracy with the baseline implementation (average decrease of repeatability is less than 0.6%).

Invasive computing for timing-predictable stream processing on MPSoCs

Abstract Multi-Processor Systems-on-a-Chip (MPSoCs) provide sufficient computing power for many applications in scientific as well as embedded applications. Unfortunately, when real-time requirements need to be guaranteed, applications suffer from the interference with other applications, uncertainty of dynamic workload and state of the hardware. Composable application/architecture design and timing analysis is therefore a must for guaranteeing real-time applications to satisfy their timing requirements independent from dynamic workload. Here, Invasive Computing is used as the key enabler for compositional timing analysis on MPSoCs, as it provides the required isolation of resources allocated to each application. On the basis of this paradigm, this work proposes a hybrid application mapping methodology that combines design-time analysis of application mappings with run-time management. Design space exploration delivers several resource reservation configurations with verified real-time guarantees for individual applications. These timing properties can then be guaranteed at run-time, as long as dynamic resource allocations comply with the offline analyzed resource configurations. This article describes our methodology and presents programming, optimization, analysis, and hardware techniques for enforcing timing predictability. A case study illustrates the timing-predictable management of real-time computer vision applications in dynamic robot system scenarios.

PEL: A Predictive Edge Linking algorithm

We propose an edge linking algorithm that takes as input a binary edge map generated by a traditional edge detection algorithm and converts it to a set of edge segments; filling in one pixel gaps in the edge map, cleaning up noisy edge pixel formations and thinning multi-pixel wide edge segments in the process. The proposed edge linking algorithm walks over the edge map based on the predictions generated from its past movements; thus the name Predictive Edge Linking (PEL). We evaluate the performance of PEL both qualitatively using visual experiments and quantitatively within the precision-recall framework of the Berkeley Segmentation Dataset and Benchmark (BSDS 300). Both visual experiments and quantitative evaluation results show that PEL greatly improves the modal quality of binary edge maps produced by traditional edge detectors, and takes a very small amount of time to execute making it suitable for real-time image processing and computer vision applications.

Acceleration of Points to Convex Region Correspondence Pose Estimation Algorithm on GPUs for Real-Time Applications

In our previous work, a novel algorithm to perform robust pose estimation was presented. The pose was estimated using points on the object to regions on image correspondence. The laboratory experiments conducted in the previous work showed that the accuracy of the estimated pose was over 99% for position and 84% for orientation estimations respectively. However, for larger objects, the algorithm requires a high number of points to achieve the same accuracy. The requirement of higher number of points makes the algorithm, computationally intensive resulting in the algorithm infeasible for real-time computer vision applications. In this paper, the algorithm is parallelized to run on NVIDIA GPUs. The results indicate that even for objects having more than 2000 points, the algorithm can estimate the pose in real time for each frame of high-resolution videos.

Study of Feature based Image Registration Algorithms for Navigation of Unmanned Aerial Vehicles

Image registration is an important preprocessing step for computer vision tasks such as object recognition by a robot or change detection over a natural terrain from satellite images. This paper discusses the basic steps for the feature based image registration, and surveys the various methodologies from literature applicable for each step. The evolution of algorithms on image features is explained with some of their mathematical concepts, along with their limitations. Two videos captured from an unmanned aerial vehicle were used for experimentation. The performance of intensity based image registration using normalized cross correlation and Speeded Up Robust Features was observed by registering various frames of the videos. Intensity based methods perform well for all type of images when the images to be registered vary by only a translation. Feature based methods are suitable when the possible transformation between the images are unknown. Feature based methods fail when the images lack distinct features such as corner points. The study presented would enable one to choose the right methodology for each step of the image registration algorithm, for various real time computer vision applications such as navigation of unmanned aerial vehicles.

Enhanced low-complexity pruning for corner detection

Low-complexity corner detection is essential for many real-time computer vision applications that need to be executed on low-cost/low-power embedded platforms such as robots. The widely used Shi---Tomasi and Harris corner detectors become prohibitive in such platforms due to their high computational complexity, which is attributed to the need to apply a complex corner measure on the entire image. In this paper, we introduce a novel and computationally efficient technique to accelerate the Shi---Tomasi and Harris corner detectors. The proposed technique consists of two steps. In the first step, the complex corner measure is replaced with simple approximations to quickly prune away non-corners. In the second step, the complex corner measure is applied to a small corner candidate set obtained after pruning. Evaluations using standard image benchmarks show that the proposed pruning technique achieves up to 75 % speedup on the Nios-II platform, while yielding corners with comparable or better accuracy than the conventional Shi---Tomasi and Harris detectors.

EDLines: A real-time line segment detector with a false detection control

We propose a linear time line segment detector that gives accurate results, requires no parameter tuning, and runs up to 11 times faster than the fastest known line segment detector in the literature; namely, the line segment detector (LSD) by Grompone von Gioi et al. The proposed algorithm makes use of the clean, contiguous (connected) chain of edge pixels produced by our novel edge detector, the Edge Drawing ( ED) algorithm; hence the name EDLines. The detector includes a line validation step due to the Helmholtz principle, which lets it control the number of false detections. With its accurate results and blazing speed, EDLines will be very suitable for the next generation real-time computer vision and image processing applications.