Image Processing Hardware Research Articles

Image Processing Hardware Acceleration—A Review of Operations Involved and Current Hardware Approaches

Image Processing Hardware Acceleration—A Review of Operations Involved and Current Hardware Approaches

A novel technique to resolve directional ambiguity for Particle Streak Velocimetry

In this study, a novel streak direction-resolving algorithm is introduced to determine the direction of the flow field for a single image frame with multiple streaks. The streak direction was resolved by varying its intensity from 100 % to 50 % of the total intensity of the streak image. This technique was applied to two main types of flows parallel and Hill's vortex flows, parallel flows were divided into further two types constant velocity parallel flow and accelerating parallel flow. The purpose of using different types of flows was to test the robustness of the algorithm with easy and complex flows. The performance of the algorithm was checked by the angular deviation between the true and least-square fitted velocity vectors. The number of correct synthetic streak directions was measured with the success rate in percentage. A high success rate means low angular deviation and a high number of velocity vectors with correct direction was obtained. In this research, four different types of image formats were considered DP (double precision), 16-bit (without noise), 16-bit (1.0 % noise), 16-bit (5.0 % noise), and 8-bit (without noise) and the best results were obtained for DP and 16-bit image formats. The results of the parallel flows indicated a 100 % streak direction success rate, Hill's vortex was a type of complex flow therefore, the algorithm hard to resolve some streak directions due to very low velocities (less than 0.1 px (pixel)/interval) and very high-velocity gradients (greater than 52 px/interval). The observation shows that Hill's vortex synthetic streak images were resolved with a success rate of 92.76 % which means that the majority of the synthetic streaks were resolved. Experimental analysis was also done by using the PDMS microchannel setup. Intensity variation streaks were recorded with long camera exposure time and for maintaining the intensity variation, as the LED switched on maximum illumination was started, and continuous decrement in illumination was set by switching off at 50 % of the total illumination intensity. The best results were achieved with a 94 % success rate. Therefore, the proposed novel approach can be used with less expensive hardware for image processing with a single image frame and is useful for multiple applications.

Implementation of two-step gradual reset scheme for enhancing state uniformity of 2D hBN-based memristors for image processing

In-memory computing facilitates efficient parallel computing based on the programmable memristor crossbar array. Proficient hardware image processing can be implemented by utilizing the analog vector-matrix operation with multiple memory states of the nonvolatile memristor in the crossbar array. Among various materials, 2D materials are great candidates for a switching layer of nonvolatile memristors, demonstrating low-power operation and electrical tunability through their remarkable physical and electrical properties. However, the intrinsic device-to-device (D2D) variation of memristors within the crossbar array can degrade the accuracy and performance of in-memory computing. Here, we demonstrate hardware image processing using the fabricated 2D hexagonal boron nitride-based memristor to investigate the effects of D2D variation on the hardware convolution process. The image quality is evaluated by peak-signal-to-noise ratio, structural similarity index measure, and Pratt’s figure of merit and analyzed according to D2D variations. Then, we propose a novel two-step gradual reset programming scheme to enhance the conductance uniformity of multiple states of devices. This approach can enhance the D2D variation and demonstrate the improved quality of the image processing result. We believe that this result suggests the precise tuning method to realize high-performance in-memory computing.

Autonomous mobile robots for outdoor tasks

This paper presents a low-cost hardware platform for outdoor robots, being suitable for education, industrial prototyping and private use. The choice of components isdiscussed, including platform, sensors and controller as well as GPS and image processing hardware. Furthermore, a software approach is proposed, allowing students and researchers to easily implement own algorithms for localization, navigation and the tasks to fulfill. Several robots can be integrated in a framework which connects various different hardware platforms, called the BOSPORUS network. Together with other components, they form an intelligent network for gathering sensor and image data, sensor data fusion, navigation and control of mobile platforms. The architecture of a reference platform on the campus of the Brandenburg University of Applied Sciences is presented and evaluated.

New approaches for low phototoxicity imaging of living cells and tissues.

Fluorescence microscopy is a powerful tool used in scientific and medical research, but it is inextricably linked to phototoxicity. Neglecting phototoxicity can lead to erroneous or inconclusive results. Recently, several reports have addressed this issue, but it is still underestimated by many researchers, even though it can lead to cell death. Phototoxicity can be reduced by appropriate microscopic techniques and carefully designed experiments. This review focuses on recent strategies to reduce phototoxicity in microscopic imaging of living cells and tissues. We describe digital image processing and new hardware solutions. We point out new modifications of microscopy methods and hope that this review will interest microscopy hardware engineers. Our aim is to underscore the challenges and potential solutions integral to the design of microscopy systems. Simultaneously, we intend to engage biologists, offering insight into the latest technological advancements in imaging that can enhance their understanding and practice.

AraDQ: an automated digital phenotyping software for quantifying disease symptoms of flood-inoculated Arabidopsis seedlings

BackgroundPlant scientists have largely relied on pathogen growth assays and/or transcript analysis of stress-responsive genes for quantification of disease severity and susceptibility. These methods are destructive to plants, labor-intensive, and time-consuming, thereby limiting their application in real-time, large-scale studies. Image-based plant phenotyping is an alternative approach that enables automated measurement of various symptoms. However, most of the currently available plant image analysis tools require specific hardware platform and vendor specific software packages, and thus, are not suited for researchers who are not primarily focused on plant phenotyping. In this study, we aimed to develop a digital phenotyping tool to enhance the speed, accuracy, and reliability of disease quantification in Arabidopsis.ResultsHere, we present the Arabidopsis Disease Quantification (AraDQ) image analysis tool for examination of flood-inoculated Arabidopsis seedlings grown on plates containing plant growth media. It is a cross-platform application program with a user-friendly graphical interface that contains highly accurate deep neural networks for object detection and segmentation. The only prerequisite is that the input image should contain a fixed-sized 24-color balance card placed next to the objects of interest on a white background to ensure reliable and reproducible results, regardless of the image acquisition method. The image processing pipeline automatically calculates 10 different colors and morphological parameters for individual seedlings in the given image, and disease-associated phenotypic changes can be easily assessed by comparing plant images captured before and after infection. We conducted two case studies involving bacterial and plant mutants with reduced virulence and disease resistance capabilities, respectively, and thereby demonstrated that AraDQ can capture subtle changes in plant color and morphology with a high level of sensitivity.ConclusionsAraDQ offers a simple, fast, and accurate approach for image-based quantification of plant disease symptoms using various parameters. Its fully automated pipeline neither requires prior image processing nor costly hardware setups, allowing easy implementation of the software by researchers interested in digital phenotyping of diseased plants.

Flood sequence mapping with multimodal remote sensing under the influence of dense vegetation

ABSTRACT Remote sensing (RS) imagery is becoming increasingly popular in surface water extent extraction thanks to the increasing availability of RS data and advancements in image processing algorithms, software, and hardware. Many studies proved that RS imagery can work independently or along with other approaches to identify flood extent. However, due to the insufficiency in the number of images from single-sourced RS and independent references for validation, most studies just depicted the inundation status near the peak of the inundation. The potential of those images to document flood events at different stages (e.g. rising and receding stages) has not been well investigated. To close that gap, this study investigated the efficacy of RS-based multi-spatiotemporal flood inundation mapping using multimodal RS images captured on different dates to describe the entire flooding process. Additionally, a Quantile-based Filling & Refining (QFR) workflow was proposed to resolve the blocking effects of dense vegetation in study areas. We tested the multimodal flood mapping workflow plus the QFR correction in four lock and dam sites on the Mississippi River by comparing the RS-based flood maps with HEC-RAS simulations. Our results demonstrated the usefulness of multimodal RS images in describing flooding events at different stages and showcased the potential of those images to serve as a reliable reference source in data-scarce areas. In addition, results showed that the standard water extraction plus post-processing will not guarantee accurate flood maps in densely vegetated areas. In contrast, map processed with QFR were noticeably more consistent with HEC-RAS maps, especially for flood maps generated with PlanetScope images, for which the median accuracy has been improved from below 0.5 to above 0.94 after the QFR postprocessing. Thanks to the simple structure, the proposed multimodal RS flood mapping workflow plus QFR correction procedures can be fully automated and can thus benefit near-real-time applications.

Performance verification and latency time evaluation of hardware image processing module for appearance inspection systems using FPGA

Performance verification and latency time evaluation of hardware image processing module for appearance inspection systems using FPGA

Hardware Acceleration of Satellite Remote Sensing Image Object Detection Based on Channel Pruning

Real-time detection of satellite remote sensing images is one of the key technologies in the field of remote sensing, which requires not only high-efficiency algorithms, but also low-power and high-performance hardware deployment platforms. At present, the image processing hardware acceleration platform mainly uses an image processing unit (GPU), but the GPU has the problem of large power consumption, and it is difficult to apply to micro-nano satellites and other devices with limited volume, weight, computing power, and power consumption. At the same time, the deep learning algorithm model has the problem of too many parameters, and it is difficult to directly deploy it on embedded devices. In order to solve the above problems, we propose a YOLOv4-MobileNetv3 field programmable gate array (FPGA) deployment scheme based on channel layer pruning. Experiments show that the acceleration strategy proposed by us can reduce the number of model parameters by 91.11%, and on the aerial remote sensing dataset DIOR, the average accuracy of the design scheme in this paper reaches 82.61%, the FPS reaches 48.14, and the average power consumption is 7.2 W, which is 317.88% FPS higher than the CPU and reduces the power consumption by 81.91%. Compared to the GPU, it reduces power consumption by 91.85% and improves FPS by 8.50%. Compared with CPUs and GPUs, our proposed lightweight algorithm model is more energy-efficient and more real-time, and is suitable for application in spaceborne remote sensing image processing systems.

Application of Machine Vision Technology in Citrus Production

The construction of standardized citrus orchards is the main trend in the future development of modern agriculture worldwide. As the most widely used and mature technology in the agricultural field, machine vision has greatly promoted the industrial development model of the citrus industry. This paper summarizes the application of machine vision technology including citrus pest and disease detection, harvesting identification and localization, and fruit grading. We compare the advantages and disadvantages of relevant research, and analyze the existing problems and prospects for future research. Due to the complex and changeable in-field environment, robots may experience unpredictable interference in the recognition process, which leads to errors in target fruit localization. The lack of datasets also affects the accuracy and stability of the algorithm. While expanding the dataset, it is necessary to conduct further research on the algorithm. In addition, the existing research focuses on indoor monitoring methods, which are not practical for the changeable outdoors environment. Therefore, realizing the diversity of sample datasets, designing agricultural robots suitable for complex environments, developing high-quality image processing hardware and intelligent parallel algorithms, and increasing dynamic monitoring methods are the future research directions. Although machine vision has certain limitations, it is still a technology with strong potential for development.

SMR-RS: An Improved Mask R-CNN Specialized for Rolled Rice Stubble Row Segmentation

As a highly productive rice, ratoon rice is widely planted worldwide, but the rolling of rice stubble in mechanical harvesting severely limits its total yield; based on this, some scholars have proposed rolled rice stubble righting machines. However, limited by the uncertainty of the field environment, the machine’s localization accuracy of the target needs to be improved. To address this problem, real-time detection of rolled rice stubble rows is a prerequisite. Therefore, this paper introduces a deep learning method for the first time to achieve this. To this end, we presented a novel approach to improve a model that is used for the simplification of Mask R-CNN, which does not require any modules to be added or replaced on the original model. Firstly, two branches in the second stage were deleted, and the region proposals output from the stage was used directly as the mask generation region, and segmentation performance was substantially improved after a simple optimization of the region proposals. Further, the contribution of the feature map was counted, and the backbone network was simplified accordingly. The resulting SMR-RS model was still able to perform instance segmentation and has better segmentation performance than Mask R-CNN and other state-of-the-art models while significantly reducing the average image processing time and hardware consumption.

AI explainability framework for environmental management research

Deep learning networks powered by AI are essential predictive tools relying on image data availability and processing hardware advancements. However, little attention has been paid to explainable AI (XAI) in application fields, including environmental management. This study develops an explainability framework with a triadic structure to focus on input, AI model and output. The framework provides three main contributions. (1) A context-based augmentation of input data to maximize generalizability and minimize overfitting. (2) A direct monitoring of AI model layers and parameters to use leaner (lighter) networks suitable for edge device deployment, (3) An output explanation procedure focusing on interpretability and robustness of predictive decisions by AI networks. These contributions significantly advance state of the art in XAI for environmental management research, offering implications for improved understanding and utilization of AI networks in this field.

The use of a neural network in solving problems of recognition and classification of spacecraft by their optical images

In recent years, the number of space objects located in near-Earth outer space, especially in the near operational zone, has increased significantly due to the build-up of space groupings, including dual-use (for example, Starlink) and the remnants of their vital activity (space debris). This factor increases the importance of the task of recognizing and classifying space objects by type in the shortest possible time and entering them into the main catalog of space objects. The developed methodology allows automated analysis of optical images of space objects using software to solve the problem of their recognition and classification by type using a convolutional neural network. The purpose of the study is to increase the efficiency of processing and analysis of optical images of spacecraft. The experimental results of the study confirm the achievement of the research goal. The developed methodology contributes to the development of software and hardware for image processing and can be used in calculations and data preparation for information support of interested officials. For the first time, a training set for a convolutional neural network has been prepared using real optical images of spacecraft obtained in the visible range.

Intelligent Pick-and-Place System Using MobileNet

The current development of a robotic arm solution for the manufacturing industry requires performing pick-and-place operations for work pieces varying in size, shape, and color across different stages of manufacturing processes. It aims to reduce or eliminate the human error and human intervention in order to save manpower costs and enhance safety at the workplace. Machine learning has become more and more prominent for object recognition in these pick-and-place applications with the aid of imaging devices and advances in the image processing hardware. One of the key tasks in object recognition is feature extraction and object classification based on convolutional neural network (CNN) models, which are generally computationally intensive. In this paper, an intelligent object detection and picking system based on MobileNet is developed and integrated into an educational six-axis robotic arm, which requires less computation resources. An experimental test is conducted on six-axis robotic arm called Niryo One to train the model and identify three objects with difference shapes and colors. It is shown by the confusion matrix that the MobileNet model achieves an accuracy of 91%, a dramatic improvement compared to 65% of the Niryo One’s original sequential model. The statistical study also shows the MobileNet can achieve a higher precision with more clustered spread of accuracy.

A hardware system for real-time decoding of in vivo calcium imaging data.

Epifluorescence miniature microscopes ('miniscopes') are widely used for in vivo calcium imaging of neural population activity. Imaging data are typically collected during a behavioral task and stored for later offline analysis, but emerging techniques for online imaging can support novel closed-loop experiments in which neural population activity is decoded in real time to trigger neurostimulation or sensory feedback. To achieve short feedback latencies, online imaging systems must be optimally designed to maximize computational speed and efficiency while minimizing errors in population decoding. Here we introduce DeCalciOn, an open-source device for real-time imaging and population decoding of in vivo calcium signals that is hardware compatible with all miniscopes that use the UCLA Data Acquisition (DAQ) interface. DeCalciOn performs online motion stabilization, neural enhancement, calcium trace extraction, and decoding of up to 1024 traces per frame at latencies of <50 ms after fluorescence photons arrive at the miniscope image sensor. We show that DeCalciOn can accurately decode the position of rats (n = 12) running on a linear track from calcium fluorescence in the hippocampal CA1 layer, and can categorically classify behaviors performed by rats (n = 2) during an instrumental task from calcium fluorescence in orbitofrontal cortex. DeCalciOn achieves high decoding accuracy at short latencies using innovations such as field-programmable gate array hardware for real-time image processing and contour-free methods to efficiently extract calcium traces from sensor images. In summary, our system offers an affordable plug-and-play solution for real-time calcium imaging experiments in behaving animals.

The Object Detection, Perspective and Obstacles In Robotic: A Review

A few years back, when the image processing hardware and software were created, it was limited, and most of the time, object detection would fail., But as with time, the advancement in technology has significantly changed the scenario. A lot of researchers worked in this field to carry out a solution through which they can detect objects in any field, especially in the robotic domain [1]. In today's world, with so much research in the field of deep learning, it is very easy to identify and detect any object using computer vision. This paper focuses on the various deep learning technologies and algorithms through which object detection can be done. A new and advanced deep learning technology known as salient object detection has been discussed. Also, the 3D object detection and the end-to-end approach for object detection are discussed. The existing methods of deep learning through which object detection can be done. The applications in which object detection is being used and the importance of object detection. It also reports; what the predecessors have done, what problems have been solved by them, how they solved these problems, the characteristics of the predecessors' methods and their future work.

Development of a general purpose verification environment for high-level-synthesis image processing hardware with support for dynamic partial reconfiguration

Development of a general purpose verification environment for high-level-synthesis image processing hardware with support for dynamic partial reconfiguration

The Development and Clinical Application of Innovative Optical Ophthalmic Imaging Techniques.

The field of ophthalmic imaging has grown substantially over the last years. Massive improvements in image processing and computer hardware have allowed the emergence of multiple imaging techniques of the eye that can transform patient care. The purpose of this review is to describe the most recent advances in eye imaging and explain how new technologies and imaging methods can be utilized in a clinical setting. The introduction of optical coherence tomography (OCT) was a revolution in eye imaging and has since become the standard of care for a plethora of conditions. Its most recent iterations, OCT angiography, and visible light OCT, as well as imaging modalities, such as fluorescent lifetime imaging ophthalmoscopy, would allow a more thorough evaluation of patients and provide additional information on disease processes. Toward that goal, the application of adaptive optics (AO) and full-field scanning to a variety of eye imaging techniques has further allowed the histologic study of single cells in the retina and anterior segment. Toward the goal of remote eye care and more accessible eye imaging, methods such as handheld OCT devices and imaging through smartphones, have emerged. Finally, incorporating artificial intelligence (AI) in eye images has the potential to become a new milestone for eye imaging while also contributing in social aspects of eye care.

Path Planning Robot Using PI-CAM

Abstract: Currently, the path planning robot is one among the foremost researched topics in autonomous robotics technology. Path planning has important applications in many areas, for example industrial robotics, autonomous systems, and 3D digital representation that can perform various types of simulations iteratively about product performances. The utilization of versatile robots for modern purposes like arranging, moving, and putting is at this point not an inclination however a need. Our path planning robot is using OpenCV, image processing and various hardware’s such as raspberry pi 4, Arduino uno , motor driver and pi -cam. The analysis of image will be done using image processing through raspberry pi and for edge detection canny edge algorithm is used. Keywords: OpenCV, Raspberry pi 4, Pi-Cam, Canny Edge Detection, Image Processing.

A Configurable and Real-Time Multi-Frequency 3D Image Signal Processor for Indirect Time-of-Flight Sensors

3D time-of-flight (ToF) cameras have recently attracted widespread attention because of their potential in wide applications. However, the lack of versatile 3D image signal processor (ISP) is still an obstacle that hinders their popularity. The main challenges lie in the management of high-throughput raw data and the complexity of 3D image processing in hardware. To address these challenges, this paper presents a top-down 3D ISP with high real-time performance and high energy efficiency for indirect ToF (IToF) imaging. First, we propose configurable cache strategies to improve the efficiency of non-streaming memory accesses and support different IToF sensors. Then, to resolve the phase ambiguity problem commonly existing in IToF sensors, we develop a hardware-optimized cascaded multi-frequency fusion method, which not only supports up to six modulation frequencies but also facilitates the pixel-wise detection of multipath interference. Moreover, we propose a scalable calculation engine for depth calculation and calibration, 3D image enhancement, and post-processing. The pipeline is prototyped on a Xilinx ZCU102 FPGA board. Results show that the 3D ISP has negligible precision loss and low latency, achieving <inline-formula> <tex-math notation="LaTeX">$53.35\times $ </tex-math></inline-formula> energy efficiency improvement compared with the implementation on Intel CPU and <inline-formula> <tex-math notation="LaTeX">$55.45\times $ </tex-math></inline-formula> improvement over the ARM CPU implementation. When running at 75 MHz, the 3D ISP can support <inline-formula> <tex-math notation="LaTeX">$1280\times 960$ </tex-math></inline-formula> depth, intensity, and point cloud output at a frame rate of 30 f/s.