Image Processing Hardware Research Articles

Comparison of bispectrum, multiframe blind deconvolution and hybrid bispectrum-multiframe blind deconvolution image reconstruction techniques for anisoplanatic, long horizontal-path imaging

The potential benefits of real-time, or near-real-time, image processing hardware to correct for turbulence-induced image defects for long-range surveillance and weapons targeting are sufficient to motivate significant resource commitment to their development. Quantitative comparisons between potential candidates are necessary to decide on a preferred processing algorithm. We begin by comparing the mean-square-error (MSE) performance of speckle imaging (SI) methods and multiframe blind deconvolution (MFBD), applied to long-path horizontal imaging of a static scene under anisoplanatic seeing conditions. Both methods are used to reconstruct a scene from three sets of 1000 simulated images featuring low, moderate, and severe turbulence-induced aberrations. The comparison shows that SI techniques can reduce the MSE up to 47%, using 15 input frames under daytime conditions. The MFBD method provides up to 40% improvement in MSE under the same conditions. The performance comparison is repeated under three diminishing light conditions, 30, 15, 8 photons per pixel on average, where improvements of up to 39% can be achieved using SI methods with 25 input frames, and up to 38% for the MFBD method using 150 input frames. The MFBD estimator is applied to three sets of field data and representative results presented. Finally, the performance of a hybrid bispectrum-MFBD estimator that uses a rapid bispectrum estimate as the starting point for the MFBD image reconstruction algorithm is examined.

Bottom‐up performance analysis of focal‐plane mixed‐signal hardware for Viola–Jones early vision tasks

SummaryFocal‐plane mixed‐signal arrays have traditionally been designed according to the general claim that moderate accuracy in processing is affordable. The performance of their circuitry has been analyzed in these terms without a comprehensive study of the ultimate consequences of such moderate accuracy. In this paper, for the first time to the best of our knowledge, we do carry out this study. We move expectable performance of mixed‐signal image processing hardware directly into the vision algorithm making use of it. This permits to close a wider design loop, enabling a more aggressive design of this kind of hardware provided that the algorithm, at the highest level—semantic interpretation of the scene—, can afford it. Thus, we present a thorough analysis of the non‐idealities associated with the implementation of a QVGA array tailored for the distinctive characteristics of the Viola–Jones processing framework. The resulting deviation models are then introduced in the processing flow of this framework provided by the OpenCV library. We have found, contrary to what could be expected, that these deviations do not necessarily degrade the performance of the Viola–Jones algorithm. They could be even beneficial for certain high‐level specifications. Additionally, we demonstrate the architectural advantages of our approach: exploitation of focal‐plane distributed memory and ultra‐low‐power operation. Copyright © 2014 John Wiley & Sons, Ltd.

Camera selection for tracking in distributed smart camera networks

Tracking persons with multiple cameras with overlapping fields of view instead of with one camera leads to more robust decisions. However, operating multiple cameras instead of one requires more processing power and communication bandwidth, which are limited resources in practical networks. When the fields of view of different cameras overlap, not all cameras are equally needed for localizing a tracking target. When only a selected set of cameras do processing and transmit data to track the target, a substantial saving of resources is achieved. The recent introduction of smart cameras with on-board image processing and communication hardware makes such a distributed implementation of tracking feasible. We present a novel framework for selecting cameras to track people in a distributed smart camera network that is based on generalized information-theory. By quantifying the contribution of one or more cameras to the tracking task, the limited network resources can be allocated appropriately, such that the best possible tracking performance is achieved. With the proposed method, we dynamically assign a subset of all available cameras to each target and track it in difficult circumstances of occlusions and limited fields of view with the same accuracy as when using all cameras.

Logarithmic Image Sensor for Wide Dynamic Range Stereo Vision System

Stereo vision is the most universal way to get 3D information passively. The fast progress of digital image processing hardware makes this computation approach realizable now. It is well known that stereo vision matches 2 or more images of a scene, taken by image sensors from different points of view. Any information loss, even partially in those images, will drastically reduce the precision and reliability of this approach. In this paper, we introduce a stereo vision system designed for depth sensing that relies on logarithmic image sensor technology.The hardware is based on dual logarithmic sensors controlled and synchronized at pixel level. This dual sensor module provides high quality contrast indexed images of a scene. This contrast indexed sensing capability can be conserved over more than 140dB without any explicit sensor control and without delay. It can accommodate not only highly non-uniform illumination, specular reflections but also fast temporal illumination changes.

Optimised computer vision system for automatic pre-grading of citrus fruit in the field using a mobile platform

The mechanisation and automation of citrus harvesting is considered to be one of the best options to reduce production costs. Computer vision technology has been shown to be a useful tool for fresh fruit and vegetable inspection, and is currently used in post-harvest fruit and vegetable automated grading systems in packing houses. Although computer vision technology has been used in some harvesting robots, it is not commonly utilised in fruit grading during harvesting due to the difficulties involved in adapting it to field conditions. Carrying out fruit inspection before arrival at the packing lines could offer many advantages, such as having an accurate fruit assessment in order to decide among different fruit treatments or savings in the cost of transport and marketing non-commercial fruit. This work presents a computer vision system, mounted on a mobile platform where workers place the harvested fruits, that was specially designed for sorting fruit in the field. Due to the specific field conditions, an efficient and robust lighting system, very low-power image acquisition and processing hardware, and a reduced inspection chamber had to be developed. The equipment is capable of analysing fruit colour and size at a speed of eight fruits per second. The algorithms developed achieved prediction accuracy with an R2 coefficient of 0.993 for size estimation and an R2 coefficient of 0.918 for the colour index.

多重情報地図を用いた隊列走行のための区画線検出システム

This paper proposes a method of lane marker detection for platooning. In order to reduce air resistance, it is desirable to shorten the distance between two vehicles. If the vehicular gap is very short, conventional methods, which detect lane markers ahead in images captured from a front camera, are useless because lane markers are occluded by a vehicle in front. To solve this problem, the proposed method recognizes a lane markers in images captured from two downward side cameras equipped with the front side and the rear side of a vehicle. First, candidate points of lane markers are extracted in each image by edge pair detection. Then, straight lines representing lane marker are detected by applying Hough transform to these candidate points. A lateral position in a traffic lane is estimated from a position of a straight line in an image of each downward side camera. A yaw angle toward a traffic lane is calculated by using these lateral positions and the distance between two downward side cameras. Because a downward side camera can take only a narrow area directly under it, lane markers must be detected from short parts of them. Therefore, the proposed method uses a multi-information map containing lane marker information to detect lane markers. The proposed method has been implemented in the image processing hardware whose CPU satisfies on-vehicle specifications. Experimental results show the effectiveness of the proposed method and a lane detection device. In experiments conducted at the test course and the highway under construction, the vehicle ran at 80 km/h along the straight lane automatically.

A Programmable High Speed Vision System with Superscalar PE and Its Parallel Computing Language

Pixel-parallel PE and SIMD architectures are widely used in high-speed image processing to enhance computing power. With fully exploiting the data level parallelism of low- and middle-level image processing, SIMD architecture is able to finish great amount of computation with much less instruction cycle thus satisfy the high-speed system requirement. The main computation parts in those SIMD image processing hardware is known as PE (processing element) and it is responsible for transferring, storing and processing the image data. This paper describes a high-speed vision system with superscalar PE to enhance system performance and its dedicated parallel computing language specifically devel-oped for this vision system. The vision system can achieve motion detection at more than 2000fps and face detection at more than 100 fps which overwhelms some general serial CPUs in the same applications.

API compilation for image hardware accelerators

We present an API-based compilation strategy to optimize image applications, developed using a high-level image processing library, onto three different image processing hardware accelerators. We demonstrate that such a strategy is profitable for both development cost and overall performance, especially as it takes advantage of optimization opportunities across library calls otherwise beyond reach. The library API provides the semantics of the image computations. The three image accelerator targets are quite distinct: the first one uses a vector architecture; the second one presents an SIMD architecture; the last one runs both on GPGPU and multicores through OpenCL. We have adapted standard compilation techniques to perform these compilation and code generation tasks automatically. Our strategy is implemented in PIPS, a source-to-source compiler which greatly reduces the development cost as standard phases are reused and parameterized. We carried out experiments with applications on hardware functional simulators and GPUs. Our contributions include: (1) a general low-cost compilation strategy for image processing applications, based on the semantics provided by library calls, which improves locality by an order of magnitude; (2) specific heuristics to minimize execution time on the target accelerators; (3) numerous experiments that show the effectiveness of our strategies. We also discuss the conditions required to extend this approach to other application domains.

Study of the retinal imaging and output in premature baby under a computer-assisted binocular indirect ophthalmoscope.

To explore a clear retinal imaging and output and enhance the development of retinopathy of prematurity (ROP) screening, which is safe and effective for ROP screening in premature infants. A computer-assisted binocular indirect ophthalmoscope imaging and output system was equipped with camera and image processing hardware and connected to computers. The process of fundus examination was videotaped (photograph) and output. Simulated eyes were utilized to debug video head and acquire stable and clear fundus images by binocular indirect ophthalmoscope for premature infants. Fundus imaging output technique was sucessfully established. The common reasons of unclear imaging and corresponding solutions were summarized. This technique can capture and output stable and clear fundus images of premature infants. Assisted by hardware and software processing, a compute assisted binocular indirect ophthalmoscope imaging and output system was established, which can be used for screening, research, treatment and follow-up of ROP in premature babies to resolve the difficulty in obtaining clear fundus photograph.

Real-time geometric lens distortion correction using a graphics processing unit

Optical imaging systems often suffer from distortion artifacts which impose important limitations on the direct interpretation of the images. It is possible to correct for these aberrations through image processing, but due to their calculation-intensive nature, the required corrections are typically performed offline. However, with image-based applications that operate interactively, real-time correction of geometric distortion artifacts can be vital. We propose a new method to generate undistorted images by implementing the required distortion correction algorithm on a commercial graphics processing unit (GPU), distributing the necessary calculations to many stream processors that operate in parallel. The proposed technique is not limited to affine lens distortions but allows for the correction of arbitrary geometric image distortion artifacts through individual pixel resampling at display rates of more than 30 frames per second for fully processed images (1024×768 pixels). Our method enables real-time GPU-based geometric lens distortion correction without the need for additional digital image processing hardware.

Strategies for Implementing Hardware-Assisted High-Throughput Cellular Image Analysis

Recent advances in imaging technology for biomedicine, including high-speed microscopy, automated microscopy, and imaging flow cytometry are poised to have a large impact on clinical diagnostics, drug discovery, and biological research. Enhanced acquisition speed, resolution, and automation of sample handling are enabling researchers to probe biological phenomena at an increasing rate and achieve intuitive image-based results. However, the rich image sets produced by these tools are massive, possessing potentially millions of frames with tremendous depth and complexity. As a result, the tools introduce immense computational requirements, and, more importantly, the fact that image analysis operates at a much lower speed than image acquisition limits its ability to play a role in critical tasks in biomedicine such as real-time decision making. In this work, we present our strategy for high-throughput image analysis on a graphical processing unit platform. We scrutinized our original algorithm for detecting, tracking, and analyzing cell morphology in high-speed images and identified inefficiencies in image filtering and potential shortcut routines in the morphological analysis stage. Using our "grid method" for image enhancements resulted in an 8.54× reduction in total run time, whereas origin centering allowed using a look up table for coordinate transformation, which reduced the total run time by 55.64×. Optimization of parallelization and implementation of specialized image processing hardware will ultimately enable real-time analysis of high-throughput image streams and bring wider adoption of assays based on new imaging technologies.

Navigation Concepts for Magnetic Resonance Imaging–Guided Musculoskeletal Interventions

Image-guided musculoskeletal (MSK) interventions are a widely used alternative to open surgical procedures for various pathological findings in different body regions. They traditionally involve one of the established x-ray imaging techniques (radiography, fluoroscopy, computed tomography) or ultrasound scanning. Over the last decades, magnetic resonance imaging (MRI) has evolved into one of the most powerful diagnostic tools for nearly the whole body and has therefore been increasingly considered for interventional guidance as well.The strength of MRI for MSK applications is a combination of well-known general advantages, such as multiplanar and functional imaging capabilities, wide choice of tissue contrasts, and absence of ionizing radiation, as well as a number of MSK-specific factors, for example, the excellent depiction of soft-tissue tumors, nonosteolytic bone changes, and bone marrow lesions. On the downside, the magnetic resonance-compatible equipment needed, restricted space in the magnet, longer imaging times, and the more complex workflow have so far limited the number of MSK procedures under MRI guidance.Navigation solutions are generally a natural extension of any interventional imaging system, in particular, because powerful hardware and software for image processing have become routinely available. They help to identify proper access paths, provide accurate feedback on the instrument positions, facilitate the workflow in an MRI environment, and ultimately contribute to procedural safety and success.The purposes of this work were to describe some basic concepts and devices for MRI guidance of MSK procedures and to discuss technical and clinical achievements and challenges for some selected implementations.

The RABiT: A Rapid Automated Biodosimetry Tool for radiological triage. II. Technological developments

Purpose: Over the past five years the Center for Minimally Invasive Radiation Biodosimetry at Columbia University has developed the Rapid Automated Biodosimetry Tool (RABiT), a completely automated, ultra-high throughput biodosimetry workstation. This paper describes recent upgrades and reliability testing of the RABiT.Materials and methods: The RABiT analyses fingerstick-derived blood samples to estimate past radiation exposure or to identify individuals exposed above or below a cut-off dose. Through automated robotics, lymphocytes are extracted from fingerstick blood samples into filter-bottomed multi-well plates. Depending on the time since exposure, the RABiT scores either micronuclei or phosphorylation of the histone H2AX, in an automated robotic system, using filter-bottomed multi-well plates. Following lymphocyte culturing, fixation and staining, the filter bottoms are removed from the multi-well plates and sealed prior to automated high-speed imaging. Image analysis is performed online using dedicated image processing hardware. Both the sealed filters and the images are archived.Results: We have developed a new robotic system for lymphocyte processing, making use of an upgraded laser power and parallel processing of four capillaries at once. This system has allowed acceleration of lymphocyte isolation, the main bottleneck of the RABiT operation, from 12 to 2 sec/sample. Reliability tests have been performed on all robotic subsystems.Conclusions: Parallel handling of multiple samples through the use of dedicated, purpose-built, robotics and high speed imaging allows analysis of up to 30,000 samples per day.

Efficient Smart CMOS Camera Based on FPGAs Oriented to Embedded Image Processing

This article describes an image processing system based on an intelligent ad-hoc camera, whose two principle elements are a high speed 1.2 megapixel Complementary Metal Oxide Semiconductor (CMOS) sensor and a Field Programmable Gate Array (FPGA). The latter is used to control the various sensor parameter configurations and, where desired, to receive and process the images captured by the CMOS sensor. The flexibility and versatility offered by the new FPGA families makes it possible to incorporate microprocessors into these reconfigurable devices, and these are normally used for highly sequential tasks unsuitable for parallelization in hardware. For the present study, we used a Xilinx XC4VFX12 FPGA, which contains an internal Power PC (PPC) microprocessor. In turn, this contains a standalone system which manages the FPGA image processing hardware and endows the system with multiple software options for processing the images captured by the CMOS sensor. The system also incorporates an Ethernet channel for sending processed and unprocessed images from the FPGA to a remote node. Consequently, it is possible to visualize and configure system operation and captured and/or processed images remotely.

New field programmable gate array-based image-oriented acquisition and real-time processing applied to plasma facing component thermal monitoring

During operation of present fusion devices, the plasma facing components (PFCs) are exposed to high heat fluxes. Understanding and preventing overheating of these components during long pulse discharges is a crucial safety issue for future devices like ITER. Infrared digital cameras interfaced with complex optical systems have become a routine diagnostic to measure surface temperatures in many magnetic fusion devices. Due to the complexity of the observed scenes and the large amount of data produced, the use of high computational performance hardware for real-time image processing is then mandatory to avoid PFC damages. At Tore Supra, we have recently made a major upgrade of our real-time infrared image acquisition and processing board by the use of a new field programmable gate array (FPGA) optimized for image processing. This paper describes the new possibilities offered by this board in terms of image calibration and image interpretation (abnormal thermal events detection) compared to the previous system.

Study of a New Kind of Low Light Level Night Vision Device

A way based on B-Spline wavelet transform is used to get the edge extraction of the night image. Compared with classical methods of edge detection, it provides higher precision and saves more details etc. Formulation of communication protocol is put forward. With the combination of laser technology and night vision technology, hardware and software program and image processing is designed to realize the auto-detection, monitoring and alarm. The visual range in full dark is 1.8km and more than 2.1km in 1/4 moonlight

The Scalable Pipeline Architecture behind HP's T300 Color Inkjet Web Press

The HP T300 Color Inkjet Web Press uses two identical print engines capable of printing full variable data at 400 fpm and 1200x600dpi on a 30” wide web. Each print engine prints four colors of ink (CMYK) and Bonding Agent (BA). Each of the five inks employs two print bars in tandem in an arrangement that provides nozzle redundancy. Each of the 10 printbar contains seven 4.25” thermal inkjet printheads with over ten thousand nozzles each. This massive array of nozzles consumes 35 Giga bits of nozzle firing data per second.The data flow starts with the Digital Front End which provides the interface to the press, processes job ticketing and decomposes incoming PDF files into small chunks that are then concurrently processed by a scalable array of raster image processors (RIPs) built on high-performance blade servers. This solution allows RIPs to be tailored to meet any demand required by a customer's print job.The raster images are then compressed and buffered before being delivered in print sequence order to the two print engines.The print engines partition the incoming raster images into bands called slices; each slice covers the width of one Printhead plus some overlap area. Image processing hardware runs in parallel for each slice transforming the compressed CMYK continuous tone raster into real time nozzle firing instructions.The result is a scalable image processing architecture that extends to wider presses by simply replicating the basic hardware building blocks across the web.

Novel micromirror design with variable pull-in voltage

A novel micromirror is presented having a variable pull-in voltage. This allows analog Pulse Width Modulation (PWM) so that the number of grey levels is not limited by the number of subframes or bitplanes. The levels can be chosen arbitrarily, allowing less severe speed requirements for the electronic layer below the MEMS, less image processing hardware and memory. The presented design of the mirror has two attracting electrodes and one landing electrode at either side of the mirror. One electrode is used as ‘fixed’ electrode, influencing the other attracting electrode’s pull-in voltage. If we apply a general triangular waveform at the former electrode, the ‘fixed’ electrode voltage determines the duty cycle of the mirror. The mirror implements analog PWM, without needing a transistors based comparator at the CMOS level. When each ‘fixed’ electrode is connected to an active matrix cell, an active matrix display can be formed. The mirrors with variable pull-in voltage were fabricated using SiGe as the structural layer. The variable pull-in principle is demonstrated by measurements performed on these SiGe mirrors [1].

Real-time photon beam localization methods using high-resolution imagers and parallel processing using a reconfigurable system

We report on a high-precision, real-time photon beam localization and characterization instrument. The device uses a 2-D (image) sensor coupled to a hardware image processing system. The system uses two different algorithms that both run in parallel on field programmable gate array logic using data from a single active pixel array sensor. The first algorithm mimics the ubiquitous quadrant photodiode design that features high precision but has limited range and requires calibration cycles. The second algorithm calculates the location of the center of gravity. We compare the merits of both methods by measuring the displacement of a light beam. It is shown that the center of gravity (or centroid) method offers the advantage of large dynamic range with excellent linearity. By the method of operating on identical image data captured from these experiments, we aim to investigate both algorithms and compare their performance.

Towards Vision-Based Autonomous Landing for Small Unmanned Air Vehicles: Image Processing Hardware Development

This paper describes the development of intelligent vision capabilities for small-unmanned aerial vehicles as part of the program CAROLO at the Technische Universitat Braunschweig, Germany. The development’s objective is to create appropriate hardware that suits the requirements of small- unmanned aerial vehicles (5 kg maximum takeoff weight). Enabling vision-based flight guidance during landing was chosen as the first task and acts as a technology demonstrator for future payload-driven vision capabilities. The paper describes the developed image processing hardware and its capabilities.Attention is also paid to implementing image processing algorithms for detecting a known landing point marker and computing its relative position and attitude.