Embedded Image Processing Research Articles

Implementation of Camera Link Interface on Virtex-5 FPGA

Camera link interface is a standard interface commonly used in current industrial camera and image acquisition card and it is usually realized by special chips. The embedded image processing platform has the FPGA chips, if the camera link interface can be implemented on the FPGA chips, the circuit design will be simplified and the system integration can be improved.

Accelerating embedded image processing for real time: a case study

Many image processing applications need real-time performance, while having restrictions of size, weight and power consumption. Common solutions, including hardware/software co-designs, are based on Field Programmable Gate Arrays (FPGAs). Their main drawback is long development time. In this work, a co-design methodology for processor-centric embedded systems with hardware acceleration using FPGAs is proposed. The goal of this methodology is to achieve real-time embedded solutions, using hardware acceleration, but achieving development time similar to that of software projects. Well established methodologies, techniques and languages from the software domain--such as Object-Oriented Paradigm design, Unified Modelling Language, and multithreading programming--are applied; and semiautomatic C-to-HDL translation tools and methods are used and compared. The methodology is applied to achieve an embedded implementation of a global vision algorithm for the localization of multiple robots in an e-learning robotic laboratory. The algorithm is specifically developed to work reliably 24/7 and to detect the robot's positions and headings even in the presence of partial occlusions and varying lighting conditions expectable in a normal classroom. The co-designed implementation of this algorithm processes 1,600 × 1,200 pixel images at a rate of 32 fps with an estimated energy consumption of 17 mJ per frame. It achieves a 16× acceleration and 92 % energy saving, which compares favorably with the most optimized embedded software solutions. This case study shows the usefulness of the proposed methodology for embedded real-time image processing applications.

The Development of Sun-Tracking System Using Image Processing

This article presents the development of an image-based sun position sensor and the algorithm for how to aim at the Sun precisely by using image processing. Four-quadrant light sensors and bar-shadow photo sensors were used to detect the Sun's position in the past years. Nevertheless, neither of them can maintain high accuracy under low irradiation conditions. Using the image-based Sun position sensor with image processing can address this drawback. To verify the performance of the Sun-tracking system including an image-based Sun position sensor and a tracking controller with embedded image processing algorithm, we established a Sun image tracking platform and did the performance testing in the laboratory; the results show that the proposed Sun tracking system had the capability to overcome the problem of unstable tracking in cloudy weather and achieve a tracking accuracy of 0.04°.

Parallel Programming Methods Based on the Multi-Core DSP TMS320C6670

While the frame rate is higher and the image size is larger, sequence images processing is harder. Good real-time can be ensured by the multi-core DSP in the embedded image processing system. TMS320C6670 which is the multi-core DSP designed by TI corporation is selected as study object. Based on hardware characteristics analyzed, the Data Flow model is adopted as the multi-core processing model. Two data processing subtasks assigning methods are analyzed by comparing their advantages and disadvantages on the system idle time and memory requirements. The data processing subtask assigning flow is design for a serial sequence images processing example. An inter-core data transfer flow design idea is put forward. Using methods and occasion of two kinds of data buffer establishing techniques is studied and defined. An inter-core notification flow design idea is put forward. Using methods and occasion of three notification methods based on the interrupt controller and the Semaphore2 module is studied and defined.

Design of Embedded Image Processing System Using FPGAs

The work aims to design an image processing system to be configured on Spartan-3E FPGA. Embedded design techniques were adopted to construct a soft core processor system. Intel strata flash parallel NOR PROM is added to the system to meet the requirement of storing huge number of image samples. Suitable software driver with C language are used for purpose of erasing the flash PROM and write operations. High- pass and unsharp digital filers were used for the purpose of image processing. Matlab program is used for verifying the results.

Advances in the Detection & Repair of Flash-Eye Defects in Digital Images - A Review of Recent Patents

Red-eye and flash-eye defects in still photography continue to cause problems for digital imaging devices. New variants of flash-eye defects have appeared as cameras and cameras sub-systems get smaller in size. This paper reviews a wide range of the more recently patented techniques that have been applied to solve various problems associated with red and non-red flash-eye, particularly solutions which are implemented in the camera. The use of in-camera metadata, multiple images of the same scene and adaptation of image acquisition conditions to improve red-eye detection and correction are explained. Four principle categories of flash-eye defect are identified and techniques are outlined to deal with each of these. The use of face, eye and scene metadata obtained within the camera is also considered. In particular data obtained regarding indoor/outdoor scenes, partial face regions, eye orientation and gaze and pupil size can all assist with some difficult cases of flash-eye defect. Some advanced techniques to improve the correction and reconstruction of eye regions are also considered. Keywords: Embedded image processing, flash-eye defects, redeye detection, redeye correction, Connected Red Segments, Simple Filter Set, Eye-Glint for Detection, Camera red-eye filter, Camera Metadata, Image Filtering

Research and Application in Image Processing Technology Based on the Embedded Parallel Computing

It is the latest trend of recent research to handle a multi-media picture by using parallel processing technology.Image processing based on the embedded parallel computation is a new field, while not using a parallel processing technique to process the image in the traditional sense. This paper investigates an embedded image processing application parallel to reduce the load imbalance in order to achieve parallel structure.

Transplant Image Processing Technology under Windows into the Platform Based on MiniGUI

Abstract MFC has a large number of digital image processing-related API functions, object-oriented and class mechanisms which provides image processing technology strong support in Windows. But in embedded systems, image processing technology dues to the restrictions of hardware and software do not have the environment of MFC in Windows. Therefore, this paper draws on the experience of image processing technology of Windows and transplants it into MiniGUI embedded systems. The results show that MiniGUI/Embedded graphical user interface applications about image processing which used in embedded image processing system can be good results.

Special issue (part II) on parallel computing for real-time image processing

The performance requirements of image processing applications have continuously increased, especially when they are executed under real-time constraints. We have organized this special issue on Parallel Computing for Real-Time Image Processing to present the current state-of-the-art in the field of parallel programming and the future trends in real-time image and video processing as related to parallel computing or real-time implementation of embedded image processing applications on parallel architectures including multi-core platforms, GPUs and dedicated parallel architectures based upon FPGAs. Due to the overwhelming number and their wide scope of submissions received for this special issue and thus the difficulty associated with finding expert reviewers, it was decided to offer this special issue in three parts. We are very grateful to the reviewers who provided valuable comments and suggestions to improve the quality of the accepted papers. This third part of this special issue on Parallel Computing for Real-Time Image Processing presents five papers addressing different parallel applications including 2D wavelet transform, tone mapping, object recognition, face tracking and dynamic video applications. The realtime implementation of these applications uses GPU, parallel multi-core architecture and dedicated architecture based upon FPGA. Brief outlines of these papers are stated below: The first paper by Franco et al. presents a GPU implementation of the 2D Fast Wavelet Transform, based on a pair of Quadrature Mirror Filters. It also investigates hardware improvements including multicores on the CPU side, and exploits them at thread-level parallelism using OpenMP and Pthreads. Overall, the GPU exhibits better scalability and parallel performance on large-scale images to become a solid alternative for computing the 2D-FWT versus those thread-level methods run on emerging multicore architectures. The second paper by Liu et al. proposes a multi-cue-based face tracking algorithm with the supporting framework using parallel multi-core and one graphic processing unit (GPU). This paper explores two parallel computing techniques to speed up the racking process, especially the most computation-intensive observational steps. One is a multi-core-based parallel algorithm with a Map-Reduce thread model, and the other is a GPU-based speedup approach. The GPU-based technique uses features-matching and particle weight computations, which have been put into the GPU kernel. The results demonstrate that the proposed face tracking algorithm can work robustly with cluttered backgrounds and differing illuminations; the multi-core parallel scheme can increase the speed by 2–6 times compared with that of the corresponding sequential algorithms. Furthermore, a GPU parallel scheme and co-processing scheme can achieve a greater increase in speed (8x–12x) compared with the corresponding sequential algorithms. The third paper by Akil et al. presents the parallel implementation on a GPU of a Real-Time Dynamic Tone Mapping Operator. It describes a generic operator that may be used by any application. However, the goal of this work is to integrate this operator into the graphic rendering process of a car driving simulator, based on its real-time implementation. The tone mapping operator outputs a low M. Akil (&) L. Perroton Universite Paris-Est, LIGM, Equipe A3SI, ESIEE Paris Cite Descartes, BP 99, 93162 Noisy-le-Grand Cedex, France e-mail: m.akil@esiee.fr

Efficient Smart CMOS Camera Based on FPGAs Oriented to Embedded Image Processing

This article describes an image processing system based on an intelligent ad-hoc camera, whose two principle elements are a high speed 1.2 megapixel Complementary Metal Oxide Semiconductor (CMOS) sensor and a Field Programmable Gate Array (FPGA). The latter is used to control the various sensor parameter configurations and, where desired, to receive and process the images captured by the CMOS sensor. The flexibility and versatility offered by the new FPGA families makes it possible to incorporate microprocessors into these reconfigurable devices, and these are normally used for highly sequential tasks unsuitable for parallelization in hardware. For the present study, we used a Xilinx XC4VFX12 FPGA, which contains an internal Power PC (PPC) microprocessor. In turn, this contains a standalone system which manages the FPGA image processing hardware and endows the system with multiple software options for processing the images captured by the CMOS sensor. The system also incorporates an Ethernet channel for sending processed and unprocessed images from the FPGA to a remote node. Consequently, it is possible to visualize and configure system operation and captured and/or processed images remotely.

Special issue on parallel computing for real-time image processing

The performance requirements of image processing applications have continuously increased, especially when they are executed under real-time constraints. We have organized this special issue on parallel computing for real-time image processing to present the current state-of-the-art in the field of parallel programming and the future trends in real-time image and video processing as related to parallel computing or real-time implementation of embedded image processing applications on parallel architectures including multi-core platforms, GPUs and dedicated parallel architectures. Due to the overwhelming number and their wide scope of submissions received for this special issue and thus the difficulty associated with finding expert reviewers, we have decided to offer this special issue in two or possibly three parts to meet the planned appearance of this special issue. The papers that are currently under review will appear in a second or a third part of this special issue immediately after their reviews and re-reviews are concluded. We are very grateful to the reviewers who provided valuable comments and suggestions to improve the quality of the accepted papers. This first part of the special issue on parallel computing for real-time image processing presents articles addressing GPU/Multi-GPU programming and is dedicated to parallel architectures based on FPGA and/or CMOS VLSI technologies for real-time image processing applications. Six papers appear in this first part. Brief outlines of these papers are stated below: The first paper by Anis Rahman, Dominique Houzet, Denis Pellerin, Sophie Marat and Nathalie Guyader describes a parallel implementation of a spatio-temporal visual saliency model with multi-GPU reaching a real-time throughput. This article focuses on the algorithms of this model and details several parallel optimizations. The second paper by Fernanda Palhano, Guillermo Andrade-Barroso and Pierre Hellier presents a method for real-time denoising of ultrasound images discussing a modified version of the NL-means method that incorporates an ultrasound dedicated noise model. It addresses a GPU implementation of this algorithm. Results demonstrate that the proposed method is quite efficient in terms of denoising quality and real-time performance. The third paper by Harald Jordan, Walter van Dyck and Rene Smodic describes a new approach for a contourtracking algorithm targeting a low power smart camera for industrial inspection. This embedded system consists of the three major components: CMOS sensor, FPGA and microprocessor. The fourth paper by Mathieu Thevenin, Michel Paindavoine, Laurent Letellier, Renaud Schmit, and Barthelemy Heyrman describes the eISP, a programmable processing architecture that combines enough computational efficiency for 1080p HD video with silicon area and power characteristics suitable for the next generation of mobile phones (lower than $1 mm and 500 mW in TSMC 65 nm technology). The fifth paper by Loic Sieler, Lionel Damez, Landrault Alexis and Jean-Pierre Derutin presents the parallelization and the embedding of a real-time image stabilization algorithm on SoPC platform. The overall hardware implementation method is based upon meeting algorithm M. Akil (&) L. Perroton Universite Paris-Est, LIGM, Equipe A3SI, ESIEE Paris Cite Descartes, BP 99, 93162 Noisy-le-Grand Cedex, France e-mail: akilm@esiee.fr

Signal Analysis and Classification for Surface Plasmon Assisted Microscopy of Nanoobjects

Abstract The need for portable and on site screening methods for viruses is evident in face of virus infections that can spread fastly in a heavily connected world. A robust and efficient method for detecting viruses is a novel technique called Plasmon Assisted Microscopy of Nanoobjects. It is based on the acquisition of images from a sensor surface exploiting the behavior of surface plasmons in the presence of nanoobjects. In this paper an efficient image analysis approach with respect to the requirements of the sensor is presented and an embedded image processing system for this purpose is introduced. The processing pipeline comprises three steps and starts with restorating the images by removing the background and filtering artifacts. The acquired image series is analyzed pixel by pixel in a second pipeline step in order to detect pixels containing nanoobjects. In a last step pixels are aggregated to nanoobject structures. The paper introduces in the context of this virus detection method a configurable embedded system that was used for rapid prototyping of the image analysis algorithms in a flexible way.

이동로봇을 위한 구조광 영상기반 거리측정

In this paper, we address an active ranging system based on laser structured light image for mobile robot application. Since the burdensome correspondence problem is avoidable, the structured light image processing has efficient computation in comparison with the conventional stereo image processing. By using a cylindrical lens in the laser generation, it is possible to convert a point laser into a stripe laser without motorized scan in the proposed system. In order to achieve robustness against environmental illumination noise, we propose an efficient integro-differential image processing algorithm. The proposed system has embedded image processing module and transmits distance data to reduce the computational burden in main control system.

A SoC based on the Gaussian Pyramid (GP) for Embedded image Applications

This paper presents a System-On-a-chip (SoC) for embedded image processing and pattern recognition applications that need Gaussian Pyramid structure. The system is fully implemented into Field-Programmable Gate Array (FPGA) based on the prototyping platform. The SoC consists of embedded processor core and a hardware accelerator for Gaussian Pyramid construction. The performance of the implementation is benchmarked against software implementations on different platforms.

Parallelism and Scalability in an Image Processing Application

The recent trends in processor architecture show that parallel processing is moving into new areas of computing in the form of many-core desktop processors and multi-processor system-on-chips. This means that parallel processing is required in application areas that traditionally have not used parallel programs. This paper investigates parallelism and scalability of an embedded image processing application. The major challenges faced when parallelizing the application were to extract enough parallelism from the application and to reduce load imbalance. The application has limited immediately available parallelism and further extraction of parallelism is limited by small data sets and a relatively high parallelization overhead. Load balance is difficult to obtain due to the limited parallelism and made worse by non-uniform memory latency. Three parallel OpenMP implementations of the application are discussed and evaluated. We show that with some modifications relative speedups in excess of 9 on a 16 CPU system can be reached.

Special issue on reconfigurable architecture for real-time image processing

The performance requirements of image processing applications have led to increase in the computing power of implementation platforms, in particular when real-time constraints are to be met. Image processing applications may consist of different standards, or different algorithms used at different stages of the processing chain. The computing paradigm of reconfigurable architectures promises a trade-off between flexibility and performance. Reconfigurable architectures can exploit fine-grain (more suitable for low-level and medium-level operations in the image processing chain) and coarse-grain parallelism (more suitable for high-level operations in the image processing chain). Many reconfigurable architectures have been constructed specifically for image processing using different processors and dedicated circuits such as ASICs and FPGAs. This special issue on Reconfigurable Architecture for Real-Time Image Processing presents articles addressing reconfigurable computing for real-time image processing, programming frameworks with FPGA-style mapping as well as real-time and embedded image processing applications and implementations. This special issue includes nine papers that are briefly outlined below: The first paper, by Denoulet and Merigot, presents a massively parallel SIMD architecture based on reconfigurability and asynchronism called Associative Mesh. The introduced virtualized associative mesh achieves real-time execution for split and merge segmentation, watershed segmentation and motion detection. The second paper, by Kessal, Karabernou and Demigny, focuses on the Ardoise project. The goal of this project is to realize a dynamically reconfigurable platform. This paper proposes a methodology that can be easily adapted to common SoC architecture. The third paper, by Sen, Hemaraj, Plishker, Shekhar and Bhattacharya, presents a novel architecture that enables dynamically-reconfigurable image registration. It also addresses some data-flow-motivated parallel architectures for image registration. The fourth paper, by Meng, Freeman, Pears and Bailey, details the implementation of a human action recognition system on a reconfigurable, FPGA based video processing architecture. The fifth paper, by Jiang, Crookes and Bouridane, describes a parallel-matching processor architecture to perform high-speed biometric fingerprint database retrieval. The processor was implemented on Xilinx Virtex-E and runs up to 65 MHz. The sixth paper, by Chandrasekaran, Amira, Minghua and Bermak, covers the presentation of an architecture for the Finite Ridgelet Transform. It presents a parallel architecture as well as FPGA and ASIC implementations. The seventh paper, by Srinam and Eng, presents various implementations of the Kolmogorov phase screen generator. The FPGA hardware implementation provides a speedup of more than 60 times of the original algorithm. The eight paper, by Saponara, Casula and Fanucci proposes to combine the application specific instruction-set processor (ASIP) paradigm with the reconfigurable hardware one. The ninth paper, by Seetharaman, Venkataramani and Lakshminarayanan, describes how to get the benefits of M. Akil (&) Department of Computer Science, Institut Gaspar-Monge, Unite mixte de recherche CNRS-UMLVPE-ESIEE (UMR 8049), ESIEE, Cite Descartes, BP 99, 93162 Noisy-le-Grand Cedex, France e-mail: akilm@esiee.fr

A Robotic Facial Expression Recognition System Using Real-Time Vision System

The capability of recognizing human facial expression plays an important role in advanced human-robot interaction development. Through recognizing facial expressions, a robot can interact with a user in a more natural and friendly manner. In this paper, we proposed a facial expression recognition system based on an embedded image processing platform to classify different facial expressions on-line in real time. A low-cost embedded vision system has been designed and realized for robotic applications using a CMOS image sensor and digital signal processor (DSP). The current design acquires thirty 640x480 image frames per second (30 fps). The proposed emotion recognition algorithm has been successfully implemented on the real-time vision system. Experimental results on a pet robot show that the robot can interact with a person in a responding manner. The developed image processing platform is effective for accelerating the recognition speed to 25 recognitions per second with an average on-line recognition rate of 74.4% for five facial expressions.

Book Review [review of "Embedded Image Processing on the TMS320C6000 DSP: Examples in Code Composer Studio and MATLAB" by S. Qureshi (2005)

This book focuses on embedded image processing and, more specifically, on the efficient implementation of image processing algorithms on the TMS320C6000 family of digital signal processors (DSPs) from Texas Instruments (TI). The text is divided into six chapters: "Introduction," "Tools," "Spatial Processing Techniques," "Image Filtering," "Edge Detection and Segmentation," and "Wavelets." The text is supported by two appendices, as well as a CD-ROM. The CD-ROM includes all source code and project files as well as many test images and one text video clip. There is no better way to learn than by example, and the book offers plenty of them. While the book is intended for signal and image processing practitioners or software developers that plan to use a TI DSP, most of the programming techniques demonstrated in the book can easily be applied to other embedded platforms. The book is well written and succeeds in filling a big void in image processing literature, tackling how to efficiently implement signal and image processing algorithms using embedded processors.

Top down structured parallelisation of embedded image processing applications

The authors present a general system design method which is intended to support parallelisation of complete image processing applications using MIMD processors. The approach is based upon the utilisation of a generic system level parallel processor architecture, the 'pipeline processor farm'(PPF), and is applicable to any embedded application with continuous input/output. The design method is illustrated using applications from the fields of computer vision and image coding. The design model accommodates several commonly exploited parallel processing paradigms, maps conveniently to the software structure of most image processing algorithms, provides incrementally scalable performance, and enables upper-bound speedups to be easily estimated from profiling data generated by the original sequential implementation of the application. It is believed that the approach has significant application in parallel embedded systems design, in the development environment, and in simulation work for computationally intensive image coding algorithms.