Real-time Color Image Research Articles

Accelerated FPGA-Based Vector Directional Filter for Real-Time Color Image Denoising with Enhanced Performance

This paper presents an accelerated implementation of the Vector Directional Filter (VDF) on a Field Programmable Gate Array (FPGA) for real-time denoising of color images. The VDF effectively suppresses noise while preserving edges and fine details, making it ideal for a range of applications such as satellite and multispectral biomedical imaging. However, the filter's high computational complexity poses challenges for real-time processing. Existing solutions either fail to meet real-time execution requirements or compromise image quality through hardware implementation approximations. To overcome these challenges, we first model the VDF using C/C++ programming, and subsequently design an efficient floating-point hardware architecture employing the High-Level Synthesis (HLS) flow. Optimal directives are selected using the Xilinx Vivado HLS tool. The VDF architecture is then integrated as a coprocessor with the Cortex-A53 hardcore processor in the XCZU9EG FPGA. To enhance data bandwidth between software and hardware components, three Direct Memory Access (DMAs) units are utilized to transfer three image lines in parallel. Furthermore, internal memory is implemented on the XCZU9EG FPGA, providing increased flexibility for managing the restored image. The VDF Software/Hardware (SW/HW) design's robustness and accuracy are validated through experimental studies on the ZCU102 board. Our design accelerates the filtering process by 21 times, maintaining visual quality and effectively removing noise from color images compared to the VDF SW design. Additionally, our solution outperforms existing approaches in terms of filtered image quality and processing time, showing a 24% improvement in the worst-case scenario.

VLSI Design Based on Block Truncation Coding for Real-Time Color Image Compression for IoT.

It has always been a major issue for a hospital to acquire real-time information about a patient in emergency situations. Because of this, this research presents a novel high-compression-ratio and real-time-process image compression very-large-scale integration (VLSI) design for image sensors in the Internet of Things (IoT). The design consists of a YEF transform, color sampling, block truncation coding (BTC), threshold optimization, sub-sampling, prediction, quantization, and Golomb-Rice coding. By using machine learning, different BTC parameters are trained to achieve the optimal solution given the parameters. Two optimal reconstruction values and bitmaps for each 4 × 4 block are achieved. An image is divided into 4 × 4 blocks by BTC for numerical conversion and removing inter-pixel redundancy. The sub-sampling, prediction, and quantization steps are performed to reduce redundant information. Finally, the value with a high probability will be coded using Golomb-Rice coding. The proposed algorithm has a higher compression ratio than traditional BTC-based image compression algorithms. Moreover, this research also proposes a real-time image compression chip design based on low-complexity and pipelined architecture by using TSMC 0.18 μm CMOS technology. The operating frequency of the chip can achieve 100 MHz. The core area and the number of logic gates are 598,880 μm2 and 56.3 K, respectively. In addition, this design achieves 50 frames per second, which is suitable for real-time CMOS image sensor compression.

A Feasibility Study of LiDAR-Enhanced Augmented Reality on a Handheld Device for Collision Detection and Patient Positioning

<h3>Purpose/Objective(s)</h3> This study investigates the feasibility of using a commercially available handheld device with augmented reality (AR) cameras, enhanced by light detection and ranging (LiDAR), for pre-treatment collision detection and patient positioning. The on-device LiDAR scanner detects object depths with direct time-of-flight (ToF) measurements, thus improving the accuracy of contrast-based visual depth estimation. Our proposed approach leverages AR for near-real-time reconstruction of patient external contour during simulation and setup. This is advantageous compared with conventional laser-tattoo alignment and photo-based patient position, by providing an intuitive 3D rendering of the patient's body and immobilization devices at arbitrary angles. <h3>Materials/Methods</h3> Two possible modes for AR-assisted positioning were investigated: mesh and point-cloud mode. In mesh mode, surface mesh of a mock pediatric phantom (50 × 40 × 15 cm³) was rapidly reconstructed via the default meshing algorithm from the software development kit (SDK). Resulting 3D contours were manually registered to a computerized tomography (CT) scan of the phantom in a third-party open source image processing suite. For point cloud mode, surface anchor points of the phantom and its supporting devices were detected and placed in the real-world frame of reference, which were subsequently overlaid on the real-time color image. Each handheld scanning session was kept at 30 seconds. The native pulse intervals of 0.2 – 0.5 nsec of the LiDAR scanner were used; depth map was updated at 60 frames per second (fps). Scanning method was kept consistent and mimicked CT scans, with the device camera rotated about the superior-inferior axis pointing towards the phantom. The resolution of the resulting surface mesh and point cloud was quantified. Extent of the volume reconstructed was estimated. <h3>Results</h3> The scan and on-device real-time reconstruction were completed within 30 seconds. We successfully extracted the reconstructed mesh and registered that to a CT scan of the phantom. A mean distance of 1.5 ± 1.0 cm between each adjacent vertex was achieved. The point cloud density was on an average 20.9 ± 0.9 points/cm², with decreased point density 35cm superior to the phantom midpoint (10.0 points/cm²). Both modes captured approximately 4 × 4 × 4m³ volume of the phantom and its supporting structures. <h3>Conclusion</h3> AR has emerged as a promising modality for surface guided patient positioning, yet several previous studies have found a lack of sufficient mesh resolution and/or accuracy. LiDAR improves both aspects with directly measured depth information. We demonstrated that sub-cm full 3D rendering of the patient body contour and supporting device is achievable with a handheld device in under 30 seconds, which could provide a viable option for improved guidance on collision check and patient positioning.

A real-time ultra-low light color imaging system based on FPGA

This article shows a low light color image acquisition system, The core components of the system are the Fairchild’s SCMOS image sensor CIS1910F1111 and XILINX’s Artix-7 XC7A100T-2CSG324I FPGA, the remarkable advantage of the system is that it can obtain better color imaging effect under lower illumination environment, and the image noise is much less than other similar products. Based on the excellent imaging performance of the image detector, a high performance real-time low-light level color imaging system is developed. This imaging system can obtain the characteristic information of the targets under ultra-low illuminance environment, including the details, colors and so on. The hardware of the low light level imaging system mainly contains a color SCMOS image sensor and a FPGA, a driving circuit of a combination of DDR3, the ultra-low noise power conversion circuit and a Camera-Link and a 3G-SDI interface circuits. The SCMOS chip is used for photoelectric conversion of the shot scene and the FPGA is used for the control of the whole imaging system, image acquisition and image processing, etc, The FPGA software system consists of SCMOS initialize configuration and timing control module, automatic exposure control module, real-time color image processing module, imaging tone mapping module, image denoising module and image enhancement module. The automatic exposure control (AEC) module adaptively adjusts the average gray value of the region of interest. The module automatically calculates the exposure time and gain value of the next frame according to the current frame image data value. The real-time color image processing module includes color restoration, automatic white balance and color spaces conversion, etc. The image denoising module uses the advanced real-time guide-filter algorithm. The image tone mapping module and enhancement module are proposed based on an improved automatic threshold logarithmic and enhancement algorithm. Combining the hardware and FPGA soft algorithm with excellent performance, the imaging results show that the system can get good color image effect of the ultra-low light level about 10-2lx.

Multilevel algorithm for color clustering of images

The fast multilevel algorithm to cluster color images (MACC – Multilevel Algorithm for Color Clustering) is presented. Currently, several well-known algorithms of image clustering, including the k‑means algorithm (which is one of the most commonly used in data mining) and its fuzzy versions, watershed, region growing ones, as well as a number of new more complex neural network and other algorithms are actively used for image processing. However, they cannot be applied for clustering large color images in real time. Fast clustering is required, for example, to process frames of video streams shot by various video cameras or when working with large image databases. The developed algorithm MACC allows the clustering of large images, for example, FullHD size, on a personal computer with an average deviation from the original color values of about five units in less than 20 milliseconds, while a parallel version of the classical k‑means algorithm performs the clustering of the same images with an average error of more than 12 units for a time exceeding 2 seconds. The proposed algorithm of multilevel color clustering of images is quite simple to implement. It has been extensively tested on a large number of color images.

실시간 고해상도 컬러 영상 워핑을 위한 FPGA 기반 하드웨어 설계

본 논문에서는 고해상도 컬러 영상 워핑의 실시간 처리를 위한 FPGA 기반 하드웨어를 설계한다. 영상 워핑은 입력 영상에 2D 좌표 변환을 적용하는 영상 처리 알고리즘으로 실시간 영상 워핑 구현을 위해서는 좌표 변환 연산, 픽셀 데이터 보간 연산, 그리고 고속 영상 버퍼 메모리 구조 등 3가지 문제에서 실시간 성능 및 정확성을 확보하여야 한다. 본 논문에서는 CFA 컬러 영상 워핑을 활용한 연산량 감소, 오프라인 캐시 스케줄링 및 캐시 메모리 분할 구조를 적용한 영상 데이터 캐시, 그리고 연산의 선후 관계 및 타이밍 조건을 고려한 병렬 처리 및 파이프라인 구조 등을 통하여 실시간 동작 성능을 확보한다. Full HD 해상도의 컬러 영상 입력에 대한 구현된 시스템의 동작 정확성을 PC 소프트웨어의 처리 결과와 비교/확인하고, 실시간 렌즈 왜곡 보정에 적용하여 구현된 시스템의 타당성을 검증한다.

Bio-inspired multimodal 3D endoscope for image-guided and robotic surgery.

Image-guided and robotic surgery based on endoscopic imaging technologies can enhance cancer treatment by ideally removing all cancerous tissue and avoiding iatrogenic damage to healthy tissue. Surgeons evaluate the tumor margins at the cost of impeding surgical workflow or working with dimmed surgical illumination, since current endoscopic imaging systems cannot simultaneous and real-time color and near-infrared (NIR) fluorescence imaging under normal surgical illumination. To overcome this problem, a bio-inspired multimodal 3D endoscope combining the excellent characteristics of human eyes and compound eyes of mantis shrimp is proposed. This 3D endoscope, which achieves simultaneous and real-time imaging of three-dimensional stereoscopic, color, and NIR fluorescence, consists of three parts: a broad-band binocular optical system like as human eye, an optical relay system, and a multiband sensor inspired by the mantis shrimp's compound eye. By introducing an optical relay system, the two sub-images after the broad-band binocular optical system can be projected onto one and the same multiband sensor. A series of experiments demonstrate that this bio-inspired multimodal 3D endoscope not only provides surgeons with real-time feedback on the location of tumor tissue and lymph nodes but also creates an immersive experience for surgeons without impeding surgical workflow. Its excellent characteristics and good scalability can promote the further development and application of image-guided and robotic surgery.

Disordered Nonlinear Metalens for Raman Spectral Nanoimaging.

Over the past decades, considerable progress has been made toward far-field optical imaging beyond the diffraction limit. However, most working proof-of-concepts are based on either time-consuming scanning of a subdiffraction focal spot over a sample or postrecovery treatment using a priori information on a sought image. To our knowledge, none of these can be regarded as being close to a perfect far-field superlensing system capable of real-time color imaging with subwavelength resolution. In this paper, we suggest a proof-of-concept for far-field nonlinear metalens that is made of a disordered metal-dielectric nanocomposite. Postoxidation of a refractory titanium nitride (TiN) thin film, used as a nonlinear plasmonic material, results in the formation of a titanium oxynitride (TiON) film comprising a mixture of multiple phases of TiOxNy. Due to a double epsilon-near-zero behavior near the percolation threshold, the TiON favors supercoupling of the incident light to surface plasmon resonance within the visible and near-infrared range. Point spread function narrowing is achieved owing to the multiplicative nature of stimulated Raman scattering (SRS) and enhanced third-order optical nonlinearity in TiN and TiO2 particle chains through plasmon resonances and Anderson localization of light, respectively. Combined with a conventional confocal optical microscope, the multimode metalens shows subwavelength resolution of λ/6NA at different visible wavelengths (SRS overtones) using multiwalled carbon nanotubes as a test sample. We are confident that our finding will bring us one step closer to developing a robust and versatile far-field super-resolution color imaging system and, eventually, implementing "eye-on-a-chip" technology.

Region Secure and Pattern Based Alternate 2B-3C Steganography Approach for Data Security in Color Images

Background and Objective: The public and group communication carried out in the social, cloud, and mobile networks suffer from unauthorized users. The data security is required for secure authentication, monetary transactions and other sensitive information. The steganography methods empower the user with invulnerable communication in public domain. In this paper, a region secure and pattern based alternate 2B-3C (2Bit-3Channel) Steganography method is proposed to improve the content reliability and data security. Methods: In this method, the cover pixels are retrieved by applying the spiral rectangular pattern over the cover image. This spiral pattern also secures the core content region and the pixels from the core ROI (Region of Interest) is used as a cover only for larger size secret information. In this proposed steganography method, two channels are considered alternatively based on the defined rules. The LSB (Least Significant Bit) of both channels is XOR with input data bits and key to perform data hiding. The method is applied on real time color images taken from COREL and USC-SIPI-ID. The evaluation of proposed steganography approach is done using MSE (Mean Square Error), PSNR (Peak Signal-to-Noise Ratio), BER (Bit Error Rate), NCC (Normalized Cross Correlation) and SSIM (Structural Similarity Index Matrix). Result: The comparative results are generated for 50 real time images against the existing LSB (Least Significant Bit), LSBM (Least Significant Bit Modulation), LSBMR (Least Significant bit Matched Revisited), SCC (Stego Color Cycle), SHSI (Simple Hue-Saturation-Intensity), HIS-MLSB(Simple Hue-Saturation-Intensity-Modified LSB) and IMMEA(Iterative Magic Matrix Encryption Algorithm) methods. Conclusion: The experimentation results verified that the proposed 2B-3C approach improved the data hiding in qualitative and quantitative aspects.

Evolution of photodynamic medicine based on fluorescence image-guided diagnosis using indocyanine green and 5-aminolevulinic acid.

New diagnostic techniques based on photodynamic medicine, such as near-infrared fluorescence using indocyanine green (NIR-ICG) and 5-aminolevulinic acid-mediated photodynamic diagnosis (ALA-PDD), are aiding navigation tasks across various fields of surgery. Specifically, NIR-ICG is being used for the intraoperative identification of sentinel lymph nodes or blood vessels in organ resection and for blood flow evaluation in surgery. These ICG-fluorescent imaging techniques could provide an additional and potentially valuable way to identify vascular and lymphatic structures in surrounding tissue. 5-Aminolevulinic acid is a precursor of a photosensitizing substance with affinity for tumors; thus, diagnostic laparoscopy using ALA-PDD in combination should improve the accuracy of detecting peritoneal dissemination in patients with advanced gastric cancer. The ability to overlay fluorescent imaging with conventional color images in real time using ALA-PDD and NIR with ICG would be of immense benefit to surgeons, providing good visualization and detection of target lesions not seen with the naked eye. A multi-center clinical study examining the safety and efficacy of ALA-PDD during laparoscopic examination for patients with advanced gastric cancer is currently underway in the form of doctor-initiated trials, and further verification studies will be conducted. Such imaging capability could have broad potential across cancer and vascular surgery.

Image steganalysis: real-time adaptive colour image segmentation for hidden message retrieval and Matthew's correlation coefficient calculation

Image steganalysis: real-time adaptive colour image segmentation for hidden message retrieval and Matthew's correlation coefficient calculation

Color sensors and their applications based on real-time color image segmentation for cyber physical systems

Color information plays an important role in the color image segmentation and real-time color sensor, which affects the result of video image segmentation and correct real-time temperature value. In this paper, a novel real-time color image segmentation method is proposed, which is based on color similarity in RGB color space. According to the color and luminance information in RGB color space, the dominant color is determined at first, and then color similarity can be calculated with the proposed calculation method of color component, which creates a color-class map. Next, the information of the corresponding color-class map is utilized to classify the pixels. Due to the characteristic that thermal inks feature color values that change in real time as the temperature changes, the segmentation results of thermal ink can be used as a real-time color sensor. Then, we also propose a method of color correction and light source compensation for the sake of potential inaccuracy of its measures. We discuss the proposed segmentation method application combining with color sensor (thermal ink) in real-time color image segmentation for Cyber physical system (CPS) by the application in fire detection and summarize a new method in identifying fire in a video based on these characteristics. The experiments showed that the proposed method in vision-based fire detection and identification in videos was effective; the results were accurate and can be used in real-time analysis.

Extraordinary perceptual color stability in low cost, real time color image compression inspired by structure tensor analysis

Extraordinary perceptual color stability in low cost, real time color image compression inspired by structure tensor analysis

고해상도 컬러 영상 워핑의 실시간 구현을 위한 영상 캐시 알고리즘

This paper presents a new image cache algorithm for real-time implementation of high-resolution color image warping. The cache memory is divided into four cache memory modules for simultaneous readout of four input image pixels in consideration of the color filter array (CFA) pattern of an image sensor and CFA image warping. In addition, a pipeline structure from the cache memory to an interpolator is shown to guarantee the generation of an output image pixel at each system clock cycle. The proposed image cache algorithm is applied to an FPGA-based real-time color image warping, and experimental results are presented to show the validity of the proposed method.

Preserving color fidelity in real-time color image compression using a ranking naturalness criterion

Preserving color fidelity in real-time color image compression using a ranking naturalness criterion

How suitable is structure tensor analysis for real-time color image compression in context of high quality display devices

How suitable is structure tensor analysis for real-time color image compression in context of high quality display devices

一种基于FPGA的彩色图像实时增强方法

一种基于FPGA的彩色图像实时增强方法

The robust reconstruction of disparity maps using multilevel processing of color image stereopairs

A novel approach, which makes it possible to realize the reconstruction of disparity maps for subsequent visualization of three-dimensional color images in real time, is proposed and substantiated.

Special issue on real-time color image processing

Color perception plays an important role in object recognition and scene understanding both for humans and intelligent vision systems. Recent advances in digital color imaging and computer hardware technology have led to an explosion in the use of color images in a variety of applications including medical imaging, content-based image retrieval, biometrics, watermarking, digital inpainting, remote sensing, visual quality inspection, among many others. As a result, automated processing and analysis of color images has become an active area of research, which is witnessed by the large number of publications during the past two decades. The multivariate nature of color image data presents new challenges for researchers and practitioners as the numerous methods developed for singlechannel images are often not directly applicable to multichannel images. In addition, common scalar image processing operations often become computationally intractable when performed in high-dimensional spaces. As a result, the development of computationally practical methods for color image processing has become an important research area. The goals of this special issue are to summarize the state-of-the-art in real-time color image processing and to provide future directions for this exciting subfield of image processing. The intended audience includes researchers and professionals, who are increasingly dealing with the processing and analysis of color images and video. The special issue opens with ‘‘Design of a Shift-andAdd Based Hardware Accelerator for Color Space Conversion’’ by Li et al. The authors propose a hardware/software co-design architecture for Nios II, a 32-bit embedded processor. The architecture integrates a pipelined color space converter hardware accelerator based on a genetic algorithm and an LCD touch module. When implemented on a system-on-a-programmable-chip, the proposed architecture is capable of converting a 512 512 RGB image to YCbCr in 0:11 s. The special issue continues with seven articles on color image enhancement. In ‘‘A Simple Gray-Edge Automatic White Balance Method with FPGA Implementation,’’ Tan et al. propose a real-time automatic white balance method based on the gray-edge hypothesis. To reduce computational time, the method employs horizontal downsampling using the mean filter and gradient computation using horizontal first-order difference operator. The authors demonstrate the practicality of their approach using an FPGA implementation. In ‘‘A Full Linear 3 3 Color Correction between Images,’’ Lecca describes a fast color correction algorithm that is capable of equalizing the colors of two images of the same scene that are acquired under different illuminants and/or by different devices. The author demonstrates the performance of her approach against two state-of-the-art approaches on a variety of synthetic and real-world image databases. The method has linear time complexity in the M. E. Celebi (&) Department of Computer Science, Louisiana State University, Shreveport, LA, USA e-mail: ecelebi@lsus.edu

Editorial: Journal of Real-Time Image Processing—first issue of volume 10

The Journal of Real-Time Image Processing is entering its 10th year of publication with a steady increase in its stature in the area of image processing. The number of downloads grew to 37K? in 2014. There are currently six special issue calls for papers, which can be viewed in the back matter of this issue. In the past few years, changes have been made to the Editorial Board to incorporate the technological trends and challenges in the field of real-time image processing. In addition, to improve the geographical representation of manuscripts being submitted, changes are being made to the Advisory Board. As a first step, we are delighted to inform our readers that Professor Touradj Ebrahimi, who is a well-known researcher and technology leader with extensive experience in image processing, has kindly agreed to join the Advisory Board. It is a great pleasure to have him in our Advisory Board. The editorial board meeting of JRTIP (see Fig. 1) was recently held in San Francisco in February 2015 during the IS&T/SPIE Conference on Real-Time Image and Video Processing discussing various issues related to the journal. The outcome of this meeting will be noted in the next regular issue editorial. As indicated in the editorial of the last regular issue, the number of pages of JRTIP is now increased to 800 pages per year. This is the reason this issue has more than the normal number of pages or papers compared to the previous volumes. This increase in the number of pages is expected to ease the backlog of papers that are already on Online First and waiting to appear in print. One item to note is that considering that the number of requests for offering special issues has been steadily growing, we have put together a set of guidelines or rules to help potential guest editors with offering and conducting special issues. These guidelines can be viewed under the Call-For-Papers link at the journal website: http://static. springer.com/sgw/documents/1488308/application/pdf/Spe cialIssueGuidelines_final.pdf. It is quite helpful for all interested guest editors to read these guidelines before sending emails to the Editors-in-Chiefs regarding the offering of special issues. Before providing an overview of papers in this regular issue, it is worth pointing out that the next issue or second issue of volume 10 will be a special issue entitled ‘‘RealTime Color Image Processing’’. There are a total of 13 papers in this regular issue. The first paper by Khalvati et al. presents a technique named window memoization for software implementation of local image processing algorithms. This technique has been applied to six image processing algorithms and it is shown that speedups ranging from 1.2 to 7.9 are achieved while the total average speedup for different algorithms with different input images is 3.95. The second paper by Karas et al. provides a review of parallel algorithms for the computation of linear opening and closing morphological operations followed by an implementation comparison of four state-of-the-art algorithms. Then, the GPU implementations of two recent algorithms are described. The experimental results show that the proposed GPU implementation is suitable for real-time processing of industrial applications involving large images. The third paper by Rodriguez-Gomez et al. describes a hardware computing engine to perform background N. Kehtarnavaz (&) University of Texas at Dallas, Richardson, USA e-mail: kehtar@utdallas.edu