Image Buffer Research Articles

Contributed Session I: Towards General Video Percepts Cone-by-Cone.

We aim to reprogram visual perception through an Adaptive Optics Scanning Laser Ophthalmoscope (AOSLO) Display, using a GPU renderer that rasterizes a target color image or video into cone-by-cone single-wavelength laser light pulses ("microdoses"). We imaged and tracked at ~ 2° eccentricity a 0.9° x 0.9° field of view of the retina in 840 nm. Stimulated in 543 nm, all resolved, spectrally classified cones receive microdoses of varying intensities. The renderer updates for each AOSLO frame (30 frames / sec) an underlying stimulation image buffer, encoding a desired color percept pattern that takes into account the cone locations, cone spectral sensitivity to the 543 nm stimulation light, and the corresponding color percept pixel values. Within one frame, the buffer gets pixelated strip-by-strip at 1 kHz into actual world-fixed microdose intensity values, each centered on a cone within that strip at that instant. The resulting frame of microdoses visually occupies the whole raster view. We showed multiple color percepts to a cone-classified subject, with logging data. The subject saw spatially-varying colors, e.g. a red box moving on a green canvas - these percepts validated the accuracy of the prototype. These initial prototyping experiments allude to the potential of presenting general percepts to a cone-classified subject, at cone-level accuracy in a fully programmable way. The technology allows us to probe neural plasticity and towards generation of novel percepts.

Image buffering in application specific processors

In many digital image-processingapplications, which are implementedin field programmable gate arrays,the currently processed image's frames are stored in external dynamic memory.The performance of such an application dependson the dynamic memoryspeed and the necessaryrequests quantity during algorithm’sruntime. This performance is being optimized through field programmable gate arrays -implemented buffer memory usage.But there is no common method for the formal buffer memory synthesis with preset throughput, input and output data sequenceorderand minimizedhardwarecosts.In this article,the featuresof image input and processing based on Field Programmable Gate Arrayareconsidered.The methods of building buffer circuits in field programmable gate arrays, due to which the intensity of data exchanges with external memory is reduced, are analyzed. Themethod of synthesizing pipeline circuits with specified performance characteristics and the data sequence order is given, which is based on the mapping of the spatial synchronous data flows into the structure implemented in the field programmable gate arrays.A method of designing buffer schemes is proposed, which is based on the mapping of spatial synchronous data flows into local memory in the form of chains of pipeline registers.The method helpsto organize the data flow of at the input of built-in pipeline units of image processing, in which the data follow in a given order, andto minimize the amount of buffer memory.The method ensures the use of dynamically adjustable register delays built into the field programmable gate arrays, which increases the efficiency of buffering.Thismethod was tested during the development of an intelligent video camera. The embedded hardware implements a video image compression algorithm with a wide dynamic range according to the Retinexalgorithm. The same time it selects characteristic points in the image for the further pattern recognition.At the same time, multiple decimation of the frame is performed. Due to themultirate buffering of the image in the field programmable gate arrays,it was possible to avoid using of external dynamic memory.

A Novel Power-Saving Reversing Camera System with Artificial Intelligence Object Detection

According to a study by the Insurance Institute for Highway Safety (IIHS), the driving collision rate of using only the reversing camera system is lower than that of using both the reversing camera system and the reversing radar. In this article, we implemented a reversing camera system with artificial intelligence object detection to increase the information of the reversing image. Our system consists of an image processing chip (IPC) with wide-angle image distortion correction and an image buffer controller, a low-power KL520 chip and an optimized artificial intelligence model MobileNetV2-YOLOV3-Optimized (MNYLO). The results of the experiment show the three advantages of our system. Firstly, through the image distortion correction of IPC, we can restore the distorted reversing image. Secondly, by using a public dataset and collected images of various weathers for artificial intelligence model training, our system does not need to use image algorithms that eliminate bad weathers such as rain, fog, and snow to restore polluted images. Objects can still be detected by our system in images contaminated by weather. Thirdly, compared with the AI model Tiny_YOLOV3, not only the parameters of our MNYLO have been reduced by 72.3%, the amount of calculation has been reduced by 86.4%, but the object detection rate has also been maintained and avoided sharp drops.

TeleParallax: Low-Motion-Blur Stereoscopic System With Correct Interpupillary Distance for 3D Head Rotations

Binocular parallax provides cues for depth information when a scene is viewed with both eyes. In visual telepresence systems, stereo cameras are commonly used to simulate human eyes. However, motion blur occurs when these cameras are rotated quickly. The use of omnidirectional cameras can reduce the motion blur, but does not provide the correct interpupillary distance (IPD) when viewers tilt or turn their heads sideways. We propose a method called TeleParallax, in which two omnidirectional cameras are separated by the IPD and the direction of the lenses are kept constant in world coordinates by robotic arms during three-dimensional head rotations. TeleParallax can suppress the increase in image buffering during head rotations because each camera can capture an omnidirectional image with the lens direction fixed. We conducted three user studies to evaluate the perceptual effect of head tilt, eye asynchrony, and delays in IPD correction for a particular rotation. The results indicate that TeleParallax can provide depth perception that is independent of the head movement with less visual discomfort. Although the results show that the users were sensitive to the asynchrony between their eyes and to camera motion during IPDs, they retained the feeling of depth perception within interocular delays of 70 ms and motion velocity of 75°/s. These results imply that TeleParallax has remarkable potential for visual telepresence systems.

Camera Obscura: Exploiting in-camera processing for image counter forensics

For the last two decades Image Forensics has been providing an arsenal of forensic tools to detect tampered images. At the same time, anti-forensics technologies kept evolving to mislead forensic detectors. Such attacks are generally designed to affect a single forensic trace (e.g. JPEG compression, sensor patter noise, histogram statistics) without considering that the image alteration can negatively affect other traces, thus making harder to mimic multiple statistics of natural images. Nevertheless, performing of several attacks can ruin the image quality. In this paper we introduce Camera Obscura, a method that, given a tampered picture, generates a pristine image with native camera metadata, JPEG structure, Quantization Tables, Preview, Thumbnails, their corresponding Quantization Tables, single compression statistics and any in-camera (even proprietary) processing. The attack is performed by exploiting and extending Magic Lantern, an alternative camera firmware, to perform in-camera image buffer substitution. As opposed to available image anti-forensic techniques, Camera Obscura accurately reproduces multiple image statistics in one attack with a limited effect on global image quality. With these characteristics, it can be applied for many different applications such as injecting reference image statistics into a computer generated image, exchange the supposed source of a given image or hide any signal-based editing operation.

Deep combiner for independent and correlated pixel estimates

Monte Carlo integration is an efficient method to solve a high-dimensional integral in light transport simulation, but it typically produces noisy images due to its stochastic nature. Many existing methods, such as image denoising and gradient-domain reconstruction, aim to mitigate this noise by introducing some form of correlation among pixels. While those existing methods reduce noise, they are known to still suffer from method-specific residual noise or systematic errors. We propose a unified framework that reduces such remaining errors. Our framework takes a pair of images, one with independent estimates, and the other with the corresponding correlated estimates. Correlated pixel estimates are generated by various existing methods such as denoising and gradient-domain rendering. Our framework then combines the two images via a novel combination kernel. We model our combination kernel as a weighting function with a deep neural network that exploits the correlation among pixel estimates. To improve the robustness of our framework for outliers, we additionally propose an extension to handle multiple image buffers. The results demonstrate that our unified framework can successfully reduce the error of existing methods while treating them as black-boxes.

X—Ray Digital Image Advanced Processing and Buffer Layer Defect Intelligent Identification of Power Cable

With the rapid modernization of the city, power cable has been widely used in the process of urban construction. In the course of cable operation, cable faults become more and more frequent, which has a great impact on national economy and people’s life. The method of digital X-ray imaging can realize the nondestructive testing of power cable body and obtain clear and intuitive X-ray digital image. But it lacks the advanced processing and defect recognition method of X-ray digital image, and can not directly detect and identify the cable body and defect from the original digital image. Therefore, this paper studies the power cable X-ray digital image advanced processing and buffer layer defect intelligent identification technology. By using gray scale processing technology, the original image gray scale range is compressed to the human eye identifiable range, and then the defect identification is carried out. Then the traditional convolution neural network CNN and the full convolution neural network FCN are used to train the image data to realize the intelligent recognition of the power cable buffer layer defect. The research shows that compared with the traditional convolution neural network CNN, the full convolution neural network FCN proposed in this paper has more clear and intuitive recognition effect.

Resource-Efficient Image Buffer Architecture for Neighborhood Processors

Neighborhood image processing operations on Field Programmable Gate Array (FPGA) are considered as memory intensive operations. A large memory bandwidth is required to transfer the required pixel data from external memory to the processing unit. On-chip image buffers are employed to reduce this data transfer rate. Conventional image buffers, implemented either by using FPGA logic resources or embedded memories are resource inefficient. They exhaust the limited FPGA resources quickly. Consequently, hardware implementation of neighborhood operations becomes expensive, and integrating them in resource constrained devices becomes unfeasible. This paper presents a resource efficient FPGA based on-chip buffer architecture. The proposed architecture utilizes full capacity of a single Xilinx Block-RAM (BRAM36 primitive) for storing multiple rows of input image. To get multiple pixels/clock in a user defined scan order, an efficient duty-cycle based memory accessing technique is coupled with a customized addressing circuitry. This accessing technique exploits switching capabilities of BRAM to read 4 pixels in a single clock cycle without degrading system frequency. The addressing circuitry provides multiple pixels/clock in any user defined scan order to implement a wide range of neighborhood operations. With the saving of 83% BRAM resources, the buffer architecture operates at 278 MHz on Xilinx Artix-7 FPGA with an efficiency of 1.3 clock/pixel. It is thus capable to fulfill real time image processing requirements for HD image resolution ( $1080\times 1920$ ) @103 fps.

MLC STT-MRAM-Aware Memory Subsystem for Smart Image Applications

Next-generation memories with high storage capacity, high performance, and low power consumption are being researched due to the ever-growing demand for artificial intelligence and high-definition applications. Among such future memories, a multi-level cell (MLC) spin-transfer magnetic torque random access memory (STT-MRAM) attracts considerable attention as an alternative to static or dynamic random access memories. An MCL STT-MRAM has the advantages of capacity and non-volatility, but the disadvantages of performance, power consumption, and endurance resulting from complicated resistance state transition and detection processes. In particular, such issues are exacerbated in the latest smart image applications employing block-based processing algorithms. In this paper, we propose a memory subsystem that mitigates the MLC STT-MRAM disadvantages in smart image applications. Our main idea is threefold: MLC-aware image buffer composing, block-aware pixel-to-memory mapping, and prediction-aware image-to-buffer allocating techniques that all make multi-step resistance state transition and detection processes less required. Experimental results show that the proposed memory subsystem achieves 24.5% shorter application execution time, and 96.4% lower memory power consumption than the conventional memory subsystems for industrial smart image applications. In addition, our memory subsystem increases the lifetime of MLC STT-MRAMs via 93.8% fewer multi-step resistance state transition processes.

Gimbal Tracking Control with Delayed Feedback of Target Information

In this paper, a power limited platform equipped with a gimbaled camera communicating with a remote station is considered. Sequence of the acquired images is downloaded to the station, and it specifies target of interest in the images. Then, target information, i.e., pixel coordinate of target in image, is sent back to the camera so image processor built in it can start target tracking and gimbal control. In this way, the burdensome task of specifying target is offloaded to the station, where task of tracking the specified target is operated by the camera. The target information is sent only once per target when the station assigns or reassigns the target to monitor. However, delay of the target information, invoked by the bandwidth limitation of the channel to the station, can disturb the image processor to start tracking. Therefore, we propose a method to remove the delay, using two image tracking tasks and an image buffer built in the camera. In addition, using the method, speed of the gimbal can be adjusted for reducing the risk of missing the target due to the camera motion.

A 1920 $\times$ 1080 25-Frames/s 2.4-TOPS/W Low-Power 6-D Vision Processor for Unified Optical Flow and Stereo Depth With Semi-Global Matching

This paper presents a unified 6-D vision processor that enables dense real-time 3-D depth and 3-D motion perception at full-high-definition (1920 × 1080, FHD) resolution. The proposed design implements a neighbor-guided semi-global matching (NG-SGM) algorithm to unify the stereo depth and optical flow matching problem and to reduce computation by 98% compared with the original SGM. We introduce a new custom-designed, high-bandwidth coalescing crossbar circuit that automatically coalesces redundant memory accesses to mitigate the highly irregular memory accesses observed in NG-SGM. The proposed 6-D vision processor also maximizes on-chip memory reuse by using 64 on-chip rotating image buffers that cover a wide optical flow and depth disparity search range of 176 pixels per dimension. The processor implements massive parallel processing with 576 compute units that are deeply pipelined with a dependency-resolving skewed-diagonal scan to hide the dynamic and variable dependency in the pipeline. The fabricated processor performs dense NG-SGM at 25 frames/s for optical flow or 30 frames/s for stereo depth at FHD resolution while consuming only 760 mW in 28-nm CMOS.

A Novel Software-Defined Convolutional Neural Networks Accelerator

The convolutional neural network (CNN) has been widely applied in computer vision applications. Due to the intensive computation, the general central processing unit (CPU) processors are not efficient to meet the real-time requirement. Various hardware accelerators based on the application specific integrated circuit (ASIC) and field programmable gate array (FPGA) have been designed to accelerate CNN models during the inference phase. The data flow architecture has been extensively adopted because of the high parallelism. However, given the continual development in the computer vision field, CNN models have become increasingly diverse. Thus, CNN accelerators based on data flow architectures face an emerging challenge to maintain high throughput while coping with various CNN models. In this paper, we design a software-defined architecture to solve this. The goal of this study is to make the hardware change as the application changes to achieve high flexibility and high performance. In our proposed architecture, all the parts can be software-defined to cope with different CNN models. A flexible software-defined processing element (PE) array is designed to compute different weight filter sizes. In addition, a software-defined data reuse technique based on two ideal reuse cases is proposed to ensure that all the parameters need to be loaded only once during the computing phase. To support this reuse technique, we also propose the software-defined on-chip buffer so that the weight and image buffers share one dynamic buffer. By using the sparsity property of the input feature map, the full-connected (FC) layer is accelerated. About 88% of the FC weight parameters can be skipped when loading the VGG-16 model. Finally, we implemented this software-defined accelerator on the FPGA. Compared to the other FPGA based accelerators, our proposed accelerator can preserve high performance while maintaining flexibility.

JANUS: A Compilation System for Balancing Parallelism and Performance in OpenVX

Embedded systems typically do not have enough on-chip memory for entire an image buffer. Programming systems like OpenCV operate on entire image frames at each step, making them use excessive memory bandwidth and power. In contrast, the paradigm used by OpenVX is much more efficient; it uses image tiling, and the compilation system is allowed to analyze and optimize the operation sequence, specified as a compute graph, before doing any pixel processing. In this work, we are building a compilation system for OpenVX that can analyze and optimize the compute graph to take advantage of parallel resources in many-core systems or FPGAs. Using a database of prewritten OpenVX kernels, it automatically adjusts the image tile size as well as using kernel duplication and coalescing to meet a defined area (resource) target, or to meet a specified throughput target. This allows a single compute graph to target implementations with a wide range of performance needs or capabilities, e.g. from handheld to datacenter, that use minimal resources and power to reach the performance target.

Mapping Bahrain’s subsurface municipal solid waste

The objective of this paper is to map possible subsurface dumpsites of solid waste in the Kingdom of Bahrain. Due to limited land availability, random garbage dumping for waste disposal and reclamation has occurred in the past and continues to occur in the Kingdom. Consequently, the process causes economic stress, creates environmental hazards, and contributes to the failure of many dwellings. The input data used to achieve research objective includes data from 300 boreholes, aerial photos, LandSat images, digital elevation model, and 1969–2015 land use maps. Data on the amounts and types of municipal solid waste (MSW) were collected from 1997 to 2017, as were the locations of 3665 deteriorated houses. All types of data are screened, georeferenced, and converted into geographic information system (GIS) formats. At first, a map of possible dumping areas is produced based on citizen testimonies, government engineer interviews, LandSat, aerial photos, and 1969 topo map, followed by data analysis of the aforementioned input data performed such as image classification, buffering, and others. In the next step, multi-criteria GIS algorithm was developed to identify, locate, and map possible sites. The criteria used are distances from dwelling outskirt and the shoreline, land use, relative elevation, and soil type based on borehole data. Lastly, ground truth validation assesses the accuracy of the obtained result. Overall, the study identified 101 confirmed dumping sites and 590 probable dumping sites. Site size is ranging from 18,000 m2 to 3.6 km2 distributed over 100 city blocks which are spread throughout the Kingdom. The Capital Governorate is the most affected by dumping, and the Southern Governorate is the least impacted. A comparison of results and ground observations yielded 87.5% accuracy. The proposed integrated method was effective, and the obtained identification and location accuracy were acceptable. The method can be used to identify, locate, and map random dumping sites in a similar situation.

Methodology for Quantitative Characterization of Fluorophore Photoswitching to Predict Superresolution Microscopy Image Quality.

Single-molecule localization microscopy (SMLM) image quality and resolution strongly depend on the photoswitching properties of fluorophores used for sample labeling. Development of fluorophores with optimized photoswitching will considerably improve SMLM spatial and spectral resolution. Currently, evaluating fluorophore photoswitching requires protein-conjugation before assessment mandating specific fluorophore functionality, which is a major hurdle for systematic characterization. Herein, we validated polyvinyl alcohol (PVA) as a single-molecule environment to efficiently quantify the photoswitching properties of fluorophores and identified photoswitching properties predictive of quality SMLM images. We demonstrated that the same fluorophore photoswitching properties measured in PVA films and using antibody adsorption, a protein-conjugation environment analogous to labeled cells, were significantly correlated to microtubule width and continuity, surrogate measures of SMLM image quality. Defining PVA as a fluorophore photoswitching screening platform will facilitate SMLM fluorophore development and optimal image buffer assessment through facile and accurate photoswitching property characterization, which translates to SMLM fluorophore imaging performance.

Context-based image acquisition from memory in digital systems

A key consideration in the design of image and video processing systems is the ever increasing spatial resolution of the captured images, which has a major impact on the performance requirements of the memory subsystem. This is further amplified by the facts that the memory bandwidth requirements and energy consumption of accessing the captured images have started to become the bottlenecks in the design of high-performance image processing systems. Inspired by the successful application of progressive image sampling techniques in various image processing tasks, this work proposes the concept of Context-based Image Acquisition for hardware systems that efficiently trades image quality for reduced cost of the image acquisition process. Based on the proposed framework, a hardware architecture is developed which alters the conventional memory access pattern, to progressively and adaptively access pixels from a memory subsystem. The sampled pixels are used to reconstruct an approximation to the ground truth, which is stored in a high-performance image buffer for further processing. An instance of the architecture is prototyped on an FPGA and its performance evaluation shows that a saving of up to 85 % of memory accessing time and 33 %/45 % of image acquisition time/energy are achieved on a set of benchmarks while maintaining a high PSNR.

The high performance readout chain for the DSSC 1Megapixel detector, designed for high throughput during pulsed operation mode

The readout chain of the DSSC 1M pixel detector currently built at DESY, Hamburg for the European X-Ray Free Electron Laser is described. The systemoperates in pulsed operation mode comparable to the new ILC. Each 0.1seconds 800 images of 1M pixels are produced and readout by the DSSCDAQ electronics. The total data production rate of the system is about134 Gbit/s.In order to deal with the high data rates, latest technologycomponents like the Xilinx Kintex 7 FPGA are used to implement fastDDR3-1600 image buffers, high speed serial FPGA to FPGA communicationand 10 GB Ethernet links concentrated in one 40 Gbit/sQSFP+ transceiver.

FPGA Based Compact and Efficient Full Image Buffering for Neighborhood Operations

Image processing systems based on neighborhood operations i.e. Neighborhood Processing Systems (NPSs) are computationally expensive and memory intensive. Field Programmable Gate Array ...

View synthesis distortion model based frame level rate control optimization for multiview depth video coding

Quality of virtual view image is one of the important issues in multiview depth video coding (MDVC). When people enjoy three dimensional (3D) applications, they may have an unpleasant perceptual experience if the virtual view has low image quality. Thus, high virtual view image quality is mostly expected in the rate control optimization for MDVC with limited channel bandwidth and buffer resources. In this paper, a view synthesis distortion model is proposed first to indicate the importance of each frame in the depth video. Second, to achieve a balance between virtual view image quality and buffer constraint, the proposed view synthesis distortion model is incorporated in the bargain game theoretic model to handle the frame level bit allocation problem for Hierarchical B-picture (HBP) structure. Experimental results demonstrate that the proposed rate control algorithm achieves 0.18dB BDPSNR gain in average compared to the benchmark. The average mismatch of the proposed algorithm is only 0.71%.

Generalized Nash Bargaining Solution to Rate Control Optimization for Spatial Scalable Video Coding.

Rate control (RC) optimization is indispensable for scalable video coding (SVC) with respect to bitstream storage and video streaming usage. From the perspective of centralized resource allocation optimization, the inner-layer bit allocation problem is similar to the bargaining problem. Therefore, bargaining game theory can be employed to improve the RC performance for spatial SVC. In this paper, we propose a bargaining game based one-pass RC scheme for spatial H.264/SVC. In each spatial layer (SL), the encoding constraints, such as bit rates, buffer size are jointly modeled as resources in the inner-layer bit allocation bargaining game. The modified rate-distortion (R-D) model incorporated with the inter-layer coding information is investigated. Then the generalized Nash bargaining solution (NBS) is employed to achieve an optimal bit allocation solution. The bandwidth is allocated to the frames from the generalized NBS adaptively based on their own bargaining powers. Experimental results demonstrate that the proposed rate control algorithm achieves appealing image quality improvement and buffer smoothness. The average mismatch of our proposed algorithm is within the range of 0:19%2:63%.