A Raw Image-Based End-to-End Object Detection Accelerator Using HOG Features

Abstract

This paper presents an end-to-end object-detection accelerator that processes raw Bayer images to generate detection results. The accelerator utilizes histogram of oriented gradients (HOG) features in combination with a support vector machine (SVM) classifier. The proposed HOG for raw images (HOGR) skips the image signal processors which consume a significant amount of power (about 2.5X of an accelerator). The proposed architecture temporally partitions the algorithms and time multiplexes the logic such that the accelerator works on the same frequency with the image sensor using affordable resources. The prototype is verified using 1080p ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1920\times 1080$ </tex-math></inline-formula> ) and VGA ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480$ </tex-math></inline-formula> ) raw videos on Altera Arria10 and Cyclone IV field programmable gate array (FPGA) platforms. The accelerator can process 1080p raw videos with a 12-scale pyramid at 60 frames per second (FPS) under the pixel frequency of corresponding image sensor (148.5 MHz), consuming 510 Kbit block memory. To the best knowledge of the authors, this is the first end-to-end HOG+SVM accelerator that takes raw Bayer images as input, skips the ISP pipeline for resource optimization from the system perspective, and is synchronized with the image sensor.