Embedded Smart Camera Research Articles

$$\text{C}^{3}\text{Net}$$: end-to-end deep learning for efficient real-time visual active camera control

The need for automated real-time visual systems in applications such as smart camera surveillance, smart environments, and drones necessitates the improvement of methods for visual active monitoring and control. Traditionally, the active monitoring task has been handled through a pipeline of modules such as detection, filtering, and control. However, such methods are difficult to jointly optimize and tune their various parameters for real-time processing in resource constraint systems. In this paper a deep Convolutional Camera Controller Neural Network is proposed to go directly from visual information to camera movement to provide an efficient solution to the active vision problem. It is trained end-to-end without bounding box annotations to control a camera and follow multiple targets from raw pixel values. Evaluation through both a simulation framework and real experimental setup, indicate that the proposed solution is robust to varying conditions and able to achieve better monitoring performance than traditional approaches both in terms of number of targets monitored as well as in effective monitoring time. The advantage of the proposed approach is that it is computationally less demanding and can run at over 10 FPS (~4x speedup) on an embedded smart camera providing a practical and affordable solution to real-time active monitoring.

Accelerating object detection via a visual-feature-directed search cascade: algorithm and field programmable gate array implementation

Object detection is a major step in several computer vision applications and a requirement for most smart camera systems. Recent advances in hardware acceleration for real-time object detection feature extensive use of reconfigurable hardware [field programmable gate arrays (FPGAs)], and relevant research has produced quite fascinating results, in both the accuracy of the detection algorithms as well as the performance in terms of frames per second (fps) for use in embedded smart camera systems. Detecting objects in images, however, is a daunting task and often involves hardware-inefficient steps, both in terms of the datapath design and in terms of input/output and memory access patterns. We present how a visual-feature-directed search cascade composed of motion detection, depth computation, and edge detection, can have a significant impact in reducing the data that needs to be examined by the classification engine for the presence of an object of interest. Experimental results on a Spartan 6 FPGA platform for face detection indicate data search reduction of up to 95%, which results in the system being able to process up to 50 1024×768 pixels images per second with a significantly reduced number of false positives.

Real-time multiple objects tracking on Raspberry-Pi-based smart embedded camera

Multiple-object tracking constitutes a major step in several computer vision applications, such as surveillance, advanced driver assistance systems, and automatic traffic monitoring. Because of the number of cameras used to cover a large area, these applications are constrained by the cost of each node, the power consumption, the robustness of the tracking, the processing time, and the ease of deployment of the system. To meet these challenges, the use of low-power and low-cost embedded vision platforms to achieve reliable tracking becomes essential in networks of cameras. We propose a tracking pipeline that is designed for fixed smart cameras and which can handle occlusions between objects. We show that the proposed pipeline reaches real-time processing on a low-cost embedded smart camera composed of a Raspberry-Pi board and a RaspiCam camera. The tracking quality and the processing speed obtained with the proposed pipeline are evaluated on publicly available datasets and compared to the state-of-the-art methods.

Yawning Detection Using Embedded Smart Cameras

Yawning detection has a variety of important applications in a driver fatigue detection, well-being assessment of humans, driving behavior monitoring, operator attentiveness detection, and understanding the intentions of a person with a tongue disability. In all of the above applications, an automatic detection of yawning is one important system component. In this paper, we design and implement such automatic system, using computer vision, which runs on a computationally limited embedded smart camera platform to detect yawning. We use a significantly modified implementation of the Viola-Jones algorithm for face and mouth detections and, then, use a backprojection theory for measuring both the rate and the amount of the changes in the mouth, in order to detect yawning. As proof-of-concept, we have also implemented and tested our system on top of an actual smart camera embedded platform, called APEX from CogniVue Corporation. In our design and implementations, we took into consideration the practical aspects that many existing works ignore, such as real-time requirements of the system, as well as the limited processing power, memory, and computing capabilities of the embedded platform. Comparisons with existing methods show significant improvements in the correct yawning detection rate obtained by our proposed method.

Exploration of preprocessing architectures for field-programmable gate array-based thermal-visual smart camera

Embedded smart cameras are gaining in popularity for a number of real-Time outdoor surveillance applications. However, there are still challenges, i.e., computational latency, variation in illumina ...

Robust and Computationally Lightweight Autonomous Tracking of Vehicle Taillights and Signal Detection by Embedded Smart Cameras

An important aspect of collision avoidance and driver assistance systems, as well as autonomous vehicles, is the tracking of vehicle taillights and the detection of alert signals (turns and brakes). In this paper, we present the design and implementation of a robust and computationally lightweight algorithm for a real-time vision system, capable of detecting and tracking vehicle taillights, recognizing common alert signals using a vehicle-mounted embedded smart camera, and counting the cars passing on both sides of the vehicle. The system is low-power and processes scenes entirely on the microprocessor of an embedded smart camera. In contrast to most existing work that addresses either daytime <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>or</i></b> nighttime detection, the presented system provides the ability to track vehicle taillights <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>and</i></b> detect alert signals regardless of lighting conditions. The mobile vision system has been tested in actual traffic scenes and the results obtained demonstrate the performance and the lightweight nature of the algorithm.

Mobile Standards-Based Traffic Light Detection in Assistive Devices for Individuals with Color-Vision Deficiency

Considering the substantial population affected by some form of color-vision deficiency (CVD), reliable traffic control signal head light detection is an important problem for driver-assistance systems. While a large number of technologies can be used to localize traffic lights, without drastic changes in infrastructure, only visual information can be used in identifying the status of the light. In addition, traffic light detection is not currently integrated into any driver-assistance systems, making driving for individuals with CVD (where permitted) dangerous to other drivers, pedestrians, and themselves. This paper presents a robust, traffic-standards-based, and computationally efficient method for detecting the status of the traffic lights without relying on Global Positioning System, lidar, radar information, or prior (map-based) knowledge. To the extent of our knowledge, this is the first work to use official Institute of Transportation Engineers (U.S.) and British Standards Institute (European Union) standards for defining traffic light colors, as well as integrating a number of fail-safe mechanisms designed to prevent erroneous detection. The algorithm can be easily ported over to an embedded smart camera platform and used as a windshield-mounted driver-assistance device by individuals with CVD. The system can accurately identify the status of the light at 400 ft away from the intersection, reliably detecting solid, faulty, arrow, and high-visibility signal lights. Over 50 h of video (over 2000 intersections) were tested with the system, containing intersections with one to four traffic lights, governing different lanes of traffic, with 97.5% accuracy of solid light detection.

Efficient network clustering for traffic reduction in embedded smart camera networks

In this work, a clustering approach for bandwidth reduction in distributed smart camera networks is presented. Properties of the environment such as camera positions and environment pathways, as well as dynamics and features of targets are used to limit the flood of messages in the network. To better understand the correlation between camera positioning and pathways in the scene on one hand and temporal and spatial properties of targets on the other hand, and to devise a sound messaging infrastructure, a unifying probabilistic modeling for object association across multiple cameras with disjointed view is used. Communication is efficiently handled using a task-oriented node clustering that partition the network in different groups according to the pathway among cameras, and the appearance and temporal behavior of targets. We propose a novel asynchronous event exchange strategy to handle sporadic messages generated by non-frequent tasks in a distributed tracking application. Using a Xilinx-FPGA with embedded Microblaze processor, we could show that, with limited resource and speed, the embedded processor was able to sustain a high communication load, while performing complex image processing computations.

Automatic Fall Detection and Activity Classification by a Wearable Embedded Smart Camera

Robust detection of events and activities, such as falling, sitting, and lying down, is a key to a reliable elderly activity monitoring system. While fast and precise detection of falls is critical in providing immediate medical attention, other activities like sitting and lying down can provide valuable information for early diagnosis of potential health problems. In this paper, we present a fall detection and activity classification system using wearable cameras. Since the camera is worn by the subject, monitoring is not limited to confined areas, and extends to wherever the subject may go including indoors and outdoors. Furthermore, since the captured images are not of the subject, privacy concerns are alleviated. We present a fall detection algorithm employing histograms of edge orientations and strengths, and propose an optical flow-based method for activity classification. The first set of experiments has been performed with prerecorded video sequences from eight different subjects wearing a camera on their waist. Each subject performed around 40 trials, which included falling, sitting, and lying down. Moreover, an embedded smart camera implementation of the algorithm was also tested on a CITRIC platform with subjects wearing the CITRIC camera, and each performing 50 falls and 30 non-fall activities. Experimental results show the success of the proposed method.

P-FAD: Real-Time Face Detection Scheme on Embedded Smart Cameras

Face detection on general embedded devices is fundamentally different from the conventional approach on personal computer or consumer digital camera due to the limited computation and power capacity. The resource-limited characteristic gives rise to new challenges for implementing a real-time video surveillance system with smart cameras. In this work, we present the design and implementation of Pyramid-like FAce Detection (P-FAD), a real-time face detection system constructed on general embedded devices. Motivated by the observation that the computation overhead increases proportionally to its pixel manipulation, P-FAD proposes a hierarchical approach to shift the complex computation to the promising regions. More specifically, P-FAD present a three-stage coarse, shift, and refine procedure, to construct a pyramid-like detection framework for reducing the computation overhead significantly. This framework also strikes a balance between the detection speed and accuracy. We have implemented P-FAD on notebook, Android phone and our embedded smart camera platform. An extensive system evaluation in terms of detailed experimental and simulation results is provided. Our empirical evaluation shows that P-FAD outperforms V-J detector calibrated color detector (VJ-CD) and color detector followed by a V-J detector (CD-VJ), the state of the art real-time face detection techniques by 4.7 -8.6 on notebook and by up to 8.2 on smart phone in terms of thembedded smart camera platfoe detection speed.

Towards commoditized smart-camera design

We propose a set of design principles for a cost-effective embedded smart camera. Our aim is to alleviate the shortcomings of the existing designs, such as excessive reliance on battery power and wireless networking, over-emphasized focus on specific use cases, and use of specialized technologies. In our opinion, these shortcomings prevent widespread commercialization and adoption of embedded smart cameras, especially in the context of visual-sensor networks. The proposed principles lead to a distinctively different design, which relies on commoditized, standardized and widely-available components, tools and knowledge. As an example of using these principles in practice, we present a smart camera, which is inexpensive, easy to build and support, capable of high-speed communication and enables rapid transfer of computer-vision algorithms to the embedded world.

Design and DSP Optimization of Real-time Multi-Camera Tracking

In recent years there is a dominanttrend towards the deploymentof advanced visual analysisalgorithms on embedded platforms. However, this deployment is very challenging as embedded systems usually afford limited resources such as calculating performance, memory and power. Therefore, to address this problem, In this paper we introduces anoptimized multi-camera tracking system based on multiple TI DSP TMS320DM6446for trackingobjects across multiple embedded smart cameras which embedsintelligent processors. Firstly, the overlapping areas and homography projection relations between adjacent cameras’ field of view is calculated. Based on the relations of cameras view obtained and tracked data of each single camera,a homography based target handover procedure is done for long-term multi-camera tracking.After that, we fully implemented the tracking system on the embedded smart cameras developed by our group.Finally, to combat the huge computational complexity, a novel hierarchical optimization method is proposed. Experimental results demonstrate the robustness and real-time efficiency in dynamic real-life environments and the computationalburden is significantly optimized by 98.84% which is low enough for further biometrics tasks such as recognition.

Securing Embedded Smart Cameras with Trusted Computing

Camera systems are used in many applications including video surveillance for crime prevention and investigation, traffic monitoring on highways or building monitoring and automation. With the shift from analog towards digital systems, the capabilities of cameras are constantly increasing. Today's smart camera systems come with considerable computing power, large memory, and wired or wireless communication interfaces. With onboard image processing and analysis capabilities, cameras not only open new possibilities but also raise new challenges. Often overlooked are potential security issues of the camera system. The increasing amount of software running on the cameras turns them into attractive targets for attackers. Therefore, the protection of camera devices and delivered data is of critical importance. In this work we present an embedded camera prototype that uses Trusted Computing to provide security guarantees for streamed videos. With a hardware-based security solution, we ensure integrity, authenticity, and confidentiality of videos. Furthermore, we incorporate image timestamping, detection of platform reboots, and reporting of the system status. This work is not limited to theoretical considerations but also describes the implementation of a prototype system. Extensive evaluation results illustrate the practical feasibility of the approach.

Cooperative Object Tracking and Composite Event Detection With Wireless Embedded Smart Cameras

Embedded smart cameras have limited processing power, memory, energy, and bandwidth. Thus, many system- and algorithm-wise challenges remain to be addressed to have operational, battery-powered wireless smart-camera networks. We present a wireless embedded smart-camera system for cooperative object tracking and detection of composite events spanning multiple camera views. Each camera is a CITRIC mote consisting of a camera board and wireless mote. Lightweight and robust foreground detection and tracking algorithms are implemented on the camera boards. Cameras exchange small-sized data wirelessly in a peer-to-peer manner. Instead of transferring or saving every frame or trajectory, events of interest are detected. Simpler events are combined in a time sequence to define semantically higher-level events. Event complexity can be increased by increasing the number of primitives and/or number of camera views they span. Examples of consistently tracking objects across different cameras, updating location of occluded/lost objects from other cameras, and detecting composite events spanning two or three camera views, are presented. All the processing is performed on camera boards. Operating current plots of smart cameras, obtained when performing different tasks, are also presented. Power consumption is analyzed based upon these measurements.

Light-weight salient foreground detection for embedded smart cameras

An embedded smart camera is a stand-alone unit that not only captures images, but also includes a processor, memory and communication interface. Battery-powered, embedded smart cameras introduce many additional challenges since they have very limited resources, such as energy, processing power and memory. Computer vision algorithms running on these camera boards should be light-weight and efficient. Considering the memory requirements of an algorithm and its portability to an embedded processor should be an integral part of the algorithm design in addition to the accuracy requirements. This paper presents a light-weight and efficient background modeling and foreground detection algorithm that is highly robust against lighting variations and non-static backgrounds including scenes with swaying trees, water fountains and rain. Compared to many traditional methods, the memory requirement for the data saved for each pixel is very small in the proposed algorithm. Moreover, the number of memory accesses and instructions are adaptive, and are decreased depending on the amount of activity in the scene. Each pixel is treated differently based on its history, and instead of requiring the same number of memory accesses and instructions for every pixel, we require less instructions for stable background pixels. The plot of the number of unstable pixels at each frame also serves as a tool to find the video portions with high activity. The proposed method selectively updates the background model with an automatically adaptive rate, thus can adapt to rapid changes. As opposed to traditional methods, pixels are not always treated individually, and information about neighbors is incorporated into decision making. The results obtained with nine challenging outdoor and indoor sequences are presented, and compared with the results of different state-of-the-art background subtraction methods. The ROC curves and memory comparison of different background subtraction methods are also provided. The experimental results demonstrate the success of the proposed light-weight salient foreground detection method.

A novel software framework for embedded multiprocessor smart cameras

Distributed smart cameras (DSC) are an emerging technology for a broad range of important applications including smart rooms, surveillance, entertainment, tracking, and motion analysis. By having access to many views and through cooperation among the individual cameras, these DSCs have the potential to realize many more complex and challenging applications than single-camera systems. This article focuses on the system-level software required for efficient streaming applications on single smart cameras as well as on networks of DSCs. Embedded platforms with limited resources do not provide middleware services well known on general-purpose platforms. Our software framework supports transparent intra- and interprocessor communication while keeping the memory and computation overhead very low. The software framework is based on a publisher--subscriber architecture and provides mechanisms for dynamically loading and unloading software components as well as for graceful degradation in case of software- and hardware-related faults. The software framework has been completely implemented and tested on our embedded smart cameras consisting of an ARM-based network processor and several digital signal processors. Two case studies demonstrate the feasibility of our approach.

The Signal Passing Interface and Its Application to Embedded Implementation of Smart Camera Applications

Embedded smart camera systems comprise computation- and resource-hungry applications implemented on small, complex but resource-hardy platforms. Efficient implementation of such applications can benefit significantly from parallelization. However, communication between different processing units is a nontrivial task. In addition, new and emerging distributed smart cameras require efficient methods of communication for optimized distributed implementations. In this paper, a novel communication interface, called the signal passing interface (SPI), is presented that attempts to overcome this challenge by integrating relevant properties of two different, yet important, paradigms in this context-dataflow and message passing interface (MPI). Dataflow is a widely used modeling paradigm for signal processing applications, while MPI is an established communication interface in the general-purpose processor community. SPI is targeted toward computation-intensive signal processing applications, and due to its careful specialization, more performance-efficient for embedded implementation in this domain. SPI is also much easier and more intuitive to use. In this paper, successful application of this communication interface to two smart camera applications has been presented in detail to validate a new methodology for efficient distributed implementation for this domain.

Autonomous Multicamera Tracking on Embedded Smart Cameras

There is currently a strong trend towards the deployment of advanced computer vision methods on embedded systems. This deployment is very challenging since embedded platforms often provide limited resources such as computing performance, memory, and power. In this paper we present a multicamera tracking method on distributed, embedded smart cameras. Smart cameras combine video sensing, processing, and communication on a single embedded device which is equipped with a multiprocessor computation and communication infrastructure. Our multicamera tracking approach focuses on a fully decentralized handover procedure between adjacent cameras. The basic idea is to initiate a single tracking instance in the multicamera system for each object of interest. The tracker follows the supervised object over the camera network, migrating to the camera which observes the object. Thus, no central coordination is required resulting in an autonomous and scalable tracking approach. We have fully implemented this novel multicamera tracking approach on our embedded smart cameras. Tracking is achieved by the well-known CamShift algorithm; the handover procedure is realized using a mobile agent system available on the smart camera network. Our approach has been successfully evaluated on tracking persons at our campus.