Smart Camera Research Articles

Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset.

We investigate privacy-preserving, video-based action recognition in deep learning, a problem with growing importance in smart camera applications. A novel adversarial training framework is formulated to learn an anonymization transform for input videos such that the trade-off between target utility task performance and the associated privacy budgets is explicitly optimized on the anonymized videos. Notably, the privacy budget, often defined and measured in task-driven contexts, cannot be reliably indicated using any single model performance because strong protection of privacy should sustain against any malicious model that tries to steal private information. To tackle this problem, we propose two new optimization strategies of model restarting and model ensemble to achieve stronger universal privacy protection against any attacker models. Extensive experiments have been carried out and analyzed. On the other hand, given few public datasets available with both utility and privacy labels, the data-driven (supervised) learning cannot exert its full power on this task. We first discuss an innovative heuristic of cross-dataset training and evaluation, enabling the use of multiple single-task datasets (one with target task labels and the other with privacy labels) in our problem. To further address this dataset challenge, we have constructed a new dataset, termed PA-HMDB51, with both target task labels (action) and selected privacy attributes (skin color, face, gender, nudity, and relationship) annotated on a per-frame basis. This first-of-its-kind video dataset and evaluation protocol can greatly facilitate visual privacy research and open up other opportunities. Our codes, models, and the PA-HMDB51 dataset are available at: https://github.com/VITA-Group/PA-HMDB51.

Fire Fighting Robot Using GSM Technology and Smart Camera

Fire accidents have been occurring frequently in these days, with or without the intervention of human. Whatever may be the reason it is one of the reason for huge blow to the mankind. Different losses would occur in the form of property, land, humans and animals too. Nature is also losing its environmental balance. Before the fire get widespread we have to inhibit it in initial stages. With the use of current technology we have developed a robot. In our paper, we will give the information about the design and construction of it. This fire fighting robot uses GSM technology. When the Robot detects fire, it gives message to the user by the use of GSM. A wireless smart camera which is fixed to it gives the video coverage of the surroundings. By the use of DTMF concept, we can control the movement of the robot in the direction we prefer. 8051 microcontroller is the main component of the project. It only sends and receives all major signals.

A Mini Review on Superhydrophobic and Transparent Surfaces.

In recent years, self-cleaning and transparent surfaces have been widely studied for application on smart windows, solar panels, camera lenses, and other optoelectronic devices. The self-cleaning properties can possibly extend the lifetime of these products and decrease, even eliminate, the requirement of chemical detergents and high labor costs of cleaning. It can also promote the overall efficiency of outdoor optoelectronic devices (e. g. solar cell panels) since dirt accumulation and bacteria growth can be slowed down, even inhibited on such surfaces. In this mini review, the fundamentals and conditions that govern superhydrophobicity and transparency are introduced, followed by the discussion of roughness as the competing factor for superhydrophobicity and transparency. Representative examples of the surface design and fabrication are introduced and future perspectives are shared. This mini review can help the research community better understand such surfaces and further accelerate its development for innovative practical applications.

Multi-Shot Technology for Color Reproduction in Digital Still Cameras

Artificial intelligence has also helped us do things faster and better, in many areas. This also applies to photography. As many of the functions of professional digital cameras, such as the functions of automatic adjustment of exposure and focus, rely on artificial intelligence. And smart phones have become equipped with cameras that work with artificial intelligence technology, and therefore the user of the camera will have more time devoted to the creative aspects, instead of wasting time on solutions to repeated problems. Artificial intelligence has also affected the development of sophisticated algorithms that gradually replace traditional methods of processing digital photographs. This allows immediate processing of images, fixing their faults, and improving them automatically that does not require any effort. To reach amazing results, obtaining them requires the photographer to spend long hours working on traditional software. And it came to getting a photo without a photographer, as is the case with the Google Clips smart camera, as well as obtaining a photo without a photographer or camera, as is the case with images generated by artificial intelligence algorithms like StyleGan. The research problem lies in the fact that lack of familiarity with the multiple capabilities of artificial intelligence, and trying to employ them effectively in all stages of the production of the photo, will lead to consuming a lot of time and effort in solving traditional problems, which artificial intelligence can accomplish quickly, easily and efficiently, such as automatic control of exposure and focus, And improve the image and fix its faults and add effects, in less time and effort. Therefore, the research aims to identify the huge potential granted by applications of artificial intelligence in photography, and the extent of its rapid development, and how to maximize the use of them to obtain better photographs in less time and effort. Either in the shooting stage, or in the stage of processing and improving images and fixing their faults, as well as in generating and creating photographs of subjects that do not exist in reality.

Vision System Prototype for Inspection and Monitoring with a Smart Camera

This paper presents the design of an artificial vision system prototype for automatic inspection and monitoring of objects over a conveyor belt and using a Smart camera 2D BOA-INS. The prototype consists of a conveyor belt and an embedded system based on an Arduino Mega card for system control, and it has as main peripherals the smart camera, a direct current motor, a photoelectric sensor, LED illumination and LEDs indicating the status (good or defect) of each evaluated object. The application of the prototype is for educational purposes, so that undergraduate, master and diploma students can simulate a continuous production line, controlled by an embedded system, and perform quality control by monitoring through a visual system and a personal computer. This allows implementing the topics of embedded systems, artificial vision, artificial intelligence, pattern recognition, automatic control, as well as automation of real processes.

Dynamic Camera Reconfiguration with Reinforcement Learning and Stochastic Methods for Crowd Surveillance.

Crowd surveillance plays a key role to ensure safety and security in public areas. Surveillance systems traditionally rely on fixed camera networks, which suffer from limitations, as coverage of the monitored area, video resolution and analytic performance. On the other hand, a smart camera network provides the ability to reconfigure the sensing infrastructure by incorporating active devices such as pan-tilt-zoom (PTZ) cameras and UAV-based cameras, thus enabling the network to adapt over time to changes in the scene. We propose a new decentralised approach for network reconfiguration, where each camera dynamically adapts its parameters and position to optimise scene coverage. Two policies for decentralised camera reconfiguration are presented: a greedy approach and a reinforcement learning approach. In both cases, cameras are able to locally control the state of their neighbourhood and dynamically adjust their position and PTZ parameters. When crowds are present, the network balances between global coverage of the entire scene and high resolution for the crowded areas. We evaluate our approach in a simulated environment monitored with fixed, PTZ and UAV-based cameras.

A REAL-TIME PHOTOGRAMMETRIC SYSTEM FOR MONITORING HUMAN MOVEMENT DYNAMICS

Abstract. The human body posture is rich with dynamic information that can be captured by algorithms, and many applications rely on this type of data (e.g., action recognition, people re-identification, human-computer interaction, industrial robotics). The recent development of smart cameras and affordable red-green-blue-depth (RGB-D) sensors has enabled cost-efficient estimation and tracking of human body posture. However, the reliability of single sensors is often insufficient due to occlusion problems, field-of-view limitations, and the limited measurement distances of the RGB-depth sensors. Furthermore, a large-scale real-time response is often required in certain applications, such as physical rehabilitation, where human actions must be detected and monitored over time, or in industries where human motion is monitored to maintain predictable movement flow in a shared workspace. Large-scale markerless motion-capture systems have therefore received extensive research attention in recent years.In this paper, we propose a real-time photogrammetric system that incorporates multithreading and a graphic process unit (GPU)-accelerated solution for extracting 3D human body dynamics in real-time. The system includes a stereo camera with preliminary calibration, from which left-view and right-view frames are loaded. Then, a dense image-matching algorithm is married with GPU acceleration to generate a real-time disparity map, which is further extended to a 3D map array obtained by photogrammetric processing based on the camera orientation parameters. The 3D body features are acquired from 2D body skeletons extracted from regional multi-person pose estimation (RMPE) and the corresponding 3D coordinates of each joint in the 3D map array. These 3D body features are then extracted and visualised in real-time by multithreading, from which human movement dynamics (e.g., moving speed, knee pressure angle) are derived. The results reveal that the process rate (pose frame-rate) can be 20 fps (frames per second) or above in our experiments (using two NVIDIA 2080Ti and two 12-core CPUs) depending on the GPU exploited by the detector, and the monitoring distance can reach 15 m with a geometric accuracy better than 1% of the distance.This real-time photogrammetric system is an effective real-time solution to monitor 3D human body dynamics. It uses low-cost RGB stereo cameras controlled by consumer GPU-enabled computers, and no other specialised hardware is required. This system has great potential for applications such as motion tracking, 3D body information extraction and human dynamics monitoring.

Always-On 674μ W@4GOP/s Error Resilient Binary Neural Networks With Aggressive SRAM Voltage Scaling on a 22-nm IoT End-Node

Binary Neural Networks (BNNs) have been shown to be robust to random bit-level noise, making aggressive voltage scaling attractive as a power-saving technique for both logic and SRAMs. In this work, we introduce the first fully programmable IoT end-node system-on-chip (SoC) capable of executing software-defined, hardware-accelerated BNNs at ultra-low voltage. Our SoC exploits a hybrid memory scheme where error-vulnerable SRAMs are complemented by reliable standard-cell memories to safely store critical data under aggressive voltage scaling. On a prototype in 22nm FDX technology, we demonstrate that both the logic and SRAM voltage can be dropped to 0.5V without any accuracy penalty on a BNN trained for the CIFAR-10 dataset, improving energy efficiency by 2.2X w.r.t. nominal conditions. Furthermore, we show that the supply voltage can be dropped to 0.42V (50% of nominal) while keeping more than 99% of the nominal accuracy (with a bit error rate ~1/1000). In this operating point, our prototype performs 4Gop/s (15.4 Inference/s on the CIFAR-10 dataset) by computing up to 13 binary ops per pJ, achieving 22.8 Inference/s/mW while keeping within a peak power envelope of 674uW - low enough to enable always-on operation in ultra-low power smart cameras, long-lifetime environmental sensors, and insect-sized pico-drones.

Special issue on smart cameras for real-time image and video processing

Special issue on smart cameras for real-time image and video processing

De-ghosted HDR video acquisition for embedded systems

This paper proposes a novel ghost-free High Dynamic Range (HDR) multi-exposure video acquisition suitable for real-time implementation in embedded systems. While the method is limited to stationary cameras, it achieves, with low requirements on resources, results comparable to state-of-the-art de-ghosting methods that are often very computationally expensive and almost impossible to implement in smart cameras and embedded systems. The paper describes the method itself and includes an evaluation of the performance on selected embedded platforms and a comparison of the results to the state of the art using HDR datasets.

Co-designing a Self-Management App Prototype to Support People With Spinal Cord Injury in the Prevention of Pressure Injuries: Mixed Methods Study.

BackgroundSpinal cord injury is a complex chronic health condition that requires individuals to actively self-manage. Therefore, an evidence-based, self-management app would be of value to support individuals with spinal cord injury in the prevention of pressure injuries.ObjectiveThe main objectives of this study were to (1) establish a co-design approach for developing a high-fidelity prototype app for the self-management of individuals with spinal cord injury, (2) design the prototype that resulted from this process, and (3) conduct the first usability assessment of the prototype app.MethodsWe adopted a co-design approach to develop an evidence-based app prototype. Starting from a preliminary content model (based on clinical guidelines for the prevention of pressure injuries) and three research-based user personas, we conducted an ideation workshop involving individuals with spinal cord injury and health care professionals. The ideation workshop formed the basis for two consecutive design sprints. The result of this co-design phase was an interactive app prototype. The prototype was evaluated in two rounds of usability testing (N=4 and N=15, respectively) using a combination of qualitative and quantitative methods.ResultsThe co-design process resulted in a high-fidelity prototype with two key components: a self-management component and a communication component. The final prototype included a combination of features to support individuals with spinal cord injury in the prevention of pressure injuries, namely a smart camera, pressure injury diary, expert consultation, reminders, and knowledge repository. Findings of the usability testing showed that most participants navigated the app fluently with little back and forth navigation and were able to successfully complete a set of assigned tasks. These positive results are supported by the average system usability score achieved (78.5/100; range 47.5-95.0) and our qualitative analysis of the semistructured interviews. Despite an overall positive evaluation of the app prototype, we identified areas for improvement (eg, inclusion of a search function).ConclusionsIndividuals with spinal cord injury often need to navigate competing interests and priorities, paired with uncertainty about the accuracy and relevance of clinical recommendations. Understanding what matters to individuals with spinal cord injury can help guide the design of behavioral interventions that are useful and acceptable to these individuals in their daily lives. This study shows that involving individuals with spinal cord injury and health care professionals in co-designing a self-management app can foster knowledge cocreation at the intersection of lived experience, medical expertise, and technical solutions.

A Vision-Based Pipeline for Vehicle Counting, Speed Estimation, and Classification

Cameras have been widely used in traffic operations. While many technologically smart camera solutions in the market can be integrated into Intelligent Transport Systems (ITS) for automated detection, monitoring and data generation, many Network Operations (a.k.a Traffic Control) Centres still use legacy camera systems as manual surveillance devices. In this paper, we demonstrate effective use of these older assets by applying computer vision techniques to extract traffic data from videos captured by legacy cameras. In our proposed vision-based pipeline, we adopt recent state-of-the-art object detectors and transfer-learning to detect vehicles, pedestrians, and cyclists from monocular videos. By weakly calibrating the camera, we demonstrate a novel application of the image-to-world homography which gives our monocular vision system the efficacy of counting vehicles by lane and estimating vehicle length and speed in real-world units. Our pipeline also includes a module which combines a convolutional neural network (CNN) classifier with projective geometry information to classify vehicles. We have tested it on videos captured at several sites with different traffic flow conditions and compared the results with the data collected by piezoelectric sensors. Our experimental results show that the proposed pipeline can process 60 frames per second for pre-recorded videos and yield high-quality metadata for further traffic analysis.

REAL TIME EYE TRACKING FOR PASSWORD AUTHENTICATION

The Generality of today’s authentication systems, including password , fingerprint scanners and Personal Identification Number(PIN) entry, are based on one time, static authentication methods which are prone to attacks. Real time eye tracking contributes significantly for Patients who are incapacitated, internet study , gaze based password entry as well as house hold security applications. The use of real time eye tracking and password authentication system has a wider possibility for reliability, but in this case the system must be secure and restrained for Eye Detection and Tracking. For the user’s security and validation (authentication) personal identification numbers are used abroad thereby reducing the chances of being attacked. In static authentication methods , since the password has to be entered physically using PIN’s there are possibilities for tracking the password using shoulder surfing, in order to avoid this spy game of tracking the password , hands off gaze- based PIN entry techniques are used. The smart camera is used to detect the eye position and for Gaze-based PIN Identification thereby doing the data processing. This project is focused on a real time hands off and gaze based PIN entry , detecting the eye position and tracking the eye movement using the smart camera.

On building a CNN-based multi-view smart camera for real-time object detection

Multi-view image sensing is currently gaining momentum, fostered by new applications such as autonomous vehicles and self-propelled robots. In this paper, we prototype and evaluate a multi-view smart vision system for object recognition. The system exploits an optimized Multi-View Convolutional Neural Network (MVCNN) in which the processing is distributed among several sensors (heads) and a camera body. The main challenge for designing such a system comes from the computationally expensive workload of real-time MVCNNs which is difficult to support with embedded processing and high frame rates. This paper focuses on the decisions to be taken for distributing an MVCNN on the camera heads, each camera head embedding a Field-Programmable Gate Array (FPGA) for processing images on the stream. In particular, we show that the first layer of the AlexNet network can be processed at the nearest of the sensors, by performing a Direct Hardware Mapping (DHM) using a dataflow model of computation. The feature maps produced by the first layers are merged and processed by a camera central processing node that executes the remaining layers. The proposed system exploits state-of-the-art deep learning optimization methods, such as parameter removing and data quantization. We demonstrate that accuracy drops caused by these optimizations can be compensated by the multi-view nature of the captured information. Experimental results conducted with the AlexNet CNN show that the proposed partitioning and resulting optimizations can fit the first layer of the multi-view network in low-end FPGAs. Among all the tested configurations, we propose 2 setups with an equivalent accuracy compared to the original network on the ModelNet40 dataset. The first one is composed of 4 cameras based on a Cyclone III E120 FPGA to embed the least expensive version in terms of logic resources while the second version requires 2 cameras based on a Cyclone 10 GX220 FPGA. This distributed computing with workload reduction is demonstrated to be a practical solution when building a real-time multi-view smart camera processing several frames per second.

A Stand-Alone Smart Camera System for Online Post-Earthquake Building Safety Assessment.

Computer vision-based approaches are very useful for dynamic displacement measurement, damage detection, and structural health monitoring. However, for the application using a large number of existing cameras in buildings, the computational cost of videos from dozens of cameras using a centralized computer becomes a huge burden. Moreover, when a manual process is required for processing the videos, prompt safety assessment of tens of thousands of buildings after a catastrophic earthquake striking a megacity becomes very challenging. Therefore, a decentralized and fully automatic computer vision-based approach for prompt building safety assessment and decision-making is desired for practical applications. In this study, a prototype of a novel stand-alone smart camera system for measuring interstory drifts was developed. The proposed system is composed of a single camera, a single-board computer, and two accelerometers with a microcontroller unit. The system is capable of compensating for rotational effects of the camera during earthquake excitations. Furthermore, by fusing the camera-based interstory drifts with the accelerometer-based ones, the interstory drifts can be measured accurately even when residual interstory drifts exist. Algorithms used to compensate for the camera’s rotational effects, algorithms used to track the movement of three targets within three regions of interest, artificial neural networks used to convert the interstory drifts to engineering units, and some necessary signal processing algorithms, including interpolation, cross-correlation, and filtering algorithms, were embedded in the smart camera system. As a result, online processing of the video data and acceleration data using decentralized computational resources is achieved in each individual smart camera system to obtain interstory drifts. Using the maximum interstory drifts measured during an earthquake, the safety of a building can be assessed right after the earthquake excitation. We validated the feasibility of the prototype of the proposed smart camera system through the use of large-scale shaking table tests of a steel building. The results show that the proposed smart camera system had very promising results in terms of assessing the safety of steel building specimens after earthquake excitations.

RETRACTED ARTICLE: Essentiality for bridging the gap between low and semantic level features in image retrieval systems: an overview

The modern world uses digital data invariably and the advent of smart phones and cameras add to images and communicating through images in social media is a very common happening due to the exposure to internet and the economical availability of data. Moreover, people explore the web for much news and on different topics. They feel satisfied to look into image type info than textual information. So, image retrieval is paramount nowadays and it is very much useful in many societal and defence applications. The methods used for retrieval started through text based and query based is popularly used now. The content based retrieval uses many stages, approaches, technologies, algorithms etc. It is intended to provide a broad perspective about CBIR in this work. Moreover, the computational and time complexities involved during retrieval relies on the performance of the entire system. One of the aspects that can improve the precision is the reduction of semantic gap. Taking this as the base, we have explored a literature and presented the points related to various points including relevance feedback so as to enable other researchers to carry out experimental work on the suggested areas.

CMRDF: A Real-Time Food Alerting System Based on Multimodal Data

A healthy diet is a major concern for everyone, especially for those with specific diseases, such as diabetes. Meanwhile, with the rapid development of new technologies, it is feasible for us to detect the deep latent relationship between daily meals and wellbeing. Advanced Internet of Things devices, such as smart bracelets and wearable cameras, make it possible for people to know how food is related to health at any time. However, it is still arduous for individuals to memorize all the health information and utilize them to regulate their diet. To deal with such problems, we propose a novel system called cross-modal retrieval on diabetogenic food (CMRDF) which realizes a real-time dietary notice based on multimodal data captured from wearable devices. In this system, we propose a new graph-based cross-modal retrieval method named graph correlation analysis with ranking loss that finds the latent information in multimodal data. We use graph convolutional networks to dig the deep latent information in modalities and represent the data in finer granularity. It uses visual and physiological information to estimate whether the food that a user tries to obtain is diabetogenic or not, and feeds back the reasons in detail. Extensive experiments on the MSCOCO data set and the new proposed multimodal diabetogenic food database real-life diabetogenic show that the proposed cross-modal retrieval method outperforms state-of-the-art methods and CMRDF can achieve reliable results on preventing diabetic patients from inappropriate food.

Visual model‐predictive localization for computationally efficient autonomous racing of a 72‐g drone

AbstractDrone racing is becoming a popular e‐sport all over the world, and beating the best human drone race pilots has quickly become a new major challenge for artificial intelligence and robotics. In this paper, we propose a novel sensor fusion method called visual model‐predictive localization (VML). Within a small time window, VML approximates the error between the model prediction position and the visual measurements as a linear function. Once the parameters of the function are estimated by the RANSAC algorithm, this error model can be used to compensate the prediction in the future. In this way, outliers can be handled efficiently and the vision delay can also be compensated efficiently. Theoretical analysis and simulation results show the clear advantage compared with Kalman filtering when dealing with the occasional large outliers and vision delays that occur in fast drone racing. Flight tests are performed on a tiny racing quadrotor named “Trashcan,” which was equipped with a Jevois smart camera for a total of 72 g. An average speed of 2 m/s is achieved while the maximum speed is 2.6 m/s. To the best of our knowledge, this flying platform is currently the smallest autonomous racing drone in the world, while still being one of the fastest autonomous racing drones.

IoT-Enabled Streaming Image Analytics With Privacy-Aware Self-Adaptive and Reflective Designs

With the development of the Internet of Things (IoT), a large number of cameras are able to generate large amounts of real-time data. This has attracted increasing attention to streaming-image analytics incorporating edge computing. Since streaming images do not have ground-truth labels, their outputs may become untrustworthy once any concept drift or uncertainty occurs. To keep analytics models updated in response to concept drift, much human intervention is required, particularly in system maintenance for IoT-enabled applications with a plethora of smart cameras. To address this issue and follow the trend of anthropomorphic machine learning, this article incorporates self-regulated learning from human pedagogy, which comprises forethought, performance, and reflection. Our article proposes counterparts for streaming-image analytics, including siaForethought, siaPerformance, and siaReflection, respectively. The experiment results show that siaForethought can effectively blur the detected pedestrians or the whole image to improve privacy awareness. Next, siaPerformance can help reduce the model adaptation time by 38%, even when using blurred pedestrians or images. Finally, siaReflection reduces about 50% of human intervention for labeling data, thus greatly reducing human interventions in case of concept drift.

A Systematic Literature Review of Intelligent Data Analysis Methods for Smart Sport Training

The rapid transformation of our communities and our way of life due to modern technologies has impacted sports as well. Artificial intelligence, computational intelligence, data mining, the Internet of Things (IoT), and machine learning have had a profound effect on the way we do things. These technologies have brought changes to the way we watch, play, compete, and also train sports. What was once simply training is now a combination of smart IoT sensors, cameras, algorithms, and systems just to achieve a new peak: The optimum one. This paper provides a systematic literature review of smart sport training, presenting 109 identified studies. Intelligent data analysis methods are presented, which are currently used in the field of Smart Sport Training (SST). Sport domains in which SST is already used are presented, and phases of training are identified, together with the maturity of SST methods. Finally, future directions of research are proposed in the emerging field of SST.