Human Object Detection Research Articles

A Deep Learning-Enhanced Multi-Modal Sensing Platform for Robust Human Object Detection and Tracking in Challenging Environments

In modern security situations, tracking multiple human objects in real-time within challenging urban environments is a critical capability for enhancing situational awareness, minimizing response time, and increasing overall operational effectiveness. Tracking multiple entities enables informed decision-making, risk mitigation, and the safeguarding of civil-military operations to ensure safety and mission success. This paper presents a multi-modal electro-optical/infrared (EO/IR) and radio frequency (RF) fused sensing (MEIRFS) platform for real-time human object detection, recognition, classification, and tracking in challenging environments. By utilizing different sensors in a complementary manner, the robustness of the sensing system is enhanced, enabling reliable detection and recognition results across various situations. Specifically designed radar tags and thermal tags can be used to discriminate between friendly and non-friendly objects. The system incorporates deep learning-based image fusion and human object recognition and tracking (HORT) algorithms to ensure accurate situation assessment. After integrating into an all-terrain robot, multiple ground tests were conducted to verify the consistency of the HORT in various environments. The MEIRFS sensor system has been designed to meet the Size, Weight, Power, and Cost (SWaP-C) requirements for installation on autonomous ground and aerial vehicles.

Semi-Supervised Federated Learning Over Heterogeneous Wireless IoT Edge Networks: Framework and Algorithms

Federated learning (FL) is a promising paradigm for future sixth-generation wireless systems to underpin network edge intelligence for smart cities applications. However, most of the data collected by the Internet of Things devices in such applications is unlabeled, necessitating the use of semi-supervised learning. Existing studies have introduced solutions to run semi-supervised FL; however, they overlooked the inherent critical impacts of the wireless characteristics at the network edge. We fill this gap by proposing novel solutions to run semi-supervised FL over wireless network edge, considering the limited computation and communication resources and deadline constraints and realizing that unlabeled data can be automatically labeled during the training rounds to improve the performance of the global model. The problem is first formulated as an optimization problem followed by a two-phase solution. In the first phase, we propose a bisection-based algorithm to find the transmit power and local processing speed that optimally fit the new injected labeled data. In the second phase, we propose three algorithms to control the local updates and injected samples that meet the deadline constraint. We analyze the performance of each algorithm concerning the tradeoffs between learning performance, training time, and total energy consumption. Targeting two applications in smart cities, human activity recognition and object detection, we conduct extensive simulations using realistic federated data sets under nonindependent and identically distributed settings. Numerical results show that the proposed algorithms effectively utilize unlabeled samples while accounting for the characteristics of wireless edge networks in smart cities.

SafeFac: Video-based smart safety monitoring for preventing industrial work accidents

This work presents SafeFac, an intelligent camera-based system for managing the safety of factory environments. In SafeFac a set of cameras installed on the assembly line are used to capture images of workers that approach the machinery under hazardous situations to alert system managers and halt the line if needed. Given a challenging set of practical application-level requirements such as multi-camera support and low response latency, SafeFac exploits a YOLOv3-based light-weight human object detection. To address the latency–accuracy tradeoff, SafeFac incorporates a set of algorithms as pre- and post-processing modules and a novel adaptive camera scheduling scheme.Our evaluation with a video dataset containing more that 113,000 frames from real assembly line activity shows that SafeFac achieves high precision (99.93%) and recall (96.44%), and SafeFac successfully satisfies such challenging requirements as a ready-for-deployment system for safe factory management.

An Enhanced Drone Technology for Detecting the Human Object in the Dense Areas Using a Deep Learning Model

During the recent decade, emerging technological and dramatic uses for drones were devised and accomplished, including rescue operations, monitoring, vehicle tracking, forest fire monitoring, and environmental monitoring, among others. Wildfires are one of the most significant environmental threats to wild areas and forest management. Traditional firefighting methods, which rely on ground operation inspections, have major limits and may threaten firefighters’ lives. As a result, remote sensing techniques, particularly UAV-based remotely sensed techniques, are currently among the most sought-after wildfire-fighting approaches. Current improvements in drone technology have resulted in significant breakthroughs that allow drones to perform a wide range of more sophisticated jobs. Rescue operations and forest monitoring, for example, demand a large security camera, making the drone a perfect tool for executing intricate responsibilities. Meanwhile, growing movement of the deep learning techniques in computer vision offers an interesting perspective into the project’s objective. They were used to identify forest fires in their beginning stages before they become out of control. This research describes a methodology for recognizing the presence of humans in a forest setting utilizing a deep learning framework and a human object detection method. The goal of identifying human presence in forestry areas is to prevent illicit forestry operations like illegal access into forbidden areas and illegal logging. In recent years, a lot of interest in automated wildfire identification utilizes UAV-based visual information and various deep learning techniques. This study focused on detecting wildfires at the beginning stages in forest and wilderness areas, utilizing deep learning-based computer vision algorithms that control and then mitigate massive damages to human life and forest management.

Textile-based pressure sensor arrays: A novel scalable manufacturing technique

Soft pressure sensors have sparked a lot of interest over the last decade because of their applications in human motion recognition, object detection, and human–computer interaction. However, their mass production and availability to end users are limited due to the complex and time-consuming steps. The scalability of working range for various applications is also a critical challenge. Therefore, a laborless, rapid, and scalable manufacturing technique for capacitive-based soft pressure sensors with high sensitivity and high working range is proposed in this work. The novel manufacturing method enables manipulation of sensor properties by varying production parameters based on specific application needs. The proposed sensor's electrode is made of conductive knit fabric, and the dielectric layers are made of thermoplastic polyurethane (TPU) sheets. As a result of the novel approach, it is possible to generate scalable air gaps between electrodes and dielectric layers to capture low pressures of less than 1 kPa. The usage of multi-layer TPU sheets also increases the working range of sensors up to 1000 kPa. Here, the proposed technology is successfully applied to create several sensor mats for different purposes such as improved gesture and shape recognition, and interactive gaming mats for children. • Simple and fast manufacturing of a novel textile pressure sensor. • Scalable with desired pattern of the sensor mat. • Adjustable sensitivity and wide working range from 1 kPa to 1000 kPa. • Pressure mapping for e-textiles, interactive software and human gesture recognition.

Anomaly Recognition Algorithm for Human Multipose Motion Behavior Using Generative Adversarial Network

Human multipose motion behavior is similar; there are many actions. However, it is difficult to recognize abnormal behavior. The existing human motion behavior anomaly recognition methods have the problems of low accuracy and being time-consuming. Therefore, an anomaly recognition method of human multipose motion behavior using Generative Adversarial Network (GAN) is proposed. The Gauss model is used to segment the human multipose motion behavior image, and the image foreground of the segmentation result is the human motion target detection result. The Shi-Tomasi algorithm is selected to extract contour feature points from human motion object detection results. The extracted contour features are set as hidden space random variables and input into the GAN. The GAN uses the generator and discriminator to recognize the multipose human motion behavior and determine whether the multipose human motion behavior is abnormal. The results show that the proposed algorithm can accurately recognize abnormal human multipose motion behavior, the recognition accuracy is higher than 99%, and the average recognition time is less than 200 ms. The shadow removal effect of the foreground image obtained by the proposed algorithm can realize the accurate recognition of human multipose motion behavior abnormalities and provide a reliable basis for research in related fields.

Toward Intelligent Surveillance as an Edge Network Service (iSENSE) Using Lightweight Detection and Tracking Algorithms

Edge computing extends the realm of information technology beyond the boundaries defined by cloud computing. Performing computation near the sensors, edge computing is promising to address the challenges in many bandwidth-and delay-sensitive applications. Although recently many smart video surveillance approaches based on Machine Learning (ML) algorithms become available, it is still challenging to efficiently migrate those smart algorithms to edge. In this paper, we propose an intelligent Surveillance as an Edge Network Service (iSENSE), which explores the feasibility of moving ML to the edge by testing two popular human-object detection schemes. Besides, a lightweight Convolutional Neural Network (L-CNN) is introduced to improve computational execution by leveraging the depth-wise separable convolution. To enhance performance on edge, we propose a hybrid lightweight tracking algorithm, Kerman (Kernelized Kalman filter), which is a decision tree based hybrid Kernelized Correlation Filter algorithm designed for human-object tracking. We have implemented both Kerman and L-CNN algorithms on edge by using different types of single board computers. The proposed iSENSE system was validated using both real-world campus surveillance video and open image sets. The experimental results present that the proposed algorithms can track the human objects in real-time with a good accuracy with limited resource in edge devices.

Human object detection: An enhanced black widow optimization algorithm with deep convolution neural network

Human object detection: An enhanced black widow optimization algorithm with deep convolution neural network

Dynamic Dual-Peak Network: A real-time human detection network in crowded scenes

Human detection in crowded scenes is challenging since the objects occlude and overlap each other. Compared to general pedestrian detection, there is also more variation in human posture. This paper proposes a real-time human detection network, Dynamic Dual-Peak Network (DDPNet), which specifically addresses human object detection in overlapping and crowded scenes. We design a deep cascade fusion module to enhance the feature extraction capability of the anchor-free model for small objects in crowded scenes. In the meantime, the head–body dual-peak activation module is used to improve the prediction score of the central region of the occluded individual through low occlusion components. By this improvement strategy, the network’s ability is enhanced to discriminate individuals in crowded scenes and alleviate the problem caused by individual posture variation. Ultimately, we propose a novel Exhale–Inhale method to adjust the feature mapping ranges for various scale objects dynamically. In the process of ground truth mapping, the overlapping of individual feature information is reduced. Our DDPNet achieves competitive performance on the CrowdHuman dataset and executes real-time inference of almost 3x∼7x faster than competitive methods.

Human Detection and Identification for Home Monitoring System

Currently, a monitoring system is needed to improve security and productivity in various sectors, for example in public services such as education centers, offices, banking, stations, roads, and the industrial sector. With a monitoring system, public service activities can be well monitored. Apart from public services, a home surveillance system can be implemented, making it easier for household members to monitor home security. The monitoring system camera will record the object, the human object detection process is carried out with the Histogram of Oriented Gradient, after the object is detected, the facial recognition process of the object is carried out whether it is a member of the house or not by using Learning Vector Quantization. If the object is not a member of the house, the system will send a notification in the form of a photo of the object’s face to the user’s email. For object detection with the Histogram of Oriented Gradient, an average accuracy of 90% is obtained and Learning Vector Quantization for facial recognition is 85%.

Comparative analysis of Human Activity Recognition and object detection

Human Activity Recognition (HAR) is the ability to interpret human body gesture or motion and determine human activity or action. Most of the tasks can be done automatically if they can be recognized via HAR system. HAR is an essential part of various scientific research contexts and the daily defence system i.e. surveillance, healthcare and human computer interaction (HCI). In computer vision, images and videos are analysed using different techniques. While recognition of activity in the real time environment the speed of the technique matters because if the system can detect the object in the images and videos at real time then it can be forwarded for the activity recognition process. How correctly the technique detects the object and differentiates the human object and other object and how much time it is taking to give the correct object, the number of frames it processes per second all matters. This paper deals with the comparison of the performance of YOLO (You Only Look Once) and Faster R-CNN in recognizing the human activity and detecting object.

Deep High-Resolution Representation Learning for Visual Recognition.

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions in series (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams in parallel and (ii) repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at https://github.com/HRNet.

Detection of the Face using Triplet Method in Messy or Crowdy Video

Today India is developing country. All the sectors are influenced by technology, Detection of the Face using Triplet method in messy or crowdy video or an input live video stream for Low enforcement or investigative agency, In Our proposed project we are using a new technique of Face detection with Human object detection the technique is called as deep metric learning.

Gun Source and Muzzle Head Detection

There is a surging need across the world for protection against gun violence. There are three main areas that we have identified as challenging in research that tries to curb gun violence: temporal location of gunshots, gun type prediction and gun source (shooter) detection. Our task is gun source detection and muzzle head detection, where the muzzle head is the round opening of the firing end of the gun. We would like to locate the muzzle head of the gun in the video visually, and identify who has fired the shot. In our formulation, we turn the problem of muzzle head detection into two sub-problems of human object detection and gun smoke detection. Our assumption is that the muzzle head typically lies between the gun smoke caused by the shot and the shooter. We have interesting results both in bounding the shooter as well as detecting the gun smoke. In our experiments, we are successful in detecting the muzzle head by detecting the gun smoke and the shooter.

A deep model method for recognizing activities of workers on offshore drilling platform by multistage convolutional pose machine

The growing diversity of image scenes brings a great challenge to human activity recognition in practice. Traditional activity recognition methods cannot satisfy the demand of precise action recognition in complex scenes. In this work, we build a training set of worker's activities on offshore drilling platform by collecting data from offshore drilling monitor, and then an improved multi-level convolutional pose machine (MCPM) method is proposed and trained to recognize activities of workers on the platforms. In human object detection, a multi-rule region proposal marker algorithm is developed to separate the seawater area, and the ducts of similar personnel is pre-discriminated by support vector machine. We use the characteristics of the human body key-points not affected by complex background noise to assist the detection of the human target. As results, it shown that our method performs better than Faster-RCNN, MobileNet-SSD and SSD algorithms in detecting human target on the offshore drilling platform, and achieves well accuracy in recognizing many key activities. To our best acknowledge, it is the first attempt of using deep model to recognize worker's activities on offshore drilling platform.

Deep Learning-Based Computational Color Constancy With Convoluted Mixture of Deep Experts (CMoDE) Fusion Technique

In the human and computer vision, color constancy is the ability to perceive the true color of objects in spite of changing illumination conditions. Color constancy is remarkably benefitting human and computer vision issues such as human tracking, object and human detection and scene understanding. Traditional color constancy approaches based on the gray world assumption fall short of performing a universal predictor, but recent color constancy methods have greatly progressed with the introduction of convolutional neural networks (CNNs). Yet, shallow CNN-based methods face learning capability limitations. Accordingly, this article proposes a novel color constancy method that uses a multi-stream deep neural network (MSDNN)-based convoluted mixture of deep experts (CMoDE) fusion technique in performing deep learning and estimating local illumination. In the proposed method, the CMoDE fusion technique is used to extract and learn spatial and spectral features in an image space. The proposed method distinctively piles up layers both in series and in parallel, selects and concatenates effective paths in the CMoDE-based DCNN, as opposed to previous works where residual networks stack multiple layers linearly and concatenate multiple paths. As a result, the proposed CMoDE-based DCNN brings significant progress towards efficiency of using computing resources, as well as accuracy of estimating illuminants. In the experiments, Shi's Reprocessed, gray-ball and NUS-8 Camera datasets are used to prove illumination and camera invariants. The experimental results establish that this new method surpasses its conventional counterparts.

Item Ownership Relationship Semantic Learning Strategy for Personalized Service Robot

In order to satisfy the robotic personalized service requirements that can select exclusive items to perform inference and planning according to different service individuals, the service robots need to have the ability to independently obtain the ownership relationship between humans and their carrying items. In this work, we present a novel semantic learning strategy for item ownership. Firstly, a human-carrying-items detection network based on human posture estimation and object detection model is used. Then, the transferred convolutional neural network is used to extract the characteristics of the objects and the back-end classifier to recognize the object instance. At the same time, the face detection and recognition model are used to identify the service individual. Finally, on the basis of the former two, the active learning of ownership items is completed. The experimental results show that the proposed ownership semantic learning strategy can determine the ownership relationship of private goods accurately and efficiently. The solution of this problem can improve the intelligence level of robot life service.

Does Semantic Activation Affect Human Object Detection in Natural Scenes?

Does Semantic Activation Affect Human Object Detection in Natural Scenes?

Development and Validation of a Camera-Based Illumination Controller for People Detection, Tracking and Recognition using Computational Intelligence

Illumination is crucial in human activities and in machine vision applications. For indoor surveillance applications, Infrared (IR) Light Emitting Diodes (LEDs) are the common means of providing illumination to the camera to cause no discomfort to human occupants. While IR provides non-obtrusive illumination for the camera, the same energy consumed does not provide the illumination to indoor spaces of the building. This is important if the premises where the camera is installed is not connected to the main power source or electric grid but derives energy from renewable sources. In this work, an illumination controller based on fuzzy logic system is developed and integrated to a vision system and an LED lighting system to provide a constant level of illumination to an object regardless of its distance from the image sensor. The computer vision system performs human object detection and face recognition and outputs fuzzy values representing the inferred distance of detected objects where the fuzzy system generates crisp output of duty cycle settings for the PWM controller for the LED lighting system to provide the required illumination needed by the vision system. Optimum illumination level for the vision system to perform the detection, tracking and face recognition operations must be provided by the system. Using visible Light Emitting Diodes as source of illumination, the system provides illumination both for the proper operation of the camera and human personnel monitoring the premises where the system is installed. This feature is significant in energy-constrained surveillance applications or where there is no power source derived from the electric grid.

Dynamic human object recognition by combining color and depth information with a clothing image histogram

Human object detection, tracking, and recognition have applications in many areas, such as in the development of assistance robots and intelligent monitoring systems. The emergence of an RGB-D camera, namely the Kinect v2, has simplified the process of human object detection and tracking. Color space methods are dependent on lighting conditions. Because skeleton-tracking algorithms are based on depth images, they are light invariant relative to color space methods. However, skeleton information may sometimes be incorrect or become lost. An algorithm for human-target recognition is thus required. Therefore, this study proposes a human-target tracking and recognition system combining RGB images, depth images, body index, and skeleton information. The system first extracts the color information of five body parts (two upper arms, the torso, and two thighs) using color, depth, and skeleton information. The system then analyzes the color information using a mixed nine-dimensional histogram and single-color analysis method. The algorithm also includes overlap detection during the process of human-target tracking to prevent misidentification caused by occlusion. To test the proposed system, various scenarios were carefully designed to simulate the extremely complex environmental changes characteristic of the real world. Furthermore, the dynamic statistical method of event statistics was used to collect results. Experiments revealed that the proposed method is robust under varying lighting conditions and increases the success rate for individuals wearing similar clothing with monochrome colors.