Inflated 3D ConvNet Research Articles

A novel approach for large-scale characterization of residential cooking-generated PM with computer vision and low-cost sensors

Cooking is one of the major sources of indoor particulate matter (PM), which poses significant health risks and is a severe health hazard. Current studies lack an economical and effective analytical framework for quantifying inhalable particles (PM10) and fine particulate matter (PM2.5) from residential cooking activities on a large scale under real-world scenarios. This study bridges this gap by employing computer vision (CV) technology and readily available sensors. We collected data over a month in real-world settings, including cooking videos and air quality data (indoor PM10, PM2.5, CO2, temperature, relative humidity, and outdoor PM10 and PM2.5 concentrations). To classify high-emission (pan-frying, stir-frying, deep-frying) and low-emission (stewing, steaming, boiling, non-cooking) activities, we developed and validated a robust CV model named “Cooking-I3D.” This model leverages a pre-trained Two-Stream Inflated 3D ConvNet (I3D) architecture. We then assessed the efficacy of the CV-predicted cooking method in PM characterization using a first-order multivariate autoregressive model, controlling for environmental factors. The Cooking-I3D model achieved exceptional performance, boasting an accuracy of 95 % and an Area Under the Curve (AUC) of 0.98. Our results indicate that a single 6-minute high-emission cooking event triggers a 21–25 % increase in indoor PM concentrations and a 23–24 % increase in the indoor/outdoor ratio, with relative errors in these estimates ranging from 10 to 21 %. This innovative method offers a powerful tool for long-term assessment of cooking-related indoor air pollution and facilitates precision exposure assessment in human health studies.

Thermal preference prediction through infrared thermography technology: Recognizing adaptive behaviors

The recognition of thermal adaptation behaviors is a crucial and noninvasive method for accurately and swiftly predicting the thermal preferences of occupants. This is instrumental in enhancing the energy efficiency, indoor personnel comfort, and overall productivity of office buildings. This study employed deep-learning models to analyze a thermal adaptive behavior video dataset captured by Infrared thermography (IRT) cameras to predict the thermal preferences of occupants. A large-scale infrared thermography dataset was developed for recognizing thermal adaptive behaviors using 9650 video samples. Two adaptive behavior recognition models for infrared videos were developed by leveraging neural network models: Two-Stream Inflated 3D ConvNet (I3D) and SlowFast (SF) networks. These models achieved the average prediction accuracies of 83.53 % and 88.56 %, respectively. Notably, both models exhibited a recognition time of milliseconds, facilitating the real-time recognition of thermal adaptive behaviors. This study offers vital technical and theoretical underpinnings for developing decision-making solutions for smart buildings based on noninvasive thermal preference prediction.

Automatic Evaluation Method for Functional Movement Screening Based on Multi-Scale Lightweight 3D Convolution and an Encoder–Decoder

Functional Movement Screening (FMS) is a test used to evaluate fundamental movement patterns in the human body and identify functional limitations. However, the challenge of carrying out an automated assessment of FMS is that complex human movements are difficult to model accurately and efficiently. To address this challenge, this paper proposes an automatic evaluation method for FMS based on a multi-scale lightweight 3D convolution encoder–decoder (ML3D-ED) architecture. This method adopts a self-built multi-scale lightweight 3D convolution architecture to extract features from videos. The extracted features are then processed using an encoder–decoder architecture and probabilistic integration technique to effectively predict the final score distribution. This architecture, compared with the traditional Two-Stream Inflated 3D ConvNet (I3D) network, offers a better performance and accuracy in capturing advanced human movement features in temporal and spatial dimensions. Specifically, the ML3D-ED backbone network reduces the number of parameters by 59.5% and the computational cost by 77.7% when compared to I3D. Experiments have shown that ML3D-ED achieves an accuracy of 93.33% on public datasets, demonstrating an improvement of approximately 9% over the best existing method. This outcome demonstrates the effectiveness of and advancements made by the ML3D-ED architecture and probabilistic integration technique in extracting advanced human movement features and evaluating functional movements.

The Establishment of a precise intelligent evaluation system for sports events: Diving

The introduction of action quality assessment technology in sports events to achieve precise intelligent evaluation can greatly enhance the objectivity and effectiveness of competition results. Taking diving as the specific application background, this study proposes a novel Multi-granularity Extraction Approach for Temporal-spatial features in judge scoring prediction (MEAT) under the conditions of action quality assessment. On the one hand, it uses dual-modal inflated 3D ConvNet to extract the temporal and spatial features of each modal diving video at the video granularity parallelly and to merge them to form a global feature. On the other hand, the human body pose is modeled, and the simulated athlete's three-dimensional splash state is taken as local characteristics at the object granularity. Finally, the global and local features are concatenated into the fully connected layer, and heuristic method inspired by competition rules using labeled distribution learning are employed to output the probability distribution of the average score of all referees. The maximum probability score is selected and multiplied by the difficulty coefficient to obtain the final diving score. Through comprehensive experiments, comparing the Spearman's rank correlation (SRC) evaluation results of existing methods on the UNIV-Dive dataset, this framework reflects the greater accuracy advantage and further lays the foundation for the actual implementation of the technology.

Deep Learning Assists Surveillance Experts: Toward Video Data Prioritization

Video summarization (VS) suppresses high-dimensional (HD) video data by only extracting only the important information. However, prior research has not focused on the need for surveillance VS, that is used for many applications to assist video surveillance experts, including video retrieval and data storage. In addition, mainstream techniques commonly use 2D deep models for VS, ignoring event occurrences. Accordingly, we present a two-fold 3D deep learning-assisted VS framework. First, we employ an inflated 3D ConvNet model to extract temporal features; these features are optimized using a proposed encoder mechanism. The input video is temporally segmented using a feature comparison technique for selecting a single frame from each video segment. The segmented shots are evaluated using our novel shot segmentation evaluation scheme and are input into a saliency computation mechanism for keyframe selection in a second fold. Qualitative and quantitative analyses over VS benchmarks and surveillance videos demonstrate the superior performance of our framework, with 0.3- and 4.2-unit increases in the F1 scores for YouTube and TVSum datasets, respectively. Along with accurate VS, a key contribution of our study is the novel shot segmentation criterion prior to VS, which can be used as a benchmark in future research to effectively prioritize HD visual data.

Suspicious Behavior Detection with Temporal Feature Extraction and Time-Series Classification for Shoplifting Crime Prevention.

The rise in crime rates in many parts of the world, coupled with advancements in computer vision, has increased the need for automated crime detection services. To address this issue, we propose a new approach for detecting suspicious behavior as a means of preventing shoplifting. Existing methods are based on the use of convolutional neural networks that rely on extracting spatial features from pixel values. In contrast, our proposed method employs object detection based on YOLOv5 with Deep Sort to track people through a video, using the resulting bounding box coordinates as temporal features. The extracted temporal features are then modeled as a time-series classification problem. The proposed method was tested on the popular UCF Crime dataset, and benchmarked against the current state-of-the-art robust temporal feature magnitude (RTFM) method, which relies on the Inflated 3D ConvNet (I3D) preprocessing method. Our results demonstrate an impressive 8.45-fold increase in detection inference speed compared to the state-of-the-art RTFM, along with an F1 score of 92%,outperforming RTFM by 3%. Furthermore, our method achieved these results without requiring expensive data augmentation or image feature extraction.

Vision-based activity recognition in children with autism-related behaviors

Advances in machine learning and contactless sensors have enabled the understanding complex human behaviors in a healthcare setting. In particular, several deep learning systems have been introduced to enable comprehensive analysis of neuro-developmental conditions such as Autism Spectrum Disorder (ASD). This condition affects children from their early developmental stages onwards, and diagnosis relies entirely on observing the child's behavior and detecting behavioral cues. However, the diagnosis process is time-consuming as it requires long-term behavior observation, and the scarce availability of specialists. We demonstrate the effect of a region-based computer vision system to help clinicians and parents analyze a child's behavior. For this purpose, we adopt and enhance a dataset for analyzing autism-related actions using videos of children captured in uncontrolled environments (e.g. videos collected with consumer-grade cameras, in varied environments). The data is pre-processed by detecting the target child in the video to reduce the impact of background noise. Motivated by the effectiveness of temporal convolutional models, we propose both light-weight and conventional models capable of extracting action features from video frames and classifying autism-related behaviors by analyzing the relationships between frames in a video. By extensively evaluating feature extraction and learning strategies, we demonstrate that the highest performance is attained through the use of an Inflated 3D Convnet and Multi-Stage Temporal Convolutional Network. Our model achieved a Weighted F1-score of 0.83 for the classification of the three autism-related actions. We also propose a light-weight solution by employing the ESNet backbone with the same action recognition model, achieving a competitive 0.71 Weighted F1-score, and enabling potential deployment on embedded systems. Experimental results demonstrate the ability of our proposed models to recognize autism-related actions from videos captured in an uncontrolled environment, and thus can assist clinicians in analyzing ASD.

Using 3D deep features from CT scans for cancer prognosis based on a video classification model: A multi-dataset feasibility study.

Cancer prognosis before and after treatment is key for patient management and decision making. Handcrafted imaging biomarkers-radiomics-have shown potential in predicting prognosis. However, given the recent progress in deep learning, it is timely and relevant to pose the question: could deep learning based 3D imaging features be used as imaging biomarkers and outperform radiomics? Effectiveness, reproducibility in test/retest, across modalities, and correlation of deep features with clinical features such as tumor volume and TNM staging were tested in this study. Radiomics was introduced as the reference image biomarker. For deep feature extraction, we transformed the CT scans into videos, and we adopted the pre-trained Inflated 3D ConvNet (I3D) video classification network as the architecture. We used four datasets-LUNG 1 (n=422), LUNG 4 (n=106), OPC (n=605), and H&N 1 (n=89)-with 1270 samples from different centers and cancer types-lung and head and neck cancer-to test deep features' predictiveness and two additional datasets to assess the reproducibility of deep features. Support Vector Machine-Recursive Feature Elimination (SVM-RFE) selected top 100 deep features achieved a concordance index (CI) of 0.67 in survival prediction in LUNG 1, 0.87 in LUNG 4, 0.76 in OPC, and 0.87 in H&N 1, while SVM-RFE selected top 100 radiomics achieved CIs of 0.64, 0.77, 0.73, and 0.74, respectively, all statistically significant differences (p<0.01, Wilcoxon's test). Most selected deep features are not correlated with tumor volume and TNM staging. However, full radiomics features show higher reproducibility than full deep features in a test/retest setting (0.89 vs. 0.62, concordance correlation coefficient). The results show that deep features can outperform radiomics while providing different views for tumor prognosis compared to tumor volume and TNM staging. However, deep features suffer from lower reproducibility than radiomic features and lack the interpretability of the latter.

SRI3D: Two‐stream inflated 3D ConvNet based on sparse regularization for action recognition

AbstractAlthough most state‐of‐the‐art action recognition models have adopted a two‐stream 3D convolutional structure as a backbone network, few works have studied the impact of loss functions on action recognition models. In addition, sparsity is used as a key prior knowledge in many fields. However, as far as is known, no one has studied the influence of the sparsity of network output on the output of deep learning‐based action recognition models. Therefore, this paper proposes a novel two‐stream inflated 3D ConvNet based on the sparse regularization (SRI3D) model for action recognition. In order to allow the network to learn the sparsity of output, the ℓ1 norm is embedded in the loss function in regularization form in a plug‐and‐play manner. It can make the classification result after the fusion of the two‐stream network only be the category with the highest confidence in one of the streams and not the other cases. The proposed loss function based on sparse regularization makes the output vector of the neural network as sparse as possible so that the classification results will not be ambiguous. Experimental results show that compared with other state‐of‐the‐art models, this SRI3D has a competitive advantage on Kinetics‐400, Something‐Something V2, UCF‐101 and HMDB‐51.

Non-invasive measurements of thermal discomfort for thermal preference prediction based on occupants' adaptive behavior recognition

With the spread of the Internet of Things in the construction industry, non-invasive thermal adaptive behavior recognition provides a new method for the real-time assessment of the indoor thermal environment. Pioneering research on non-invasive measurements of thermal discomfort for thermal preference prediction based on occupants' adaptive behavior recognition was undertaken. Questionnaire surveys on the thermal adaptive behavior of office building users in five climatic regions of China were undertaken. From 654 valid questionnaires, 14 thermal adaptive behaviors related to thermal discomfort were clustered. Then experiments were conducted to collect thermal discomfort adaptive behavioral video data from 8 views (front, rear, left, right, left-front, right-front, left-rear, and right-rear) at two heights (1.5 m and 2 m). A large-scale dataset for thermal adaptive behavior recognition with 27,504 video samples was developed. Two adaptive behavior recognition models were constructed based on the neural network model Two-Stream Inflated 3D ConvNet (I3D) and SlowFast. The results show that there is no significant difference in the types of indoor thermal adaptive behaviors among office users in different climate zones of China. The established I3D and SlowFast recognition models adopt an end-to-end approach for recognition, which overcomes dependence on human key point recognition and enables the models to utilize more environmental information. The average prediction accuracy of both models reaches 95% or above, even when complex background and human key points are partially obscured. Additionally, the recognition time of the two models is at the millisecond level, which can support real-time thermal adaptive behavior recognition.

Inflated 3D ConvNet context analysis for violence detection

According to the Wall Street Journal, one billion surveillance cameras will be deployed around the world by 2021. This amount of information can be hardly managed by humans. Using a Inflated 3D ConvNet as backbone, this paper introduces a novel automatic violence detection approach that outperforms state-of-the-art existing proposals. Most of those proposals consider a pre-processing step to only focus on some regions of interest in the scene, i.e., those actually containing a human subject. In this regard, this paper also reports the results of an extensive analysis on whether and how the context can affect or not the adopted classifier performance. The experiments show that context-free footage yields substantial deterioration of the classifier performance (2% to 5%) on publicly available datasets. However, they also demonstrate that performance stabilizes in context-free settings, no matter the level of context restriction applied. Finally, a cross-dataset experiment investigates the generalizability of results obtained in a single-collection experiment (same dataset used for training and testing) to cross-collection settings (different datasets used for training and testing).

Surgical workflow recognition with 3DCNN for Sleeve Gastrectomy

PurposeSurgical workflow recognition is a crucial and challenging problem when building a computer-assisted surgery system. Current techniques focus on utilizing a convolutional neural network and a recurrent neural network (CNN–RNN) to solve the surgical workflow recognition problem. In this paper, we attempt to use a deep 3DCNN to solve this problem.MethodsIn order to tackle the surgical workflow recognition problem and the imbalanced data problem, we implement a 3DCNN workflow referred to as I3D-FL-PKF. We utilize focal loss (FL) to train a 3DCNN architecture known as Inflated 3D ConvNet (I3D) for surgical workflow recognition. We use prior knowledge filtering (PKF) to filter the recognition results.ResultsWe evaluate our proposed workflow on a large sleeve gastrectomy surgical video dataset. We show that focal loss can help to address the imbalanced data problem. We show that our PKF can be used to generate smoothed prediction results and improve the overall accuracy. We show that the proposed workflow achieves 84.16% frame-level accuracy and reaches a weighted Jaccard score of 0.7327 which outperforms traditional CNN–RNN design.ConclusionThe proposed workflow can obtain consistent and smooth predictions not only within the surgical phases but also for phase transitions. By utilizing focal loss and prior knowledge filtering, our implementation of deep 3DCNN has great potential to solve surgical workflow recognition problems for clinical practice.

AI 딥러닝을 이용한 싸움(다툼) 행동 인식 방법

In this study, we developed an AI deep learning-based fighting behavior recognition method for a video surveillance system and proved its effectiveness through various experiments. The proposed method consists of a two-step fighting behavior recognition framework. First, continuous video frames of the target surveillance video are transmitted to the Inflated 3D ConvNet (I3D) network, which shows a good behavior-recognition performance, to extract the spatiotemporal features. These extracted 3D features are then used as the inputs in the next step, where a fight situation is detected using a classification model consisting of a fully connected layer. To use the proposed aggressive behavior detection framework effectively, first, it is necessary to train the fight detection model. However, it is not possible to collect sufficient fighting videos in various outdoor environments. To overcome this limitation, we generated a large amount of learning data through data augmentation. Therefore, instead of directly learning from the training videos transmitted to the I3D network, the classifier trains itself to recognize various fighting actions using the Kinetics video dataset. That is, the action features are extracted from the transmitted consecutive frames using the pretrained I3D network and subsequently used to train the fully connected layer classification model. In addition, we proposed a learning method that includes recognizing ambiguous conflict boundaries using multiple instance learning to mitigate the ambiguous starting and ending of the contention videos. The effectiveness of the proposed method was verified through several experiments by drawing comparisons between the present results and those of the previously reported studies.

Three-stream spatio-temporal attention network for first-person action and interaction recognition

The problem of action and interaction recognition of human activities from the perspective of first-person view-point is an interesting area of research in the field of human action recognition (HAR). This paper presents a data-driven spatio-temporal network to combine different modalities computed from first-person videos using a temporal attention mechanism. First, our proposed approach uses three-stream inflated 3D ConvNet (I3D) to extract low-level features from RGB frame difference (FD), optical flow (OF) and magnitude-orientation (MO) streams. An I3D network has the advantage to directly learn spatio-temporal features over short video snippets (like 16 frames). Second, the extracted features are fused together and fed to a Bidirectional long short-term memory (BiLSTM) network to model high-level temporal feature sequences. Third, we propose to incorporate attention mechanism with our BiLSTM network to automatically select the most relevant temporal snippets in the given video sequence. Finally, we conducted extensive experiments and achieve state-of-the-art results on JPL (98.5%), NUS (84.1%), UTK (91.5%) and DogCentric (83.3%) datasets. These results show that features extracted from three-stream network are complementary to each other, and attention mechanism further improves the results by a large margin than previous attempts based on handcrafted and deep features.

Attentive 3D-Ghost Module for Dynamic Hand Gesture Recognition with Positive Knowledge Transfer.

Hand gesture recognition is a challenging topic in the field of computer vision. Multimodal hand gesture recognition based on RGB-D is with higher accuracy than that of only RGB or depth. It is not difficult to conclude that the gain originates from the complementary information existing in the two modalities. However, in reality, multimodal data are not always easy to acquire simultaneously, while unimodal RGB or depth hand gesture data are more general. Therefore, one hand gesture system is expected, in which only unimordal RGB or Depth data is supported for testing, while multimodal RGB-D data is available for training so as to attain the complementary information. Fortunately, a kind of method via multimodal training and unimodal testing has been proposed. However, unimodal feature representation and cross-modality transfer still need to be further improved. To this end, this paper proposes a new 3D-Ghost and Spatial Attention Inflated 3D ConvNet (3DGSAI) to extract high-quality features for each modality. The baseline of 3DGSAI network is Inflated 3D ConvNet (I3D), and two main improvements are proposed. One is 3D-Ghost module, and the other is the spatial attention mechanism. The 3D-Ghost module can extract richer features for hand gesture representation, and the spatial attention mechanism makes the network pay more attention to hand region. This paper also proposes an adaptive parameter for positive knowledge transfer, which ensures that the transfer always occurs from the strong modality network to the weak one. Extensive experiments on SKIG, VIVA, and NVGesture datasets demonstrate that our method is competitive with the state of the art. Especially, the performance of our method reaches 97.87% on the SKIG dataset using only RGB, which is the current best result.

An Improved Two-stream Inflated 3D ConvNet for Abnormal Behavior Detection

An Improved Two-stream Inflated 3D ConvNet for Abnormal Behavior Detection

Online Spatio-Temporal Action Detection in Long-Distance Imaging Affected by the Atmosphere

Current state-of-the-art approaches for spatio-temporal action detection deal with stable videos and quite sterilized environments, as seen in the UCF-101 benchmark. In addition, the objects of interest are typically relatively close to the camera, and therefore fairly clear and easily distinguished. This study presents an approach method for online human action detection in long-distance imaging affected by atmospheric distortions. We created a unique dataset of typical actions in long-range imaging. Various CNN frameworks were examined for the initial moving object detection phase, including 2D, 3D, one stream, and two-stream (RGB frames and optical flow). The basic object detection methods examined within these frameworks include the YOLOv3 and an extension of the inflated 3D ConvNet with a Feature-Fused Single Shot Multibox Detector (FFSSD) to improve small object detection. To cope with the harmful effect of the spatio-temporal random movements induced by atmospheric effects on motion estimation, we first fit the optical flow stream characteristics to a temporally noisy turbulent environment. A significant improvement of the action detection quality under such noisy conditions was obtained by constructing an online tracking algorithm that incrementally constructs and labels the objects' tracks from the network's frame-level detections. Experimental results show that our approach outperforms the state-of-the-art on our dataset in terms of the mAP measure.

Pose-Guided Inflated 3D ConvNet for action recognition in videos

Human action recognition in videos is still an important while challenging task. Existing methods based on RGB image or optical flow are easily affected by clutters and ambiguous backgrounds. In this paper, we propose a novel Pose-Guided Inflated 3D ConvNet framework (PI3D) to address this issue. First, we design a spatial–temporal pose module, which provides essential clues for the Inflated 3D ConvNet (I3D). The pose module consists of pose estimation and pose-based action recognition. Second, for multi-person estimation task, the introduced pose estimation network can determine the action most relevant to the action category. Third, we propose a hierarchical pose-based network to learn the spatial–temporal features of human pose. Moreover, the pose-based network and I3D network are fused at the last convolutional layer without loss of performance. Finally, the experimental results on four data sets (HMDB-51, SYSU 3D, JHMDB and Sub-JHMDB) demonstrate that the proposed PI3D framework outperforms the existing methods on human action recognition. This work also shows that posture cues significantly improve the performance of I3D.

Development of a Mimic Robot—Learning From Demonstration Incorporating Object Detection and Multiaction Recognition

With the trends of DIY movements and maker economy, great needs for the applications of low volume automation are expected. Various approaches have been proposed by moving from low level coding to more intuitive methods for robots to perform tasks. In this article, a vision-based learning from demonstration (LfD) system is proposed, where a mimic robot is developed to learn from human demonstrations to manipulate a coffee maker. Employing two cameras as sensing devices, the proposed LfD system integrates object detection and multiaction recognition to build an action base, where object recognition is achieved by You only look once (YOLO) deep learning architecture, while multiaction recognition is carried out by an Inflated 3D ConvNet (I3D) deep learning network followed by a proposed statistically fragmented approach to enhance the action recognition results. Based on the sequential order of actions in the action base, tasks demonstrated by humans can be reproduced by the mimicking robot. To validate the proposed approach, manipulation of a coffee maker is adopted as an example, where the proposed system is capable of reproducing the tasks demonstrated by a human under different circumstances without knowledge in computer programming or robotics.

Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks

Human action recognition in video is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. This paper presents a new Twin Spatio-Temporal Convolutional Neural Network (TSTCNN) for this purpose. When applied to table tennis, it is possible to detect and recognize 20 table tennis strokes. The model has been trained on a specific dataset, so called TTStroke-21, recorded in natural conditions at the Faculty of Sports of the University of Bordeaux. Our model takes as inputs an RGB image sequence and its computed Optical Flow. The proposed Twin architecture is a two stream network both comprising 3 spatio-temporal convolutional layers, followed by a fully connected layer where data are fused. Our method reaches an accuracy of 91.4% against 43.1% for our baseline, a Two-Stream Inflated 3D ConvNet (I3D).