Multi-task Cascaded Convolutional Networks Research Articles

A Fuzzy-Based Transformation to Mitigate the Impact of Adversarial Attacks in Deep Learning-Based Real-Time Surveillance Systems

Deep learning (DL) techniques have become integral to smart city projects, including video surveillance systems (VSS). These advanced technologies offer significant benefits, such as enhanced accuracy and efficiency in monitoring and managing urban environments. However, despite their advantages, these systems are not without vulnerabilities. One of the most pressing challenges is their susceptibility to adversarial attacks, which can lead to critical misclassifications during inference. To address these challenges, our research focuses on developing a more robust smart city VSS. Our research unfolds across two pivotal initiatives. In our initial exploration, we introduce a pioneering framework that extends the reach of adversarial attacks to real-time VSS. A practical manifestation involved implementing a real-time face mask surveillance system based on Multi-Task Cascaded Convolutional Networks (MTCNN) for face detection and MobileNet-v2 for face mask classification, subjecting it to the Fast Gradient Sign Method (FGSM) adversarial attack in real-time. In our subsequent endeavor, we propose a sophisticated defense mechanism deploying Fuzzy Image Transformation as a pre-processing unit (FITP). This strategic defense fortification significantly reinforces our real-time VSS against adversarial intrusions. Experimental findings highlight the effectiveness of the proposed adversarial attack framework in real-time, resulting in a marked reduction in the model's performance from a precision (P) of 93%, recall (R) of 93%, F1 score (F) of 93%, and accuracy (A) of 93% to 22%, 21%, 22%, and 22%, respectively. However, the post-implementation efficacy of our defense mechanism is striking, enhancing the model's average performance to a noteworthy improvement, with P, R, F, and A ascending to 91%, 90%, 91%, and 91%. This research illuminates the vulnerabilities intrinsic to VSS in the face of adversarial threats, underscoring the critical need for heightened awareness and the development of robust defense mechanisms before real-world deployment.

Investigating Digital Illiterate Classification Techniques Based on DeepFace Technology

This paper presents an algorithm to identify digital illiterates by analyzing age and emotions from facial recognition. In this paper, digital illiterates refer to people who struggle in using digital devices. The study assumed that older individuals who display surprised or angry facial expressions while using digital devices are more likely to be identified as digitally illiterate. For age detection, the study used MTCNN (Multi-task Cascaded Convolutional Networks), and for emotion detection, it employed the VGG-Face model. MTCNN detects facial features and landmarks to preprocess images and distinguish facial characteristics. The VGG-Face model uses convolution operations to analyze facial images and classify emotional states. The dataset used in the study consisted of 3,000 facial images collected from the internet. The research team categorized the images into faces of individuals aged over 50, angry expressions, and surprised expressions. The dataset included 411 individuals (13.7%) aged over 50, 163 individuals (5.4%) with angry expressions, and 145 individuals (4.8%) with surprised expressions. Accuracy was estimated by comparing the results from the DeepFace algorithm with those from the research team’s classifications. The DeepFace algorithm achieved 95.77% accuracy in detecting individuals aged over 50, 83.45% accuracy for surprise, and 76.07% for anger. The results demonstrate that it is possible to identify digital illiterates based on their age and emotional expressions and could enable the development of personalized services to directly or indirectly support digital illiterates, potentially improving digital accessibility.

Efficient detection of driver fatigue state based on all-weather illumination scenarios

Among the causes of the annually traffic accidents, driving fatigue is the main culprit. In consequence, it is of great practical significance to carry out the research of driving fatigue detection and early warning system. However, there are still two problems in the latest methods of driving fatigue detection: one is that a single information cannot precisely reflect the actual state of the driver in different fatigue phases, another one is the detection effect is not very well or even difficult to detect under abnormal illumination. In this paper, the multi-task cascaded convolutional networks (MTCNN) and infrared-based remote photo-plethysmography (rPPG) theory are used to extract the driver’s facial and physiological information, and the multi-modal specific fatigue information is deeply excavated, and the multi-modal feature fusion model is constructed to comprehensively analyze the driver’s fatigue variation tendency. Aiming at the matter of low detection accuracy under abnormal illumination, the multi-modal features extracted from visible light images and infrared images are fused by multi-loss reconstruction (MLR) module, and the driving fatigue detection module is established which is based on Bi-LSTM model by utilizing fatigue timing. The experiments were validated under all-weather illumination scenarios and were carried out on the datasets NTHU-DDD, UTA-RLDDD and FAHD. The results show that the multi-modal driving fatigue detection model has better performance than the single-modal model, and the accuracy is improved by 8.1%. In the abnormal illumination such as strong and weak light, the accuracy of the method can reach 91.7% at the highest and 83.6% at the lowest. Meanwhile, in the normal illumination, it can reach 93.2%.

Face mask classification using convolutional neural networks with facial image regions and super resolution

<p>Face mask classification is relevant to public health and safety, so an approach for face mask classification using Multi-Task Cascaded Convolutional Networks (MTCNN) for face detection on image data, ResNet152 architecture for feature extraction, and super-resolution method, BSRGAN, for enhanced image quality was proposed. The classification model was trained by a fully connected layer of neural networks. The goal is to classify each facial image into three classes: the image with a mask, without a mask, or with an incorrectly worn mask. The performance of each classification model on two real-world datasets was evaluated by Accuracy, Precision, Recall, and F1 score for different sets of input patterns which were features extracted from the facial image regions including their combinations. Using multiple image regions, i.e. face, nose, and mouth, as resources for preparing input features showed the improved classification performance compared to using single image regions. In addition, the super-resolution technique applied to medium or large-sized images can improve the performance of the face mask classification model. Our findings may further guide the development for greater effective models and techniques on face mask classification contributing to practical scenarios.</p>

Investigating the Evolving Landscape of Deepfake Technology: Generative AI's Role in it's Generation and Detection

The world of artificial intelligence is constantly changing, with Generative AI and Large Language Models (LLMs) leading the way in bringing new technological advancements. This paper offers a detailed look at these groundbreaking technologies and how they are shaping the digital world today. We explore the technical aspects of Generative AI and LLMs, explain their unique features, and compare them to traditional AI models.One of the key focuses of our research is the growing issue of DeepFakes—artificial intelligence-generated media that presents a significant challenge in verifying content. We conduct a thorough examination of few deepfake detection techniques out of which we will be implementing and analyzing one of them. Our research implements a framework for Deep Fake Image Detection. The suggested solution utilizes a RESNET-50(Residual Network with 50 layers) and MTCNN (Multi-task Cascaded Convolutional Networks) models for detecting whether the images are real or fake. This study conducts the Hypothesis testing for the proposed solution taking in consideration that the current Deepfake detection algorithms are less effective in detecting highly realistic Deepfakes compared to less sophisticated manipulations. By investigating the convergence of deep learning, neural networks, and sophisticated algorithms, we set the stage for advancements in AI-based content verification.

ENHANCING DEEPFAKE DETECTION: LEVERAGING MTCNN AND INCEPTION RESNET

In response to the escalating threat posed by manipulated facial imagery in the digital age, our research project is dedicated to developing an advanced framework for the detection and classification of such content, augmented by a proactive user reporting mechanism. Leveraging state-of-the-art deep learning models like Multi-Task Cascaded Convolutional Networks (MTCNN) and InceptionResnetV1, our framework achieves an impressive accuracy rate of 92% in distinguishing between genuine and manipulated faces. The integration of explainability methods such as Grad-CAM enhances model interpretability, empowering users to understand model predictions. Additionally, our user-centric reporting interface enables active user participation in identifying and flagging potentially manipulated content, fostering transparency and accountability in digital media platforms. With the continued proliferation of deepfake technology, our research endeavors not only advance facial image analysis techniques but also uphold principles of trust and integrity in the digital realm, aiming to safeguard the credibility of information dissemination through vigilance, innovation, and collaborative action. Keywords: Deepfake detection, Facial image manipulation, Deep learning models, User reporting mechanism, Model interpretability, Transparency in digital media, Trust and integrity, Information dissemination, Vigilance, Collaborative action.

Age-Invariant Face Recognition Using FaceNet and Multi-task Cascaded Convolutional Networks (MTCNN)

In an era where face recognition technology serves as a cornerstone for security, surveillance, and human-computer interaction, the persistence of age-related variations in facial appearance presents a significant challenge. This research introduces an innovative solution that amalgamates the capabilities of the FaceNet deep learning model with Multi-task Cascaded Convolutional Networks (MTCNN) to achieve precise and robust face recognition across diverse age groups. Leveraging FaceNet's prowess in extracting distinctive features from facial images and mapping them to a high-dimensional feature space for efficient face matching, our system incorporates MTCNN as a pre-processing step to accurately detect and align faces, effectively mitigating age-related changes in facial geometry. Crucially, this novel approach obviates the need for age-specific databases and age group categorization, rendering it a versatile and practical solution for age-invariant face recognition. Rigorous experimentation on benchmark datasets underscores the system's resilience and accuracy across a broad spectrum of age groups, culminating in state-of-the-art results in age-invariant face recognition. The implications of this proposed approach are profound, promising to fortify security measures and enhance user experiences in applications such as access control, personal identification, and customer service, while ensuring dependable and accurate face recognition, irrespective of an individual's age. Key Words: Face recognition, face aging, generative adversarial networks

Design of an online education student learning status evaluation model based on dual-improved neural networks

With the continuous development of network technology, online education has become an important form of education. However, in the online education model, it is difficult for educators to effectively evaluate students' learning status, and using a learning status evaluation model can effectively solve this problem. The main goal of this model is to comprehensively evaluate students' learning behavior, progress, and outcomes, in order to understand their learning status, provide effective teaching feedback to teachers, help students improve learning methods, and improve learning efficiency. The current automatic evaluation model for student learning status has certain limitations in terms of applicability and accuracy. A student learning state evaluation model based on Multi task Cascaded Convolutional Networks (MTCNN) is proposed to address the effectiveness of online education student learning state evaluation. Use the facial image acquisition function to extract students' facial features, process each feature through label classification, and then analyze the students' attention and learning emotions. Finally, analyze the effectiveness of the research method application. The results showed that the train_loss value of the learning state evaluation model proposed in the study can be reduced to about 0.1; the train_acc value can reach more than 95 %, and the overall volatility is small; the overall evaluation accuracy of facial expressions can reach 74.71 %, which is significantly better than cpc, VGG19 and other evaluation methods; compared with the comprehensive evaluation results and multi-modal analysis methods, only two evaluations at the critical value are different. The experimental results show that the online education students’ learning status evaluation model designed by the research has a high accuracy rate and has a certain application potential in the field of online education.

Edge-Oriented Social Distance Monitoring System Based on MTCNN

Social distance monitoring (SDM) systems are vital in fighting the spread of the coronavirus (COVID-19). Existing SDM systems employ bounding box-method, which imposes inaccurate distance estimation due to the high variance in its output coordinates. To solve this problem, an SDM system based on multitask cascaded convolutional neural networks (MTCNN) is proposed. Instead of using bounding box coordinates, face detection and facial landmarks localization of MTCNN is used to provide fixed coordinates and increase the distance estimation accuracy of SDM. However, while the accuracy issue is solved by using MTCNN, the SDM system suffer from large computational requirements due to the cascaded networks added on top of the distance estimation process. To deal with this challenge, a constrained optimization technique is employed to each stage of MTCNN with the goal of reducing its hardware requirements while keeping the same reliability as the original implementation. Experimental results show that the SDM system based on the optimized MTCNN achieves higher accuracy performance with reduced computational requirements as compared to conventional SDM systems. This allows the proposed SDM system using optimized MTCNN to be deployed efficiently on edge devices.

Proposed Face Detection Classification Model Based on Amazon Web Services Cloud (AWS)

One of the most important features of the Amazon Web Services (AWS) cloud is that the program can be run and accessed from any location. You can access and monitor the result of the program from any location, saving many images and allowing for faster computation. This work proposes a face detection classification model based on AWS cloud aiming to classify the faces into two classes: a non-permission class, and a permission class, by training the real data set collected from our cameras. The proposed Convolutional Neural Network (CNN) cloud-based system was used to share computational resources for Artificial Neural Networks (ANN) to reduce redundant computation. The test system uses Internet of Things (IoT) services through our cameras system to capture the images and upload them to the Amazon Simple Storage Service (AWS S3) cloud. Then two detectors were running, Haar cascade and multitask cascaded convolutional neural networks (MTCNN), at the Amazon Elastic Compute (AWS EC2) cloud, after that the output results of these two detectors are compared using accuracy and execution time. Then the classified non-permission images are uploaded to the AWS S3 cloud. The validation accuracy of the offline augmentation face detection classification model reached 98.81%, and the loss and mean square error were decreased to 0.0176 and 0.0064, respectively. The execution time of all AWS cloud systems for one image when using Haar cascade and MTCNN detectors reached three and seven seconds, respectively.

Covid-19 Control: Face Mask Detection Using Deep Learning for Balanced and Unbalanced Dataset

Facemask wearing is becoming a norm in our daily lives to curb the spread of Covid-19. Ensuring facemasks are worn correctly is a topic of concern worldwide. It could go beyond manual human control and enforcement, leading to the spread of this deadly virus and many cases globally. The main aim of wearing a facemask is to curtail the spread of the covid-19 virus, but the biggest concern of most deep learning research is about who is wearing the mask or not, and not who is incorrectly wearing the facemask while the main objective of mask wearing is to prevent the spread of the covid-19 virus. This paper compares three state-of-the- art object detection approaches: Haarcascade, Multi-task Cascaded Convolutional Networks (MTCNN), and You Only Look Once version 4 (YOLOv4) to classify who is wearing a mask, who is not wearing a mask, and most importantly, who is incorrectly wearing the mask in a real-time video stream using FPS as a benchmark to select the best model. Yolov4 got about 40 Frame Per Seconds (FPS), outperforming Haarcascade with 16 and MTCNN with 1.4. YOLOv4 was later used to compare the two datasets using Intersection over Union (IoU) and mean Average Precision (mAP) as a comparative measure; dataset2 (balanced dataset) performed better than dataset1 (unbalanced dataset). Yolov4 model on dataset2 mapped and detected images of masks worn incorrectly with one correct class label rather than giving them two label classes with uncertainty in dataset1, this work shows the advantage of having a balanced dataset for accuracy. This work would help decrease human interference in enforcing the COVID-19 face mask rules and create awareness for people who do not comply with the facemask policy of wearing it correctly. Hence, significantly reducing the spread of COVID-19.

A real-time driver fatigue identification method based on GA-GRNN.

It is of great practical and theoretical significance to identify driver fatigue state in real time and accurately and provide active safety warning in time. In this paper, a non-invasive and low-cost method of fatigue driving state identification based on genetic algorithm optimization of generalized regression neural network model is proposed. The specific work is as follows: (1) design simulated driving experiment and real driving experiment, determine the fatigue state of drivers according to the binary Karolinska Sleepiness Scale (KSS), and establish the fatigue driving sample database. (2) Improved Multi-Task Cascaded Convolutional Networks (MTCNN) and applied to face detection. Dlib library was used to extract the coordinate values of face feature points, collect the characteristic parameters of driver's eyes and mouth, and calculate the Euler Angle parameters of head posture. A fatigue identification model was constructed by using multiple characteristic parameters. (3) Genetic Algorithm (GA) was used to find the optimal smooth factor of Generalized Regression Neural Network (GRNN) and construct GA-GRNN fatigue driving identification model. Compared with K-Nearest Neighbor (KNN), Random Forest (RF), and GRNN fatigue driving identification algorithms. GA-GRNN has the best generalization ability and high stability, with an accuracy of 93.3%. This study provides theoretical and technical support for the application of driver fatigue identification.

Pedestrian Detection Using Integrated Aggregate Channel Features and Multitask Cascaded Convolutional Neural-Network-Based Face Detectors.

Pedestrian detection is a challenging task, mainly owing to the numerous appearances of human bodies. Modern detectors extract representative features via the deep neural network; however, they usually require a large training set and high-performance GPUs. For these cases, we propose a novel human detection approach that integrates a pretrained face detector based on multitask cascaded convolutional neural networks and a traditional pedestrian detector based on aggregate channel features via a score combination module. The proposed detector is a promising approach that can be used to handle pedestrian detection with limited datasets and computational resources. The proposed detector is investigated comprehensively in terms of parameter choices to optimize its performance. The robustness of the proposed detector in terms of the training set, test set, and threshold is observed via tests and cross dataset validations on various pedestrian datasets, including the INRIA, part of the ETHZ, and the Caltech and Citypersons datasets. Experiments have proved that this integrated detector yields a significant increase in recall and a decrease in the log average miss rate compared with sole use of the traditional pedestrian detector. At the same time, the proposed method achieves a comparable performance to FRCNN on the INRIA test set compared with sole use of the Aggregated Channel Features detector.

Multi-face detection and alignment using multiple kernels

Multi-face detection and alignment techniques under unlimited environment are challenging issues. In recent years, the demand for face detection and alignment techniques has increased in many areas, including automatic drive and security. However, some mainstream algorithms, such as the Multi-task Cascaded Convolutional Networks (MTCNN) algorithm cannot transfer to multi-face detection and alignment, which introduces new multi-size, multi-resolution, and multi-angle challenges. This paper proposes an improved algorithm—Multi-face-MTCNN for precise original face detection and alignment algorithm when there is an overlapping face scenario. We design two new network structures: Pixelfusion-MTCNN and Twoconv-MTCNN. Moreover, we propose new data augmentation method and optimized detection process which are applied in the Multi-face-MTCNN algorithm. The limitations associated with single-scale kernel size in MTCNN are solved to obtain a satisfactory performance. Compared to the MTCNN algorithm, experimental analysis of the FDDB dataset show that 1.766% improves Multi-face-MTCNN. Meanwhile, on the WIDER FACE verification benchmark, respectively, with regards to the three sub-datasets, the proposed algorithm’s performance is improved by 3.426%, 2.776%, and 21.576%. Besides, average alignment error of the left eye, right eye, nose, left mouth, and right mouth is performed on the proposed algorithm.

A novel design of a smart interactive guiding robot for busy airports

Abstract The COVID-19 pandemic has had a massive impact on the global aviation industry. As a result, the airline industry has been forced to embrace new technologies and procedures in order to provide a more secure and bio-safe travel. Currently, the role of smart technology in airport systems has expanded significantly as a result of the contemporary Industry 4.0 context. The article presents a novel construction of an intelligent mobile robot system to guide passengers to take the plane at the departure terminals at busy airports. The robot provides instructions to the customer through the interaction between the robot and the customer utilizing voice communications. The usage of the Google Cloud Speech-to-Text API combined with technical machine learning to analyze and understand the customer's requirements are deployed. In addition, we use a face detection technique based on Multi-task Cascaded Convolutional Networks (MTCNN) to predict the distance between the robot and passengers to perform the function. The robot can guide passengers to desired areas in the terminal. The results and evaluation of the implementation process are also mentioned in the article and show promise.

Research on Facial Expression Capture Based on Two-Stage Neural Network

To generate realistic three-dimensional animation of virtual character, capturing real facial expression is the primary task. Due to diverse facial expressions and complex background, facial landmarks recognized by existing strategies have the problem of deviations and low accuracy. Therefore, a method for facial expression capture based on two-stage neural network is proposed in this paper which takes advantage of improved multi-task cascaded convolutional networks (MTCNN) and high-resolution network. Firstly, the convolution operation of traditional MTCNN is improved. The face information in the input image is quickly filtered by feature fusion in the first stage and Octave Convolution instead of the original ones is introduced into in the second stage to enhance the feature extraction ability of the network, which further rejects a large number of false candidates. The model outputs more accurate facial candidate windows for better landmarks recognition and locates the faces. Then the images cropped after face detection are input into high-resolution network. Multi-scale feature fusion is realized by parallel connection of multi-resolution streams, and rich high-resolution heatmaps of facial landmarks are obtained. Finally, the changes of facial landmarks recognized are tracked in real-time. The expression parameters are extracted and transmitted to Unity3D engine to drive the virtual character's face, which can realize facial expression synchronous animation. Extensive experimental results obtained on the WFLW database demonstrate the superiority of the proposed method in terms of accuracy and robustness, especially for diverse expressions and complex background. The method can accurately capture facial expression and generate three-dimensional animation effects, making online entertainment and social interaction more immersive in shared virtual space.

A multi-modal personality prediction system

The behavior, mental-health, emotion, life choices, social nature, and thought patterns of an individual are revealed by personality. Cyber forensics, personalized services, recommender systems are some of the examples of automatic personality prediction. A deep learning based personality prediction system has been developed in this work. Facial and ambient features are extracted from the visual modality using Multi-task Cascaded Convolutional Networks (MTCNN) and ResNet, respectively; the audio features are extracted using the VGGish Convolutional Neural Networks (VGGish CNN), and the text features are extracted using n-gram Convolutional Neural Networks (CNN). The extracted features are then passed to a fully connected layer followed by sigmoid for the final output prediction. Finally, the text, visual and audio modalities are combined in different ways: (i) concatenation of features in multi-modal setting, and (ii) application of different attention mechanisms for fusing features. The dataset released in Chalearn-17 is used for evaluating the performance of the system. From the obtained results, it can be concluded that, the concatenation of features extracted from different modalities attains comparable results with the averaging method (late fusion). It is also shown that a hand full of images are enough for attaining comparable performance.

A Lightweight Convolutional Neural Network for Real-Time Facial Expression Detection

In this paper our group proposes and designs a lightweight convolutional neural network (CNN) for detecting facial emotions in real-time and in bulk to achieve a better classification effect. We verify whether our model is effective by creating a real-time vision system. This system employs multi-task cascaded convolutional networks (MTCNN) to complete face detection and transmit the obtained face coordinates to the facial emotions classification model we designed firstly. Then it accomplishes the task of emotion classification. Multi-task cascaded convolutional networks have a cascade detection feature, one of which can be used alone, thereby reducing the occupation of memory resources. Our expression classification model employs Global Average Pooling to replace the fully connected layer in the traditional deep convolution neural network model. Each channel of the feature map is associated with the corresponding category, eliminating the black box characteristics of the fully connected layer to a certain extent. At the same time, our model marries the residual modules and depth-wise separable convolutions, reducing large quantities of parameters and making the model more portable. Finally, our model is tested on the FER-2013 dataset. It only takes 3.1% of the 16GB memory, that is, only 0.496GB memory is needed to complete the task of classifying facial expressions. Not only can our model be stored in an 872.9 kilobytes file, but also its accuracy has reached 67% on the FER-2013 dataset. And it has good detection and recognition effects on those figures which are out of the dataset.

A deep learning approach to building an intelligent video surveillance system

Recent advances in the field of object detection and face recognition have made it possible to develop practical video surveillance systems with embedded object detection and face recognition functionalities that are accurate and fast enough for commercial uses. In this paper, we compare some of the latest approaches to object detection and face recognition and provide reasons why they may or may not be amongst the best to be used in video surveillance applications in terms of both accuracy and speed. It is discovered that Faster R-CNN with Inception ResNet V2 is able to achieve some of the best accuracies while maintaining real-time rates. Single Shot Detector (SSD) with MobileNet, on the other hand, is incredibly fast and still accurate enough for most applications. As for face recognition, FaceNet with Multi-task Cascaded Convolutional Networks (MTCNN) achieves higher accuracy than advances such as DeepFace and DeepID2+ while being faster. An end-to-end video surveillance system is also proposed which could be used as a starting point for more complex systems. Various experiments have also been attempted on trained models with observations explained in detail. We finish by discussing video object detection and video salient object detection approaches which could potentially be used as future improvements to the proposed system.

A Teaching Evaluation System Based On Visual Recognition Technology

This paper designs and implements a student-centered teaching evaluation system based on face recognition and pose estimation technology. Our work firstly combines classroom attendance and behavior analysis in an evaluation system. For checking attendance, we select student faces as the identification object, employing a multi-task cascaded convolutional networks (MTCNN) as a face detector and a deep learning network FaceNet to extract face features. Then the head pose information is analyzed using Ensemble of Regression Trees (ERT) algorithm, which is able to detect 68 key feature points of faces. At last, we design and implement the whole system, including designs of functional modules, service software, database and telecommunication of various parts. This system can check attendance and collect student behavior information automatically, enhancing the intelligent level of the learning and teaching system.