Recognition Scenarios Research Articles

Action recognition in rehabilitation: combining 3D convolution and LSTM with spatiotemporal attention.

This study addresses the limitations of traditional sports rehabilitation, emphasizing the need for improved accuracy and response speed in real-time action detection and recognition in complex rehabilitation scenarios. We propose the STA-C3DL model, a deep learning framework that integrates 3D Convolutional Neural Networks (C3D), Long Short-Term Memory (LSTM) networks, and spatiotemporal attention mechanisms to capture nuanced action dynamics more precisely. Experimental results on multiple datasets, including NTU RGB + D, Smarthome Rehabilitation, UCF101, and HMDB51, show that the STA-C3DL model significantly outperforms existing methods, achieving up to 96.42% accuracy and an F1 score of 95.83% on UCF101, with robust performance across other datasets. The model demonstrates particular strength in handling real-time feedback requirements, highlighting its practical application in enhancing rehabilitation processes. This work provides a powerful, accurate tool for action recognition, advancing the application of deep learning in rehabilitation therapy and offering valuable support to therapists and researchers. Future research will focus on expanding the model's adaptability to unconventional and extreme actions, as well as its integration into a wider range of rehabilitation settings to further support individualized patient recovery.

ProtoMed: Prototypical networks with auxiliary regularization for few-shot medical image classification

Although deep learning has shown impressive results in computer vision, the scarcity of annotated medical images poses a significant challenge for its effective integration into Computer-Aided Diagnosis (CAD) systems. Few-Shot Learning (FSL) opens promising perspectives for image recognition in low-data scenarios. However, applying FSL for medical image diagnosis presents significant challenges, particularly in learning disease-specific and clinically relevant features from a limited number of images. In the medical domain, training samples from different classes often exhibit visual similarities. Consequently, certain medical conditions may present striking resemblances, resulting in minimal inter-class variation. In this paper, we propose a prototypical network-based approach for few-shot medical image classification for low-prevalence diseases detection. Our method leverages meta-learning to use prior knowledge gained from common diseases, enabling generalization to new cases with limited data. However, the episodic training inherent in meta-learning tends to disproportionately emphasize the connections between elements in the support set and those in the query set, which can compromise the understanding of complex relationships within medical image data during the training phase. To address this, we propose an auxiliary network as a regularizer in the meta-training phase, designed to enhance the similarity of image representations from the same class while enforcing dissimilarity between representations from different classes in both the query and support sets. The proposed method has been evaluated using three medical diagnosis problems with different imaging modalities and different levels of visual imaging details and patterns. The obtained model is lightweight and efficient, demonstrating superior performance in both efficiency and accuracy compared to state-of-the-art. These findings highlight the potential of our approach to improve performance in practical applications, balancing resource limitations with the need for high diagnostic accuracy.

Advanced montane bird monitoring using self-supervised learning and transformer on passive acoustic data

Passive acoustic monitoring combined with deep learning-based bird sound classifiers is an effective tool, particularly in remote areas. While self-supervised learning has recently excelled in natural language processing and image recognition, its application to bird sound recognition remains limited. This study proposes an innovative self-supervised learning approach, which leverages vast amounts of passive acoustic recordings for pre-training, followed by fine-tuning of target species. Compared to the three state-of-the-art models based on transfer learning from ImageNet, the proposed method demonstrated improvements across all species, with even more significant gains for tail-end species. These results confirm that domain-specific pre-training in self-supervised learning enhances downstream recognition tasks and provides greater robustness, benefiting tail-end species in imbalanced ecological datasets. Our experiments further demonstrate that integrating open-source datasets and data augmentation techniques is the most effective strategy for mitigating data imbalances and cross-domain issues. In addition, introducing a ‘catch-all’ category into training datasets has been shown to improve model robustness in open set recognition scenarios. We also identified the minimum viable sample size requirements for our proposed model and explored the impact of overlapping bird vocalizations during dawn choruses on model performance. Targeting 31 bird species in the montane regions of subtropical Taiwan, the model achieved a class-wise mean average precision of 0.782 and an overall precision of 85.6 % at the F0.5 threshold in dawn chorus soundscape recordings. This study confirms the effectiveness and advantages of self-supervised learning in bird sound recognition, supporting long-term monitoring of bird distribution and vocal activity in remote montane areas.

A Review of Research on Image Classification Methods

As one of the important research directions in the field of computer vision, image classification has a wide range of applications. Image classification is an important foundation for pattern recognition, machine learning, and artificial intelligence. Image classification generally includes three steps: region of interest selection, feature extraction, and classifier modeling. Among them, feature extraction of images is an important foundation for completing other tasks. In most pattern recognition scenarios, appropriate feature representation is a crucial step that directly affects the performance of the entire classification system. Among them, the most representative is the deep learning method that directly learns complex feature expressions from massive data. This article first reviews the research background and development history of image classification, then analyzes the application of image classification in different fields and lists case studies. It also outlines traditional image classification methods and mainstream image classification techniques in deep learning, including convolutional neural networks, model pre-training, and data augmentation. At the end of the article, the main challenges currently faced by image classification were analyzed, and future directions for improvement and technologies that can be combined were discussed. This review aims to provide researchers in image classification with certain research directions.

Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition

The ubiquitous availability of smartphones and smartwatches with integrated inertial measurement units (IMUs) enables straightforward capturing of human activities through collecting movement data. For specific applications of sensor based human activity recognition (HAR), however, logistical challenges and burgeoning costs render especially the ground truth annotation of such data a difficult endeavor, resulting in limited scale and diversity of datasets available for deriving effective HAR systems and less than ideal recognition capabilities. Transfer learning, i.e., leveraging publicly available labeled datasets to first learn useful representations that can then be fine-tuned using limited amounts of labeled data from a target domain, can alleviate some of the performance issues of contemporary HAR systems. Yet they can fail when the differences between source and target conditions are too large and / or only few samples from a target application domain are available – each of which are typical challenges in real-world human activity recognition scenarios. In this paper, we present an approach for economic use of publicly available labeled HAR datasets for effective transfer learning. We introduce a novel transfer learning framework–Cross-Domain HAR–which follows the teacher-student self-training paradigm to more effectively recognize activities with very limited label information. It bridges conceptual gaps between source and target domains, including sensor locations and type of activities. Cross-Domain HAR enables substantial performance improvements over the state-of-the-art in sensor-based HAR scenarios. Through our extensive experimental evaluation on a range of benchmark datasets we specifically demonstrate the effectiveness of our approach for practically relevant few shot activity recognition scenarios. We also present a detailed analysis into how the individual components of our framework affect downstream performance and provide practical suggestions for using the framework in real-world applications.

HNet: A deep learning based hybrid network for speaker dependent visual speech recognition

Visual Speech Recognition (VSR) is a popular area in computer vision research, attracting interest for its ability to precisely analyze lip motion and seamlessly convert them into textual representation. VSR systems leverage visual features to augment the understanding of automated speech and predict text. VSR finds various applications, including enhancing speech recognition in scenarios with degraded acoustic signals, aiding individuals with hearing impairments, bolstering security by reducing reliance on text based passwords, facilitating biometric authentication for liveness detection, and enabling underwater communications. Despite the various techniques proposed for improving the resilience and precision of automatic speech recognition, VSR still has challenges like homophones of words, gradient descent issues with varying sequence lengths, and lip reading demands accounting for short and long range correlations between consecutive video frames. We have proposed a hybrid network (HNet) with multilayered three dimensional dilated convolution neural network (3D-CNN). The spatio-temporal feature extraction process will be facilitated by dilated 3D-CNN. HNet integrates two bidirectional recurrent neural networks (BiGRU and BiLSTM) to process the feature sequences bidirectionally to establish the temporal relationship. The fusion of BiGRU-BiLSTM capabilities allows the model to process feature sequences more comprehensively and effectively. The proposed work focuses on face based biometric authentication for liveness detection using the VSR model to boost security against face spoofing. The existing face based biometric systems are widely used for individual authentication and verification but are still vulnerable to 3D masks and adversarial attacks. The VSR system can be added to existing face based verification systems as a second level authentication technique to identify a person with liveness. The working ideology of the VSR system will be based on the challenge response technique, where a person has to pronounce the passcode silently displayed on the screen. The VSR model assesses its effectiveness using word error rate (WER), which matches the pronounced passcode to the one presented on the screen. Overall, the proposed work aims to enhance the accuracy of VSR so that it can be combined with existing face based authentication systems. The proposed system outperforms the existing VSR system and obtained 1.3% WER. The significance of the proposed hybrid model is that it efficiently captures temporal dependencies, enhancing context embedding, improving robustness to input variability, reducing information loss, and enhancing performance and accuracy in modeling and analyzing passcode pronunciation patterns.

AIRHF-Net: an adaptive interaction representation hierarchical fusion network for occluded person re-identification

To tackle the high resource consumption in occluded person re-identification, sparse attention mechanisms based on Vision Transformers (ViTs) have become popular. However, they often suffer from performance degradation with long sequences, omission of crucial information, and token representation convergence. To address these issues, we introduce AIRHF-Net: an Adaptive Interaction Representation Hierarchical Fusion Network, named AIRHF-Net, designed to enhance pedestrian identity recognition in occluded scenarios. Our approach begins with the development of an Adaptive Local-Window Interaction Encoder (AL-WIE), which aims to overcome the inherent subjective limitations of traditional sparse attention mechanisms. This innovative encoder merges window attention, adaptive local attention, and interaction attention, facilitating automatic localization and focusing on visible pedestrian regions within images. It effectively extracts contextual information from window-level features while minimizing the impact of occlusion noise. Additionally, recognizing that ViTs may lose spatial information in deeper structural layers, we implement a Local Hierarchical Encoder (LHE). This component segments the input sequence in the spatial dimension, integrating features from various spatial positions to construct hierarchical local representations that substantially enhance feature discriminability. To further augment the quality and breadth of datasets, we adopt an Occlusion Data Augmentation Strategy (ODAS), which bolsters the model’s capacity to extract critical information under occluded conditions. Extensive experiments demonstrate that our method achieves improved performance on the Occluded-DukeMTMC dataset, with a rank-1 accuracy of 69.6% and an mAP of 61.6%.

Advancements in Image Recognition: Comparing CNNs and Vision Transformers

Abstract. This paper explores advancements in image recognition technologies, highlighting the shift from conventional methodologies to contemporary deep learning techniques, specifically focusing on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The study examines key architectures including CNNs, and various Transformer-based models, analyzing their performance evaluating their effectiveness in diverse tasks such as image classification, object detection, and facial recognition. The research highlights the strengths and limitations of CNNs and ViTs, focusing on their ability to handle complex and diverse datasets. A detailed comparative analysis is conducted, emphasizing performance metrics, robustness, and adaptability across different image recognition scenarios. The results reveal that while CNNs excel in traditional image processing tasks, ViTs demonstrate significant improvements in capturing long-range dependencies, thereby enhancing recognition accuracy in more complex contexts. This analysis offers critical perspectives on selecting and applying image recognition models, guiding future exploration and practical use in various industries. It underscores the impact of deep learning innovations on advancing image recognition capabilities and highlights potential directions for ongoing development in the field.

Combinative model compression approach for enhancing 1D CNN efficiency for EIT-based Hand Gesture Recognition on IoT edge devices

Tiny Machine Learning is rapidly evolving in edge computing and intelligent Internet of Things (IoT) devices. This paper investigates model compression techniques with the aim of determining their compatibility when combined, and identifying an effective approach to improve inference speed and energy efficiency within IoT edge devices. The study is carried out on the application scenario of Hand Gesture Recognition (HGR) based on Electrical Impedance Tomography (EIT), which involves complex signal processing and needs real-time processing and energy efficiency. Therefore, a customized 1-Dimensional Convolutional Neural Network (1D CNN) HGR classification model has been designed. An approach based on strategically combining model compression techniques was then implemented resulting in a model customized for faster inference and improved energy efficiency for IoT embedded devices. The model size became compact at 10.42 kB, resulting in a substantial size reduction of 98.8%, and an inference gain of 94.73% on a personal computer with approximately 8.56% decrease in accuracy. The approach of combinative model compression techniques was applied to a wide range of edge-computing IoT devices with limited processing power, resulting in a significant improvement in model execution speed and energy efficiency for these devices. Specifically, there was an average power consumption gain of 52% for Arduino Nano BLE and 34.05% for Raspberry Pi 4. Inference time was halved for Arduino Nano BLE Sense, Nicla Sense, and Raspberry Pi 4, with a remarkable gain of 94% for ESP32.

Capturing multiple emotions from conversational data using fine tuned transformers

Emotion detection is one of the crucial topics of Natural Language Processing (NLP) in recent years, and now one of the biggest motivating factors in correct identification and interpretation of a wide range of emotional expressions in textual data. This research examines the use of Bidirectional Encoder Representations from Transformers (BERT), a trained transformer model for emotion detection in textual data. This analysis evaluates how good BERT is at identifying emotions such as surprise, anger, fear, happiness and sadness compared with ordinary machine learning as well as deep learning techniques. The addition of weighted emotions approach enhances the model performance and gives a deeper emotional context awareness making it more effective to deal with complicated emotional utterances. The purpose of the research is to develop a fine-tuned BERT model with a weighted emotion framework to enhance the accuracy of emotion classification in conversational text. The context is improving emotion recognition in scenarios where multiple emotions co-exist, addressing limitations in traditional models by capturing subtle and overlapping emotional expressions. In terms of training the methods multiple datasets are considered and also the research examines various models performance. The article further discusses possible application areas of BERT modified in light of NLP.

SOAMC: A Semi-Supervised Open-Set Recognition Algorithm for Automatic Modulation Classification

Traditional automatic modulation classification methods operate under the closed-set assumption, which proves to be impractical in real-world scenarios due to the diverse nature of wireless technologies and the dynamic characteristics of wireless propagation environments. Open-set environments introduce substantial technical challenges, particularly in terms of detection effectiveness and computational complexity. To address the limitations of modulation classification and recognition in open-set scenarios, this paper proposes a semi-supervised open-set recognition approach, termed SOAMC (Semi-Supervised Open-Set Automatic Modulation Classification). The primary objective of SOAMC is to accurately classify unknown modulation types, even when only a limited subset of samples is manually labeled. The proposed method consists of three key stages: (1) A signal recognition pre-training model is constructed using data augmentation and adaptive techniques to enhance robustness. (2) Feature extraction and embedding are performed via a specialized extraction network. (3) Label propagation is executed using a graph convolutional neural network (GCN) to efficiently annotate the unlabeled signal samples. Experimental results demonstrate that SOAMC significantly improves classification accuracy, particularly in challenging scenarios with limited amounts of labeled data and high signal similarity. These findings are critical for the practical identification of complex and diverse modulation signals in real-world wireless communication systems.

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement.

In smart classroom environments, accurately recognizing students' facial expressions is crucial for teachers to efficiently assess students' learning states, timely adjust teaching strategies, and enhance teaching quality and effectiveness. In this paper, we propose a student facial expression recognition model based on multi-scale and deep fine-grained feature attention enhancement (SFER-MDFAE) to address the issues of inaccurate facial feature extraction and poor robustness of facial expression recognition in smart classroom scenarios. Firstly, we construct a novel multi-scale dual-pooling feature aggregation module to capture and fuse facial information at different scales, thereby obtaining a comprehensive representation of key facial features; secondly, we design a key region-oriented attention mechanism to focus more on the nuances of facial expressions, further enhancing the representation of multi-scale deep fine-grained feature; finally, the fusion of multi-scale and deep fine-grained attention-enhanced features is used to obtain richer and more accurate facial key information and realize accurate facial expression recognition. The experimental results demonstrate that the proposed SFER-MDFAE outperforms the existing state-of-the-art methods, achieving an accuracy of 76.18% on FER2013, 92.75% on FERPlus, 92.93% on RAF-DB, 67.86% on AffectNet, and 93.74% on the real smart classroom facial expression dataset (SCFED). These results validate the effectiveness of the proposed method.

Molecular Reconfiguration of Disordered Tellurium Oxide Transistors with Biomimetic Spectral Selectivity.

Reconfigurable devices with field-effect transistor features and neuromorphic behaviors are promising for enhancing data processing capability and reducing power consumption in next-generation semiconductor platforms. However, commonly used 2D materials for reconfigurable devices require additional modulation terminals and suffer from complex and stringent operating rules to obtain specific functionalities. Here, a p-type disordered tellurium oxide is introduced that realizes dual-mode reconfigurability as a logic transistor and a neuromorphic device. Due to the disordered film surface, the enhanced adsorption of oxygen molecules and laser-induced desorption concurrently regulate the carrier concentration in the channel. The device exhibits high-performance p-type characteristics with a field-effect hole mobility of 10.02 cm2 V-1 s-1 and an Ion/Ioff ratio exceeding 106 in the transistor mode. As a neuromorphic device, the vision system exhibits biomimetic bee vision, explicitly responding to the blue-to-ultraviolet light. Finally, in-sensor denoising and invisible image recognition in static and dynamic scenarios are achieved.

Efficient license plate recognition in unconstrained scenarios

Automatic license plate recognition (ALPR) is a critical technology for intelligent transportation systems. Most existing ALPR methods are focused on specific application scenarios. Although there are a few methods that focus on unconstrained scenarios, they are very time-consuming. In this work, we propose an efficient ALPR (EALPR) framework, where we can handle distorted license plates (LP) caused by perspective problems with high efficiency. We design a light LPD structure based on efficient object detection methods and use anchor-free strategies for LPD to alleviate the problem of expensive costs. Benefitting from these optimizations and a united framework structure, the proposed EALPR has real-time efficiency. We evaluate our method on five datasets and the results show that our method achieves state-of-the-art accuracy: 98.15% on OpenALPR(EU), 95.61% on OpenALPR(BR), 99.51% on AOLP(RP), 88.81% on SSIG, 79.41% on CD-HARD. Additionally, our method achieves an impressive speed of 74.9 FPS (Frames Per Second), outperforming existing approaches and demonstrating its efficiency. Our source code can be accessed at https://github.com/wechao18/Efficient-alpr-unconstrained.

Exploring Machine Learning and Deep Learning Techniques for Occluded Face Recognition: A Comprehensive Survey and Comparative Analysis

Face recognition occluded by occlusions, such as glasses or shadows, remains a challenge in many security and surveillance applications. This study aims to analyze the performance of various machine learning and deep learning techniques in face recognition scenarios with occlusions. We evaluate KNN (standard and FisherFace), CNN, DenseNet, Inception, and FaceNet methods combined with a pre-trained DeepFace model using three public datasets: YALE, Essex Grimace, and Georgia Tech. The results show that KNN maintains the highest accuracy, reaching 100% on two datasets (Essex Grimace and YALE), even in the presence of occlusions. Meanwhile, CNN shows strong performance, with accuracy remaining 100% on YALE, both with and without occlusions, although its performance drops slightly on Essex Grimace (94% with occlusion). DenseNet and Inception show a more significant drop in accuracy when faced with occlusion, with DenseNet dropping from 81% to 72% on Essex Grimace and Inception dropping from 100% to 92% on the same dataset. FaceNet + DeepFace excels on more large dataset (Georgia Tech) with 98% accuracy, but its performance drops dramatically to 53% and 70% on Essex Grimace and YALE with occlusion. These findings indicate that while deep learning methods show high accuracy under ideal conditions, machine learning methods such as KNN are more flexible and robust to occlusion in face recognition.

ACA-Net: adaptive context-aware network for basketball action recognition.

The advancements in intelligent action recognition can be instrumental in developing autonomous robotic systems capable of analyzing complex human activities in real-time, contributing to the growing field of robotics that operates in dynamic environments. The precise recognition of basketball players' actions using artificial intelligence technology can provide valuable assistance and guidance to athletes, coaches, and analysts, and can help referees make fairer decisions during games. However, unlike action recognition in simpler scenarios, the background in basketball is similar and complex, the differences between various actions are subtle, and lighting conditions are inconsistent, making action recognition in basketball a challenging task. To address this problem, an Adaptive Context-Aware Network (ACA-Net) for basketball player action recognition is proposed in this paper. It contains a Long Short-term Adaptive (LSTA) module and a Triplet Spatial-Channel Interaction (TSCI) module to extract effective features at the temporal, spatial, and channel levels. The LSTA module adaptively learns global and local temporal features of the video. The TSCI module enhances the feature representation by learning the interaction features between space and channels. We conducted extensive experiments on the popular basketball action recognition datasets SpaceJam and Basketball-51. The results show that ACA-Net outperforms the current mainstream methods, achieving 89.26% and 92.05% in terms of classification accuracy on the two datasets, respectively. ACA-Net's adaptable architecture also holds potential for real-world applications in autonomous robotics, where accurate recognition of complex human actions in unstructured environments is crucial for tasks such as automated game analysis, player performance evaluation, and enhanced interactive broadcasting experiences.

FireDA: A Domain Adaptation-Based Method for Forest Fire Recognition with Limited Labeled Scenarios

Vision-based forest fire detection systems have significantly advanced through Deep Learning (DL) applications. However, DL-based models typically require large-scale labeled datasets for effective training, where the quality of data annotation is crucial to their performance. To address challenges related to the quality and quantity of labeling, a domain adaptation-based approach called FireDA is proposed for forest fire recognition in scenarios with limited labels. Domain adaptation, a subfield of transfer learning, facilitates the transfer of knowledge from a labeled source domain to an unlabeled target domain. The construction of the source domain FBD is initiated, which includes three common fire scenarios: forest (F), brightness (B), and darkness (D), utilizing publicly available labeled data. Subsequently, a novel algorithm called Neighborhood Aggregation-based 2-Stage Domain Adaptation (NA2SDA) is proposed. This method integrates feature distribution alignment with target domain Proxy Classification Loss (PCL), leveraging a neighborhood aggregation mechanism and a memory bank designed for the unlabeled samples in the target domain. This mechanism calibrates the source classifier and generates more accurate pseudo-labels for the unlabeled sample. Consequently, based on these pseudo-labels, the Local Maximum Mean Discrepancy (LMMD) and the Proxy Classification Loss (PCL) are computed. To validate the efficacy of the proposed method, the publicly available forest fire dataset, FLAME, is employed as the target domain for constructing a transfer learning task. The results demonstrate that our method achieves performance comparable to the supervised Convolutional Neural Network (CNN)-based state-of-the-art (SOTA) method, without requiring access to labels from the FLAME training set. Therefore, our study presents a viable solution for forest fire recognition in scenarios with limited labeling and establishes a high-accuracy benchmark for future research.

Sustainable utilization of road assets concerning obscured traffic signs recognition

Traffic sign recognition is crucial for sustainable utilization of road assets. However, on real roads, traffic signs are often obscured, making their recognition challenging. Unfortunately, existing research lacks specific analysis of the impact of traffic sign occlusion and methods to improve recognition in such scenarios. This study quantitatively investigates the relationship between the degree of traffic sign occlusion and recognition accuracy. Moreover, a dedicated deep learning model is proposed, utilizing a multi-scale convolutional stacked input layer, to enhance the recognition of obscured traffic signs. Using the Chinese Traffic Sign Recognition Database (TSRD), this study analyzes the recognition performance of four traffic sign categories: indication signs, prohibition signs, speed limit signs, and warning signs, under different occlusion levels. Three widely used deep learning methods, Visual Geometry Group (VGG), Residual Network (ResNet), and Dense Convolutional Network (DenseNet), are compared with the dedicated model. Experimental results demonstrate a significant decrease in recognition accuracy when traffic signs are obscured. Importantly, the proposed dedicated model outperforms the other methods, achieving accuracies of 80.95%, 90.50%, and 97.22% in the scenarios of 50% occlusion, 25% occlusion, and no occlusion, respectively. This study holds implications for sustainable utilization of road assets.

Improving YOLOv7 for Large Target Classroom Behavior Recognition of Teachers in Smart Classroom Scenarios

Deep learning technology has recently become increasingly prevalent in the field of education due to the rapid growth of artificial intelligence. Teachers’ teaching behavior is a crucial component of classroom teaching activities, and identifying and examining teachers’ classroom teaching behavior is an important way to assess teaching. However, the traditional teaching evaluation method involves evaluating by either listening to the class on-site or playing back the teaching video afterward, which is a time-consuming and inefficient manual method. Therefore, this paper obtained teaching behavior data from a real smart classroom scenario and observed and analyzed the teacher behavior characteristics in this scenario. Aiming at the problems of complex classroom environments and the high similarity between teaching behavior classes, a method to improve YOLOv7 for large target classroom behavior recognition in smart classroom scenarios is proposed. First, we constructed the Teacher Classroom Behavior Data Set (TCBDS), which contains 6660 images covering six types of teaching behaviors: facing the board (to_blackboard, tb), facing the students (to_student, ts), writing on the board (writing, w), teaching while facing the board (black_teach, bt), teaching while facing the students (student_teach, st), and interactive (interact, i). This research adds a large target detection layer to the backbone network so that teachers’ instructional behaviors can be efficiently identified in complex classroom circumstances. Second, the original model’s backbone was extended with an effective multiscale attention module (EMA) to construct cross-scale feature dependencies under various branches. Finally, the bounding box loss function of the original model was replaced with MPDIoU, and a bounding box scaling factor was introduced to propose the Inner_MPDIoU loss function. Experiments were conducted using the TCBDS dataset. The method proposed in this study achieved mAP@.50, mAP@.50:.95, and recall values of 96.2%, 82.5%, and 92.9%, respectively—improvements of 1.1%, 2.0%, and 2.3% over the original model. This method outperformed other mainstream models compared to the current state of the art. The experimental results demonstrate the method’s excellent performance, its ability to identify various classroom behaviors of teachers in realistic scenarios, and its potential to facilitate the analysis and visualization of teacher classroom behaviors.

Sports action recognition algorithm based on multi-modal data recognition

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.