Average Frame Research Articles

Wildfire Identification Based on an Improved MobileNetV3-Small Model

In this paper, an improved MobileNetV3-Small algorithm model is proposed for the problem of poor real-time wildfire identification based on convolutional neural networks (CNNs). Firstly, a wildfire dataset is constructed and subsequently expanded through image enhancement techniques. Secondly, an efficient channel attention mechanism (ECA) is utilised instead of the Squeeze-and-Excitation (SE) module within the MobileNetV3-Small model to enhance the model’s identification speed. Lastly, a support vector machine (SVM) is employed to replace the classification layer of the MobileNetV3-Small model, with principal component analysis (PCA) applied before the SVM to reduce the dimensionality of the features, thereby enhancing the SVM’s identification efficiency. The experimental results demonstrate that the improved model achieves an accuracy of 98.75% and an average frame rate of 93. Compared to the initial model, the mean frame rate has been elevated by 7.23. The wildfire identification model designed in this paper improves the speed of identification while maintaining accuracy, thereby advancing the development and application of CNNs in the field of wildfire monitoring.

Multi-Dimensional Fuzzy Clustering-Based Trajectory Initialization Algorithm for Infrared Weak Target Trajectories in Robust Clutter Environments

When conducting maneuver target tracking, trajectory initialization plays a crucial role in enhancing the accuracy of tracking algorithms. During maneuver target tracking, the accuracy of the tracking algorithm can be significantly improved through trajectory initialization. However, the traditional trajectory initialization algorithms face issues such as susceptibility to noise interference, lack of universality, and poor robustness in environments with high clutter levels. To address these issues, this study proposes a trajectory initialization algorithm based on multidimensional fuzzy clustering (MDF-clustering). The algorithm utilizes multidimensional feature information of the target, such as speed and irradiance, to determine point trajectory affiliation by assigning weights based on the clustering center of each feature type. Subsequently, it updates the clustering center and weight assignment using the new target features, ultimately deriving the correct trajectory through iterative processes. Experimental results demonstrate that the proposed method achieves an average stable initialization frame number of 3.12 frames, an average correct trajectory initialization rate of 99.59%, an average false trajectory occupancy rate of 0.04%, and an average missed batch rate of 0.06%. These results indicate improvements of at least 0.87 frames, 27.11%, 60.28%, and 6.48%, respectively, in terms of initialization rate, false trajectory rate, and missed batch rate, when compared to traditional methods. The algorithm enhances the accuracy and robustness of trajectory initialization in challenging environments characterized by solid clutter and target maneuvers, offering significant practical value for target tracking in complex scenarios.

Real-Time Smoke Detection in Surveillance Videos Using an Enhanced RT-DETR Framework with Triplet Attention and HS-FPN

This study addresses the urgent need for an efficient and accurate smoke detection system to enhance safety measures in fire monitoring, industrial safety, and urban surveillance. Given the complexity of detecting smoke in diverse environments and under real-time constraints, our research aims to solve challenges related to low-resolution imagery, limited computational resources, and environmental variability. This study introduces a novel smoke detection system that utilizes the real-time detection Transformer (RT-DETR) architecture to enhance the speed and precision of video analysis. Our system integrates advanced modules, including triplet attention, ADown, and a high-level screening-feature fusion pyramid network (HS-FPN), to address challenges related to low-resolution imagery, real-time processing constraints, and environmental variability. The triplet attention mechanism is essential for detecting subtle smoke features, often overlooked due to their nuanced nature. The ADown module significantly reduces computational complexity, enabling real-time operation on devices with limited resources. Furthermore, the HS-FPN enhances the system’s robustness by amalgamating multi-scale features for reliable detection across various smoke types and sizes. Evaluation using a diverse dataset showcased notable improvements in average precision (AP50) and frames per second (FPS) metrics compared to existing state-of-the-art networks. Ablation studies validated the contributions of each component in achieving an optimal balance between accuracy and operational efficiency. The RT-DETR-based smoke detection system not only meets real-time requirements for applications like fire monitoring, industrial safety, and urban surveillance but also establishes a new performance benchmark in this field.

Lightweight algorithm based on you only look once version 5 for multiple class defect detection on wind turbine blade surfaces

A lightweight wind turbine blade surface multiple class defect detection algorithm based on You Only Look Once version 5 (YOLOv5) is proposed to address the issues of low accuracy, poor efficiency, and high power consumption associated with manual and traditional automation algorithms used for detecting surface defects on wind turbine blades. Firstly, by replacing the Concentrated-Comprehensive Convolution Block module in YOLOv5 with the GhostBottleneckv2 module, the overall parameter count of the model is significantly reduced. Additionally, advocating the inclusion of the Convolutional Block Attention Module (CBAM) into the YOLOv5 backbone network to address the issue of small targets and significant scale variations in surface defects found on wind turbine blades. The YOLOv5 feature fusion module replaces the Feature Pyramid Network (FPN) structure with the Bidirectional Feature Pyramid Network (BiFPN) structure to enhance the fusion capability of multi-scale weighted features. Finally, the optimized model files underwent structured pruning through the application of pruning strategies, retraining fine-tuning, and knowledge distillation techniques. The experimental results on the self-made dataset of surface defects on wind turbine blades indicate that the optimized model has a parameter count that is only 48.79% of the small version of the YOLOv5 model. However, its mean Average Precision (mAP) and frame rate have increased by 5.05% and 8.1 frames/s, respectively, reaching 94.56% and 43.7 frames/s. The lightweight algorithm proposed in this study meets the accuracy and real-time requirements for surface defect detection on wind turbine blades.

Deep learning neural network-assisted badminton movement recognition and physical fitness training optimization

This work aims to solve the problem of low accuracy in recognizing the trajectory of badminton movement. This work focuses on the visual system in badminton robots and conducts side detection and tracking of flying badminton in two-dimensional image plane video streams. Then, the cropped video images are input into a convolutional neural network frame by frame. By adding an attention mechanism, it helps identify the badminton movement trajectory. Finally, to address the detection challenge of flying badminton as a small target in video streams, the deep learning one-stage detection network, Tiny YOLOv2, is improved from both the loss function and network structure perspectives. Moreover, it is combined with the Unscented Kalman Filter algorithm to predict the trajectory of badminton movement. Simulation results show that the improved algorithm performs excellently in tracking and predicting badminton trajectories compared with the existing algorithms. The average accuracy of the proposed method for tracking badminton trajectories is 91.40 %, and the recall rate is 84.60 %. The average precision, recall, and frame rate of the measured trajectories in four simple and complex scenarios of badminton flight video streams are 96.7 %, 95.7 %, and 29.2 frames/second, respectively. They are all superior to other classic algorithms. It is evident that the proposed method can provide powerful support for badminton trajectory recognition and help improve the accuracy of badminton movement recognition.

Micro computed tomography analysis of barley during the first 24 hours of germination

BackgroundGrains make up a large proportion of both human and animal diets. With threats to food production, such as climate change, growing sustainable and successful crops is essential to food security in the future. Germination is one of the most important stages in a plant’s lifecycle and is key to the success of the resulting plant as the grain undergoes morphological changes and the development of specific organs. Micro-computed tomography is a non-destructive imaging technique based on the differing x-ray attenuations of materials which we have applied for the accurate analysis of grain morphology during the germination phase.ResultsMicro Computed Tomography conditions and parameters were tested to establish an optimal protocol for the 3-dimensional analysis of barley grains. When comparing optimal scanning conditions, it was established that no filter, 0.4 degrees rotation step, 5 average frames, and 2016 × 1344 camera binning is optimal for imaging germinating grains. It was determined that the optimal protocol for scanning during the germination timeline was to scan individual grains at 0 h after imbibition (HAI) and then the same grain again at set time points (1, 3, 6, 24 HAI) to avoid any negative effects from X-ray radiation or disruption to growing conditions.ConclusionHere we sought to develop a method for the accurate analysis of grain morphology without the negative effects of possible radiation exposure. Several factors have been considered, such as the scanning conditions, reconstruction, and possible effects of X-ray radiation on the growth rate of the grains. The parameters chosen in this study give effective and reliable results for the 3-dimensional analysis of macro structures within barley grains while causing minimal disruption to grain development.

SmokeFireNet: A Lightweight Network for Joint Detection of Forest Fire and Smoke

In recent years, forest fires have been occurring frequently around the globe, affected by extreme weather and dry climate, causing serious economic losses and environmental pollution. In this context, timely detection of forest fire smoke is crucial for realizing real-time early warning of fires. However, fire and smoke from forest fires can spread to cover large areas and may affect distant areas. In this paper, a lightweight joint forest fire and smoke detection network, SmokeFireNet, is proposed, which employs ShuffleNetV2 as the backbone for efficient feature extraction, effectively addressing the computational efficiency challenges of traditional methods. To integrate multi-scale information and enhance the semantic feature extraction capability, a feature pyramid network (FPN) and path aggregation network (PAN) are introduced in this paper. In addition, the FPN network is optimized by a lightweight DySample upsampling operator. The model also incorporates efficient channel attention (ECA), which can pay more attention to the detection of forest fires and smoke regions while suppressing irrelevant features. Finally, by embedding the receptive field block (RFB), the model further improves its ability to understand contextual information and capture detailed features of fire and smoke, thus improving the overall detection accuracy. The experimental results show that SmokeFireNet is better than other mainstream target detection algorithms in terms of average APall of 86.2%, FPS of 114, and GFLOPs of 8.4, and provides effective technical support for forest fire prevention work in terms of average precision, frame rate, and computational complexity. In the future, the SmokeFireNet model is expected to play a greater role in the field of forest fire prevention and make a greater contribution to the protection of forest resources and the ecological environment.

Centralized feature pyramid‐based supervised deep learning for object detection model from GPR data

AbstractTo address low detection accuracy and speed due to the multisolvability of the ground‐penetrating radar signal, we proposed a novel centralized feature pyramid‐YOLOv6l–based model to enhance detection precision and speed in road damage and pipeline detection. The centralized feature pyramid was used to obtain rich intra‐layer features and improve the network performance. Our proposed model achieves higher accuracy compared with the existing detection models. We also built two new evaluating indexes, relative average precision and relative mean average precision, to fully evaluate the detection accuracy. To verify the applicability of our model, we conducted a road field detection experiment on a ground‐penetrating radar dataset we collected and found that the proposed model had good performance in increasing detection precision, achieving the highest mean average precision compared with YOLOv7, YOLOv5 and YOLOx models, with relative mean average precision and frame rate per second at 16.38% and 30.5%, respectively. The detection information for the road damage and pipeline were used to conduct three‐dimensional imaging. Our model is suitable for object detection in ground‐penetrating radar images, thereby providing technical support for road damage and underground pipeline detection.

Environment Texture Optimization for Augmented Reality

Augmented reality (AR) platforms now support persistent, markerless experiences, in which virtual content appears in the same place relative to the real world, across multiple devices and sessions. However, optimizing environments for these experiences remains challenging; virtual content stability is determined by the performance of device pose tracking, which depends on recognizable environment features, but environment texture can impair human perception of virtual content. Low-contrast 'invisible textures' have recently been proposed as a solution, but may result in poor tracking performance when combined with dynamic device motion. Here, we examine the use of invisible textures in detail, starting with the first evaluation in a realistic AR scenario. We then consider scenarios with more dynamic device motion, and conduct extensive game engine-based experiments to develop a method for optimizing invisible textures. For texture optimization in real environments, we introduce MoMAR, the first system to analyze motion data from multiple AR users, which generates guidance using situated visualizations. We show that MoMAR can be deployed while maintaining an average frame rate > 59fps, for five different devices. We demonstrate the use of MoMAR in a realistic case study; our optimized environment texture allowed users to complete a task significantly faster (p=0.003) than a complex texture.

WBi-YOLOSF: improved feature pyramid network for aquatic real-time target detection under the artificial rabbits optimization

As the pillar industry of coastal areas, aquaculture needs artificial intelligence technology to promote economic development. To realize the automation of the aquaculture industry, this paper proposes a new underwater object detection network: WBi-YOLOSF. It realizes the automatic classification and detection of aquatic products, improves the production efficiency of the aquaculture industry, and promotes its economic development. This paper creates an image dataset containing 15 aquatic products to lay the data foundation for model training. In the data preprocessing part, an underwater image enhancement algorithm is proposed to improve the quality of the data set effectively. Aiming at the problem of high false detection rate and missed detection rate of underwater dense small targets, a new data enhancement method was proposed to improve the training set’s data quality comprehensively. Inspired by the weighted bidirectional feature pyramid network, this paper proposes a new feature extraction network: AU-BiFPN, which solves the gradient problem caused by the network hierarchy’s deepening on enhancing the network’s multi-scale feature fusion. The AU-BiFPN network structure is embedded into the YOLO series network framework, significantly improving the basic network’s feature extraction and feature fusion ability and dramatically improving the prediction accuracy without affecting the network inference speed. Here, the swarm intelligence algorithm is introduced to optimize the model hyperparameters, accelerating the convergence speed of model training and significantly reducing the computational cost. At the same time, the model’s accuracy is improved by a cliff. In addition, the Funnel Activation is introduced in the network’s backbone, and the simple, parameter-free attention module is integrated, effectively improving the accuracy and speed of the model prediction. Ablation and comparison experiments show the effectiveness and superiority of the proposed model. Verified by the mean average precision and frame rate evaluation indicators, the experimental results of the WBi-YOLOSF target detection network can reach 0.982 and 203 frames per second, which are 1.4% and five frames per second higher than the network with the second score. In summary, this method can quickly and accurately identify aquatic products, realize real-time target detection of aquatic products, and lay the foundation for developing an aquaculture automation system.

An efficient object detection by autonomous vehicle using deep learning

The automation industries have been developing since the first demonstration in the period 1980 to 2000 it is mainly used on automated driving vehicle. Now a day’s automotive companies, technology companies, government bodies, research institutions and academia, investors and venture capitalists are interested in autonomous vehicles. In this work, object detection on road is proposed, which uses deep learning (DL) algorithms. You only look once (YOLO V3, V4, V5). In this system object detection on the road data set is taken as input and the objects are mainly on-road vehicles, traffic signals, cars, trucks and buses. These inputs are given to the models to predict and detect the objects. The Performance of the proposed system is compared with performance of deep learning algorithms convolution neural network (CNN). The proposed system accuracy greater than 76.5% to 93.3%, mean average precision (Map) and frame per second (FPS) are 0.895 and 43.95%.

Design of image classification system for fabric inspection process using Raspberry Pi

This research is designed as a prototype of defect inspection system on fabric production using machine learning-based image processing technology using the open source Google teachable machine application integrated with Raspberry Pi-3B. The prototype of fabric defect inspection system is built by utilizing two rollers that function as a fabric roll house before and after the inspection process. On both rollers, a fabric is stretched to be inspected, so that from a roll of fabric with a certain length, it can be seen how many defects occur on the fabric. The inspection system is carried out using a web camera with a certain level of lighting connected to a raspberry pi as a control device. Raspberry Pi functions as an image processing device and fabric rolling motor controller. In addition to the category of fabric in good condition, this system classifies into two categories of defects, namely slap defects and sparse defects. The test results show that this system has an average frame per second (FPS) of 4.85, an average inference time of 181.1 ms, with an accuracy of image classification results of 98.4 %.

Fabric defect detection based on feature enhancement and complementary neighboring information

Abstract Fabric defect detection is a crucial aspect of quality control in the textile industry. Given the complexities of fabric backgrounds, the high similarity between patterned backgrounds and defects, and the variety of defect scales, we propose a fabric defect detection method based on feature enhancement and complementary neighboring information . The core of this method lies in two main components: the feature enhancement module and the neighboring information complementation strategy. The feature enhancement module includes two sub-modules: similarity feature enhancement(SFE) and edge detail feature enhancement(EDFE). The SFE aims to capture the similarities between features to strengthen the distinction between defects and complex backgrounds, thereby highlighting the correlations among defects and the differences between defects and the background. The EDFE focuses on improving the network's ability to capture the edge details of fabrics, preventing edge information from becoming blurred or lost due to deeper network layers. The neighboring information complementation strategy consists of shallow-level information complementation(SLIC) and top-down information fusion complementation(TDIFC). The SLIC integrates newly introduced shallow features with neighboring features that have a smaller semantic gap, injecting richer detail information into the network. The TDIFC adaptively guides the interaction of information between adjacent feature maps, effectively aggregating multi-scale features to ensure information complementarity between features of different scales. Additionally, to further optimize model performance, we introduced partial convolution(PConv) in the backbone of the feature extraction network. PConv reduces redundant computations and decreases the model's parameter count. Experimental results show that our proposed method achieved an mAP@50 of 82.4%, which is a 6.6% improvement over the baseline model YOLOv8s. The average inference frame rate reached 61.8 FPS, meeting the real-time detection requirements for fabric defects. Moreover, the model demonstrated good generalization capabilities, effectively adapting to detecting defects in different types and colors of fabrics.

Stereo visual-inertial localization algorithm for orchard robots based on point-line features

The vision-based localization of orchard mobile robots is an indispensable component of orchard intelligent management, applicable to scenarios such as navigation and harvesting. Due to the rugged terrain and lack of texture features in orchard environments, the accuracy of Visual-Inertial Odometry (VIO) based on feature point is compromised, making it challenging to meet practical application requirements. Addressing these issues, this paper proposes an algorithm for stereo visual-inertial localization based on point-line features, by enhancing VINS-Fusion. The algorithm incorporates the line feature detector algorithm (LSD) into the odometry front-end to detect line features, addressing the insufficient effective feature points and low-quality feature matching issues in orchard environments. Simultaneously, strategies for length filtering and feature matching are introduced for line features, eliminating poor-quality line features and matches caused by rugged terrain, thereby reducing adverse impacts on localization accuracy. To prevent keyframe redundancy due to a scarcity of stable feature points in areas with missing textures, such as tree trunks and leaves, a keyframe criteria optimization method based on point-line features is designed. Back-end optimization minimizes the cost function containing residuals of point-line features, prior information, and IMU, ultimately improving the localization accuracy of mobile robots. To validate the effectiveness and practicality of the proposed algorithm, a self-built dataset was created in real orchard environments, including scenes from banana plantation, mango orchard, and wampee orchard. Localization experiments were conducted using Rosario, EuRoC public datasets and self-built orchard dataset. Results indicate that the proposed algorithm consistently achieves lower ATE-RMSE than VINS-Fusion, with a maximum reduction of 79.2 % in the Rosario dataset sequence, 23.3 % in the EuRoC dataset sequence, and ATE-RMSE values of 0.091 m, 0.107 m, and 0.093 m in banana plantation, mango orchard, and wampee orchard, respectively. These represent reductions of 22.9 %, 7.6 %, and 27.3 % compared to VINS-Fusion. In the self-built orchard dataset, the average frame processing times for the algorithm front-end is 56.3 ms, while for the back-end is 31 ms, meeting the practical localization requirements. The proposed algorithm effectively enhances the localization accuracy of orchard mobile robots, offering a viable solution for autonomous navigation in orchard robot systems.

Real-Time Speed-of-Sound Estimation In Vivo via Steered Plane Wave Ultrasound.

Speed-of-sound (SoS) is an intrinsic acoustic property of human tissues and has been regarded as a potential biomarker of tissue health. To foster the clinical use of this emerging biomarker in medical diagnostics, it is important for SoS estimates to be derived and displayed in real time. Here, we demonstrate that concurrent global SoS estimation and B-mode imaging can be achieved live on a portable ultrasound scanner. Our innovation is hinged upon the design of a novel pulse-echo SoS estimation framework that is based on steered plane wave imaging. It has accounted for the effects of refraction and imaging depth when the medium SoS differs from the nominal value of 1540 m/s that is conventionally used in medical imaging. The accuracy of our SoS estimation framework was comparatively analyzed with through-transmit time-of-flight measurements in vitro on 15 custom agar phantoms with different SoS values (1508-1682 m/s) and in vivo on human calf muscles ( N = 9 ; SoS range: 1560-1586 m/s). Our SoS estimation framework has a mean signed difference (MSD) of - 0.6 ± 2.3 m/s in vitro and - 2.2 ± 11.2 m/s in vivo relative to the reference measurements. In addition, our real-time system prototype has yielded simultaneous SoS estimates and B-mode imaging at an average frame rate of 18.1 fps. Overall, by realizing real-time tissue SoS estimation with B-mode imaging, our innovation can foster the use of tissue SoS as a biomarker in medical ultrasound diagnostics.

Sistem Klasifikasi Citra untuk Proses Inspeksi Kain Menggunakan Teachable Machine dan Raspberry Pi

The textile industry has been growing rapidly, even the growth of textile products exceeds the growth of the national economy. The demand for textile products is not only for domestic consumption but also for export. In an effort to meet quality standards and maintain customer satisfaction, quality control of fabric production is very important, especially in controlling fabric production defects. The types of defects that exist in fabrics are holes, stains, rare defects due to broken/lost threads, floating, color fading, broken patterns, double threads, thick threads (slubs), mixed ends, pin marks, and others. In this research, a system is designed that can detect production defects in fabrics using machine learning-based image processing methods using Raspberry Pi. The types of defects modeled are sparse defects and stain defects, or in factory terms often called slap defects. The test results show that this system has an average frame per second (FPS) of 4.85, an average inference time of 181.1 ms, with an image classification result accuracy of 98.4%

Dual-branch information extraction and local attention anchor-free network for defect detection

In the production process, the presence of surface defects seriously affects the quality of industrial products. Existing defect detectors are not suitable for surface with scattered distribution and complex texture of defects. In this study, a dual-branch information extraction and local attention anchor-free network for defect detection (DLA-FCOS), which is based on the fully convolutional one-stage network, is proposed to accurately locate and detect surface defects of industrial products. Firstly, a dual-branch feature extraction network (DFENeT) is proposed and used to improve the extraction ability of complex defects. Then, a local feature enhancement module is proposed, and a residual connection is established to enrich local semantic information. Meanwhile, the self-attention mechanism is introduced to form local attentional residual feature pyramid networks (LA-RFPN) to eliminate the influences of feature misalignments. The mean average accuracy (mAP) and frames per second (FPS) of the proposed DLA-FCOS on the cut layer of the tobacco packet defect dataset (CLTP-DD) are 96.8% and 20.7, respectively, which meets the requirements for accurate and real-time defect detection. Meanwhile, the average accuracy of the proposed DLA-FCOS on the NEU-DET and GC10-DET datasets is 78.4% and 67.7%, respectively. The results demonstrate that the DLA-FCOS has good feasibility and high generalization capability to perform defect detection tasks of industrial products.

Efficient Task-driven Video Data Privacy Protection for Smart Camera Surveillance System

As one of the most commonly used AIoT sensors, smart cameras and their supporting services, namely cloud video surveillance (CVS) systems, have brought great convenience to people’s lives. Recent CVS providers use different machine learning techniques to improve their services (regarded as tasks) based on the uploaded video. However, uploading data to the CVS providers may cause severe privacy issues. Existing works that remove privacy information could not achieve a high tradeoff between data usability and privacy, because the importance of information varies with the task. In addition, it is challenging to design a real-time privacy protection mechanism, especially in resource-constrained smart cameras. In this work, we design a task-driven and efficient video privacy protection mechanism for a better tradeoff between privacy and data usability. We use Class Activation Mapping to protect privacy while preserving data usability. To improve the efficiency, we utilize the motion vector and residual matrix produced during video codec. Our work outperforms the region of interest–based methods in data protection while preserving data usability. The attack accuracy drops 70%, while the task accuracy is comparable to those without protection (within ± 4%). The average protection frame rate of the High Definition video can exceed 16 fps+ even on a CPU.

Video anomaly detection guided by clustering learning

With the fuzzy boundary between normal and abnormal video data, which cannot be well distinguished by most methods, anomaly detection in video requires better characterization of the data. First we give a convolution-enhanced self-attentive video auto-encoder based on the U-Net architecture, which can extract richer image features. Secondly we design a dual-scale feature clustering structure for this encoder, which simultaneously compresses the channel and spatial structure features of the image to represent the features to obtain good coding characteristics and expand the boundary between normal and abnormal data. We also verify that our approach is equivalent to a class of auto-encoders for memory-guided learning. Finally, in the reconstruction task, since video auto-encoders are capable of triggering temporal time leakage phenomena that can lead to network performance degradation, we propose an anomaly score computation paradigm for video auto-encoders that utilizes the average frame anomaly score of a video clip to compute the first frame anomaly score in that video clip.Extensive experiments on three benchmark datasets show that our method outperforms most existing methods on large datasets with complex patterns. The code will be published at the following link: Anomaly-detection-guided-by-clustering-learning

A Multi-Element Identification System Based on Deep Learning for the Visual Field of Percutaneous Endoscopic Spine Surgery.

Lumbar disc herniation is a common degenerative lumbar disease with an increasing incidence. Percutaneous endoscopic lumbar discectomy can treat lumbar disc herniation safely and effectively with a minimally invasive procedure. However, the learning curve of this technology is steep, which means that initial learners are often not sufficiently proficient in endoscopic operations, which can easily lead to iatrogenic damage. At present, the application of computer deep learning technology to clinical diagnosis, treatment, and surgical navigation has achieved satisfactory results. The objective of our team is to develop a multi-element identification system for the visual field of endoscopic spine surgery using deep learning algorithms and to evaluate the feasibility of this system. We established an image database by collecting surgical videos of 48 patients diagnosed with lumbar disc herniation, which was labeled by two spinal surgeons. We selected 6000 images of the visual field of percutaneous endoscopic spine surgery (including various tissue structures and surgical instruments), divided into the training data, validation data, and test data according to 2:1:2. We developed convolutional neural network models based on instance segmentation-Solov2, CondInst, Mask R-CNN and Yolact, and set the four network model backbone as ResNet101 and ResNet50 respectively. Mean average precision (mAP) and frames per second (FPS) were used to measure the performance of each model for classification, localization and recognition in real time, and AP (average) is used to evaluate how easily an element is detected by neural networks based on computer deep learning. Comprehensively comparing mAP and FSP of each model for bounding box test and segmentation task for the test set of images, we found that Solov2 (ResNet101) (mAP = 73.5%, FPS = 28.9), Mask R-CNN (ResNet101) (mAP = 72.8%, FPS = 28.5) models are the most stable, with higher precision and faster image processing speed. Combining the average precision of the elements in the bounding box test and segmentation tasks in each network, the AP(average) was highest for tool 3 (bbox-0.85, segm-0.89) and lowest for tool 5 (bbox-0.63, segm-0.72) in the instrumentation, whereas in the anatomical tissue elements, the fibrosus annulus (bbox-0.68, segm-0.69) and ligamentum flavum (bbox-0.65, segm-0.62) had higher AP(average),while extra-dural fat (bbox-0.42, segm-0.44) was lowest. Our team has developed a multi-element identification system for the visual field of percutaneous endoscopic spine surgery adapted to the interlaminar and foraminal approaches, which can identify and track anatomical tissue (nerve, ligamentum flavum, nucleus pulposus, etc.) and surgical instruments (endoscopic forceps, an high-speed diamond burr, etc.), which can be used in the future as a virtual educational tool or applied to the intraoperative real-time assistance system for spinal endoscopic operation.