Detection Of Surgical Instruments Research Articles

DBH-YOLO: a surgical instrument detection method based on feature separation in laparoscopic surgery.

Accurately locating and analysing surgical instruments in laparoscopic surgical videos can assist doctors in postoperative quality assessment. This can provide patients with more scientific and rational solutions for healing surgical complications. Therefore, we propose an end-to-end algorithm for the detection of surgical instruments. Dual-Branched Head (DBH) and Overall Intersection over Union Loss (OIoU Loss) are introduced to solve the problem of inaccurate surgical instrument detection, both in terms of localization and classification. An effective method (DBHYOLO) for the detection for laparoscopic surgery in complex scenarios is proposed. This study manually annotates a new laparoscopic gastric cancer resection surgical instrument location dataset LGIL, which provides a better validation platform for surgical instrument detection methods. The proposed method's performance was tested using the m2cai16-tool-locations, LGIL, and Onyeogulu datasets. The mean Average Precision (mAP) values obtained were 96.8%, 95.6%, and 98.4%, respectively, which were higher than the other classical models compared. The improved model is more effective than the benchmark network in distinguishing between surgical instrument classes with high similarity and avoiding too many missed detection cases. In this paper, the problem of inaccurate detection of surgical instruments is addressed from two different perspectives: classification and localization. And the experimental results on three representative datasets verify the performance of DBH-YOLO. It is shown that this method has a good generalization capability.

A Deep Learning Approach for Surgical Instruments Detection System in Total Knee Arthroplasty—Automatic Creation of Training Data and Reduction of Training Time—

In total knee arthroplasty (TKA), many surgical instruments are available. Many of these surgical instruments are similar in shape and size. For this reason, there have been accidents due to incorrect selection of implants. Furthermore, a shortage of nurses is expected worldwide. There will also be a shortage of scrub nurses, which will result in an increased burden on each scrub nurse. For these reasons, we have developed a surgical instrument detection system for TKA to reduce the burden on scrub nurses and the number of accidents, such as implant selection errors. This study also focuses on automating the acquisition of data for training. We also develop a method to reduce the additional training time when the number of detection targets increases. In this study, YOLOv5 is used as the object detection method. In experiments, we examine the accuracy of the training data automatically acquired and the accuracy of object detection for surgical instruments. In object detection, several training files are created and compared. The results show that the training data is sufficiently effective, and high accuracy is obtained in object detection. Object detection is performed in several cases, and one of the results shows an IoU of 0.865 and an F-measure of 0.930.

DIPO: Differentiable Parallel Operation Blocks for Surgical Neural Architecture Search.

Deep learning has been used across a large number of computer vision tasks, however designing the network architectures for each task is time consuming. Neural Architecture Search (NAS) promises to automatically build neural networks, optimised for the given task and dataset. However, most NAS methods are constrained to a specific macro-architecture design which makes it hard to apply to different tasks (classification, detection, segmentation). Following the work in Differentiable NAS (DNAS), we present a simple and efficient NAS method, Differentiable Parallel Operation (DIPO), that constructs a local search space in the form of a DIPO block, and can easily be applied to any convolutional network by injecting it in-place of the convolutions. The DIPO block's internal architecture and parameters are automatically optimised end-to-end for each task. We demonstrate the flexibility of our approach by applying DIPO to 4 model architectures (U-Net, HRNET, KAPAO and YOLOX) across different surgical tasks (surgical scene segmentation, surgical instrument detection, and surgical instrument pose estimation) and evaluated across 5 datasets. Results show significant improvements in surgical scene segmentation (+10.5% in CholecSeg8K, +13.2% in CaDIS), instrument detection (+1.5% in ROBUST-MIS, +5.3% in RoboKP), and instrument pose estimation (+9.8% in RoboKP).

SINet: A hybrid deep CNN model for real-time detection and segmentation of surgical instruments

ObjectiveDetection and segmentation of surgical instruments is an indispensable technology in robot-assisted surgery that enables doctors to obtain more comprehensive visual information and further improve the safety of surgery. However, the results of a detection are more easily interfered by environmental factors, such as instrument shaking, incomplete displays and insufficient light. To overcome those issues, we designed a hybrid deep-CNN model (SINet) for real-time surgical instrument detection and segmentation. MethodsThe framework employs YOLOv5 as the object detection model and introduces a GAM attention mechanism to improve its feature extraction abilities. During training, the SiLU activation function is adopted to avoid gradient explosions and unstable training situations. Specifically, the vector angle relationship between the ground truth boxes and the prediction boxes was applied in the SIoU loss function to reduce the degree of freedom of the regression and accelerate the network convergence. Finally, a semantic segmentation head is used to implement detections of the surgical instruments by paralleling the detection and segmentation. ResultsThe proposed method is evaluated on the m2cai16-tool-locations public dataset and achieved a significant 97.9% mean average precision (mAP), 133 frames per second (FPS), 85.7% mean intersection over union (MIoU) and 86.6% Dice. Experiment based on simulated surgery platform also shows satisfactory detection performance. ConclusionExperimental results demonstrated that the SINet can effectively detect the pose of surgical instruments and achieves a better performance than most of the current algorithms. The method has the potential to help perform a series of surgical operations efficiently and safely.

ToolNet-X: Surgical Instrument Detection Combined With High-Order Spatial Interaction

ToolNet-X: Surgical Instrument Detection Combined With High-Order Spatial Interaction

Improving instrument detection for a robotic scrub nurse using multi-view voting

PurposeA basic task of a robotic scrub nurse is surgical instrument detection. Deep learning techniques could potentially address this task; nevertheless, their performance is subject to some degree of error, which could render them unsuitable for real-world applications. In this work, we aim to demonstrate how the combination of a trained instrument detector with an instance-based voting scheme that considers several frames and viewpoints is enough to guarantee a strong improvement in the instrument detection task.MethodsWe exploit the typical setup of a robotic scrub nurse to collect RGB data and point clouds from different viewpoints. Using trained Mask R-CNN models, we obtain predictions from each view. We propose a multi-view voting scheme based on predicted instances that combines the gathered data and predictions to produce a reliable map of the location of the instruments in the scene.ResultsOur approach reduces the number of errors by more than 82% compared with the single-view case. On average, the data from five viewpoints are sufficient to infer the correct instrument arrangement with our best model.ConclusionOur approach can drastically improve an instrument detector’s performance. Our method is practical and can be applied during an actual medical procedure without negatively affecting the surgical workflow. Our implementation and data are made available for the scientific community (https://github.com/Jorebs/Multi-view-Voting-Scheme).

Surgical Instrument Detection Algorithm Based on Improved YOLOv7x.

The counting of surgical instruments is an important task to ensure surgical safety and patient health. However, due to the uncertainty of manual operations, there is a risk of missing or miscounting instruments. Applying computer vision technology to the instrument counting process can not only improve efficiency, but also reduce medical disputes and promote the development of medical informatization. However, during the counting process, surgical instruments may be densely arranged or obstruct each other, and they may be affected by different lighting environments, all of which can affect the accuracy of instrument recognition. In addition, similar instruments may have only minor differences in appearance and shape, which increases the difficulty of identification. To address these issues, this paper improves the YOLOv7x object detection algorithm and applies it to the surgical instrument detection task. First, the RepLK Block module is introduced into the YOLOv7x backbone network, which can increase the effective receptive field and guide the network to learn more shape features. Second, the ODConv structure is introduced into the neck module of the network, which can significantly enhance the feature extraction ability of the basic convolution operation of the CNN and capture more rich contextual information. At the same time, we created the OSI26 data set, which contains 452 images and 26 surgical instruments, for model training and evaluation. The experimental results show that our improved algorithm exhibits higher accuracy and robustness in surgical instrument detection tasks, with F1, AP, AP50, and AP75 reaching 94.7%, 91.5%, 99.1%, and 98.2%, respectively, which are 4.6%, 3.1%, 3.6%, and 3.9% higher than the baseline. Compared to other mainstream object detection algorithms, our method has significant advantages. These results demonstrate that our method can more accurately identify surgical instruments, thereby improving surgical safety and patient health.

Detection of surgical instruments based on Gaussian kernel

In minimally invasive laparoscopic surgery, it is of practical significance to quickly locate the location and category information of the surgical instrument. It can remind medical personnel of irreversible injury caused to patients due to leaving surgical instruments after the operation. In this paper, the Gaussian kernel is introduced into each ground truth, which is conducive to making full use of label information to allocate positive and negative samples and improve the accuracy of location and classification. Then, we introduce SIoU Loss and Harmonic Loss function into a total loss. The former uses relative coordinates to make the network converge more quickly, and the latter solves the problem of asynchronous optimization of the two branches of classification and regression. Our experiment proves that the strategy based on Gaussian kernel sample allocation is very effective on a pubic data set m2cai16-tool-locations, displaying our method possesses conspicuous accuracy of classification and regression than other work.

Anchor-Free Feature Aggregation Network for Instrument Detection in Endoscopic Surgery

Endoscopic endonasal approach has been widely used for removing various sellae tumors including pituitary adenomas, meningiomas, etc. While, performing these surgeries in such a narrow space with different instruments remains a challenge for surgeons, due to the limited field of view, varying illumination, and occlusion of instruments during the operation. Thus, a proper surgical instrument detection method that can provide classification and location information of the operated surgical instrument is critical for surgeons to understand the surgical scenarios and enhance the safety of the clinical operation. To this end, we propose an anchor-free feature aggregation network (AFA-Net) to improve the detection precision of surgical instruments from the endoscopic operation view field. The proposed method utilizes the improved feature pyramid network (FPN) with the depthwise separable convolution and a weighted feature aggregation module to enhance the feature information of the operated surgical instruments. Based on the anchor-free method, a weighted heatmap aggregation module is used to detect surgical instruments. Experimental studies on a public dataset Cholec80 and an intraoperative dataset from a local hospital are conducted, and the detection performance is assessed by the mean precision (AP) and average recall (AR). From both datasets and comparisons, the proposed method achieves 74.1% AP, 67.0% AR and 73.6% AP, 66.7% AR, respectively, which show significant advantages over five mainstream methods in terms of detection performance.

Research on laparoscopic surgical instrument detection technology based on multi-attention-enhanced feature pyramid network

Laparoscopic surgery is a very active area of research in clinical medicine. Detection of tools in surgical videos can help physicians operate surgical instruments, reduce complications, and ensure patient safety. However, the size of laparoscopic surgical instruments is highly variable, leading to poor detection. Feature pyramid networks (FPNs) can effectively solve the problem of multi-scale target detection, but FPNs still have some problems that limit the full utilization of multi-scale features. By analyzing the FPN design problem, we propose the Multi-Attention Augmented Feature Pyramid Network (MAFPN), which can fully utilize the multi-scale features. First, we replace the convolutional block with feature selection module (FSM) that combines channel attention and global attention, which selectively maintains important information and enhances the expressiveness of features at each scale. Second, the global contextual information is captured by the self-attentive augmented fusion module (AAFM), which enriches the high-level feature information in the FPN and enhances the feature fusion effect. Finally, we use Dynamic Convolution Decomposition (DCD) to alleviate the impact of upsampling while enhancing the feature expression ability. The experimental results on the laparoscopic surgical instrument detection dataset m2cai16-tool-locations show that the average precision of MAFPN is 96.5 when the IOU is 0.5, which is 1.8% better than the baseline method RetinaNet, and the average precision is more than 1.6% better than the comparison network. Compared with the state-of-the-art method, the performance of laparoscopic surgical instrument detection is superior.

Surgical instrument detection and tracking technologies: Automating dataset labeling for surgical skill assessment.

Surgical skills can be improved by continuous surgical training and feedback, thus reducing adverse outcomes while performing an intervention. With the advent of new technologies, researchers now have the tools to analyze surgical instrument motion to differentiate surgeons' levels of technical skill. Surgical skills assessment is time-consuming and prone to subjective interpretation. The surgical instrument detection and tracking algorithm analyzes the image captured by the surgical robotic endoscope and extracts the movement and orientation information of a surgical instrument to provide surgical navigation. This information can be used to label raw surgical video datasets that are used to form an action space for surgical skill analysis. Instrument detection and tracking is a challenging problem in MIS, including robot-assisted surgeries, but vision-based approaches provide promising solutions with minimal hardware integration requirements. This study offers an overview of the developments of assessment systems for surgical intervention analysis. The purpose of this study is to identify the research gap and make a leap in developing technology to automate the incorporation of new surgical skills. A prime factor in automating the learning is to create datasets with minimal manual intervention from raw surgical videos. This review encapsulates the current trends in artificial intelligence (AI) based visual detection and tracking technologies for surgical instruments and their application for surgical skill assessment.

Deep Learning for Instrument Detection and Assessment of Operative Skill in Surgical Videos

Surgical performance has been shown to be directly related to patient outcomes. There is significant variation in surgical performance and therefore a need to measure operative skill accurately and reliably. Despite this, current means of surgical performance assessment rely on expert observation which is labor-intensive, prone to rater bias and unreliable. We present an automatic approach to surgical performance assessment through the tracking of instruments in endoscopic video. We annotate the spatial bounds of surgical instruments in 2600 images and use this new dataset to train Mask R-CNN, a state-of-the-art instance segmentation framework. We show that we can successfully achieve spatial detection of surgical instruments by generating a pixel-by-pixel mask over the detected instrument and achieving an overall mAP of 0.839 for an IoU of 0.5. We leverage the results from our instrument detection framework to assess surgical performance through the generation of instrument trajectory maps and instrument metrics such as moving distance, smoothness of instrument movement and concentration of instrument movement.

Deep-learning-based instrument detection for intra-operative robotic assistance

Purpose:Robotic scrub nurses have the potential to become an attractive solution for the operating room. Surgical instrument detection is a fundamental task for these systems, which is the focus of this work. We address the detection of the complete surgery set for wisdom teeth extraction, and propose a data augmentation technique tailored for this task.Methods:Using a robotic scrub nurse system, we create a dataset of 369 unique multi-instrument images with manual annotations. We then propose the Mask-Based Object Insertion method, capable of automatically generating a large amount of synthetic images. By using both real and artificial data, different Mask R-CNN models are trained and evaluated.Results:Our experiments reveal that models trained on the synthetic data created with our method achieve comparable performance to that of models trained on real images. Moreover, we demonstrate that the combination of real and our artificial data can lead to a superior level of generalization.Conclusion:The proposed data augmentation technique is capable of dramatically reducing the labelling work required for training a deep-learning-based detection algorithm. A dataset for the complete instrument set for wisdom teeth extraction is made available for the scientific community, as well as the raw information required for the generation of the synthetic data (https://github.com/Jorebs/Deep-learning-based-instrument-detection-for-intra operative-robotic-assistance).

Visual detection and tracking algorithms for minimally invasive surgical instruments: A comprehensive review of the state-of-the-art

Minimally invasive surgical instrument visual detection and tracking is one of the core algorithms of minimally invasive surgical robots. With the development of machine vision and robotics, related technologies such as virtual reality, three-dimensional reconstruction, path planning, and human–machine collaboration can be applied to surgical operations to assist clinicians or use surgical robots to complete clinical operations. The minimally invasive surgical instrument vision detection and tracking algorithm analyzes the image transmitted by the surgical robot endoscope, extracting the position of the surgical instrument tip in the image, so as to provide the surgical navigation. This technology can greatly improve the accuracy and success rate of surgical operations. The purpose of this paper is to further study the visual detection and tracking technology of minimally invasive surgical instruments, summarize the existing research results, and apply it to the surgical robot project. By reading the literature, the author summarized the theoretical basis and related algorithms of this technology in recent years. Finally, the author compares the accuracy, speed and application scenario of each algorithm, and analyzes the advantages and disadvantages of each algorithm. The papers included in the review were selected through Web of Science, Google Scholar, PubMed and CNKI searches using the keywords: “object detection”, “object tracking”, “surgical tool detection”, “surgical tool tracking”, “surgical instrument detection” and “surgical instrument tracking” limiting results to the year range 1985–2021. Our study shows that this technology will have a great development prospect in the aspects of accuracy and real-time improvement in the future.

Automated detection of electrocautery instrument in videos of open neck procedures using YOLOv3.

With the rapid development of deep learning approaches, tremendous progress has been made in computer- assisted analysis of minimally-invasive, videoscopic surgery. However, surgery through open incisions ("open surgery"), which constitutes a much larger portion of surgical procedures performed, is rarely investigated because of the difficulty in obtaining high-quality open surgical video footage. Automated detection of surgical instruments shows promise for evaluating surgical activities, and provides a foundation for quality/safety review, education, and identification of surgical performance. In this paper, we present results using YOLOv3 to successfully identify an electrocautery surgical instrument in a library of images derived from 22 open neck procedures (an 887-image training/validation set, and a 1149-image testing set) captured using a wearable surgical camera. We show that our method effectively detects the spatial bounds of the electrocautery pencil in still images and we further demonstrate the ability of our method to detect the location of this instrument in video footage. Our work serves as the first demonstration of open surgical instrument detection using first-person video footage from a wearable camera and sets the stage for further work in this field.Clinical Relevance- Detection of instrumentation in surgical video is the necessary first step towards automating surgical task identification and skills assessment, which will be useful for surgical quality improvement and training.

Automatic tip detection of surgical instruments in biportal endoscopic spine surgery

BackgroundRecent advances in robotics and deep learning can be used in endoscopic surgeries and can provide numerous advantages by freeing one of the surgeon's hands. This study aims to automatically detect the tip of the instrument, localize a point, and evaluate the detection accuracy in biportal endoscopic spine surgery (BESS). The tip detection could serve as a preliminary study for the development of vision intelligence in robotic endoscopy. MethodsThe dataset contains 2310 frames from 9 BESS videos with x and y coordinates of the tip annotated by an expert. We trained two state-of-the-art detectors, RetinaNet and YOLOv2, with bounding boxes centered around the tip annotations with specific margin sizes to determine the optimal margin size for detecting the tip of the instrument and localizing the point. We calculated the recall, precision, and F1-score with a fixed box size for both ground truth tip coordinates and predicted midpoints to compare the performance of the models trained with different margin size bounding boxes. ResultsFor RetinaNet, a margin size of 150 pixels was optimal with a recall of 1.000, precision of 0.733, and F1-score of 0.846. For YOLOv2, a margin size of 150 pixels was optimal with a recall of 0.864, precision of 0.808, F1-score of 0.835. Also, the optimal margin size of 150 pixels of RetinaNet was used to cross-validate its overall robustness. The resulting mean recall, precision, and F1-score were 1.000 ± 0.000, 0.767 ± 0.033, and 0.868 ± 0.022, respectively. ConclusionsIn this study, we evaluated an automatic tip detection method for surgical instruments in endoscopic surgery, compared two state-of-the-art detection algorithms, RetinaNet and YOLOv2, and validated the robustness with cross-validation. This method can be applied in different types of endoscopy tip detection.

Lightweight Deep Neural Network for Real-Time Instrument Semantic Segmentation in Robot Assisted Minimally Invasive Surgery

Vision-based detection and tracking of surgical instrument is attractive because it relies purely on existing setup already in the operating scenario for Robot assisted minimally invasive surgery (RMIS). While significant advanced approaches have been made in recent years, how to effectively carry out the efficient algorithm in the abdominal environment for real-time surgical instrument semantic segmentation is still a fundamental challenging topic. For this purpose, this letter reports the development of a surgical instrument semantic segmentation method for RMIS scenario. We have demonstrated a novel lightweight deep neural network to exploit surgical instrument semantic segmentation that meets real-time, accuracy and robustness. Specifically, our model contains an innovative design of two stages lightweight network by integrating the Ghost modules with MobileNetV3. Given that reducing computation cost with holding high accuracy is crucial for the task, in the first stage, Ghost module is explored to model the bottleneck for extracting the semantic features. We also design the lightweight Segmentation head as the second stage to further reduce computation cost, the model outputs a pixel-by-pixel mask to represent the surgical instrument parts semantic segmentation results. Promising validation has been conducted to evaluate the proposed lightweight network, the results report that the method can exhibit the outstanding surgical instrument semantic segmentation and surpass existing algorithms in the real-time performance with holding the high accuracy. The letter demonstrates the feasibility to perform real-time surgical instrument detection by taking advantage of the components of surgical robot system, contributing to the reference for further surgical intelligence.

MASSD: Multi-scale attention single shot detector for surgical instruments

Surgical instrument detection is a significant task in computer-aided minimal invasive surgery for providing real-time feedback to physicians, evaluating surgical skills, and developing a training plan for surgeons. In this study, a multi-scale attention single detector is designed for surgical instruments. In the field of object detection, accurate detection of small objects is always a challenging task. We propose an innovative feature fusion technique aimed at small surgical instrument detection. First, the attention map is created from high-level features to act on the low-level features and enrich the semantic information of the low-level features. The original and processed features are then fused by skip connection. Finally, multi-scale feature maps are created to predict fusion features. The experiments on the ATLAS Dione dataset yielded results with a detection time of 0.066 s per frame and a mean average precision of 90.08%. Our proposed feature fusion module can obtain more semantic information for low-level features and significantly enhance the performance of small surgical instrument detection.

Convolutional neural network-based surgical instrument detection.

BACKGROUND: Minimally invasive surgery (MIS), unlike open surgery in which surgeons can perform surgery directly, is performed using miniaturized instruments with indirect but careful observation of the surgical site.OBJECTIVE: Instrument detection is a crucial requirement in conventional and robot-assisted MIS, which can also be very useful during surgical training. In this paper, we propose a novel framework of using two three-layer convolutional neural networks (CNNs) in a series to detect surgical instrument in in-vivo video frames. METHODS: The two convolutional neural networks proposed in this paper have different tasks. (i) The former CNN is trained to detect the edges points of the instrument shaft directly from images patches. (ii) The latter is trained to locate the instrument tip also from images patches after the former detection finishes.RESULTS: We validated our method on the publicly available EndoVisSub dataset and a standard dataset, and it detected tools with an accuracy of 91.2% and 75% respectively.CONCLUSION: Our two-step detection method achieves better performance than other existing approaches in terms of detection accuracy.

Automated Surgical Instrument Detection from Laparoscopic Gastrectomy Video Images Using an Open Source Convolutional Neural Network Platform

The common use of laparoscopic intervention produces impressive amounts of video data that are difficult to review for surgeons wishing to evaluate and improve their skills. Therefore, a need exists for the development of computer-based analysis of laparoscopic video to accelerate surgical training and assessment. We developed a surgical instrument detection system for video recordings of laparoscopic gastrectomy procedures. This system, the use of which might increase the efficiency of the video reviewing process, is based on the open source neural network platform, YOLOv3. A total of 10,716 images extracted from 52 laparoscopic gastrectomy videos were included in the training and validation data sets. We performed 200,000 iterations of training. Video recordings of 10 laparoscopic gastrectomies, independent of the training and validation data set, were analyzed by our system, and heat maps visualizing trends of surgical instrument usage were drawn. Three skilled surgeons evaluated whether each heat map represented the features of the corresponding operation. After training, the testing data set precision and sensitivity (recall) was 0.87 and 0.83, respectively. The heat maps perfectly represented the devices used during each operation. Without reviewing the video recordings, the surgeons accurately recognized the type of anastomosis, time taken to initiate duodenal and gastric dissection, and whether any irregular procedure was performed, from the heat maps (correct answer rates ≥ 90%). A new automated system to detect manipulation of surgical instruments in video recordings of laparoscopic gastrectomies based on the open source neural network platform, YOLOv3, was developed and validated successfully.