Intelligent Operating Room Research Articles

Ontology-based surgical workflow recognition and prediction

BackgroundSurgical context-aware systems can adapt to the current situation in the operating room and thus provide computer-aided assistance functionalities and intraoperative decision-support. To interact with the surgical team perceptively and assist the surgical process, the system needs to monitor the intraoperative activities, understand the current situation in the operating room at any time, and anticipate the following possible situations. MethodsA structured representation of surgical process knowledge is a prerequisite for any applications in the intelligent operating room. For this purpose, a surgical process ontology, which is formally based on standard medical terminology (SNOMED CT) and an upper-level ontology (GFO), was developed and instantiated for a neurosurgical use case. A new ontology-based surgical workflow recognition and a novel prediction method are presented utilizing ontological reasoning, abstraction, and explication. This way, a surgical situation representation with combined phase, high-level task, and low-level task recognition and prediction was realized based on the currently used instrument as the only input information. ResultsThe ontology-based approach performed efficiently, and decent accuracy was achieved for situation recognition and prediction. Especially during situation recognition, the missing sensor information were reasoned based on the situation representation provided by the process ontology, which resulted in improved recognition results compared to the state-of-the-art. ConclusionsIn this work, a reference ontology was developed, which provides workflow support and a knowledge base for further applications in the intelligent operating room, for instance, context-aware medical device orchestration, (semi-) automatic documentation, and surgical simulation, education, and training.

The Latest Glioma Surgical Support Devices and Intraoperative Applications of MRI

We previously proposed an information-guided surgery based on objective intraoperative information to achieve a high precision and maximum resection of gliomas. In 2000, an "intelligent operating room" was constructed and put into operation as a place to train new "eyes," "hands," and "brains" for surgeons. Based on the positive experience from this room, it was subsequently developed into the Smart Cyber Operating Theater (SCOT), and from October 2019, Hyper SCOT was introduced at Tokyo Women's Medical University Hospital. This report introduces the latest surgical support devices that we have implemented in glioma surgery with Hyper SCOT. As an intraoperative image information support device, intraoperative MRI can correct brain shift, visualize residual tumor, and aid in performing maximal tumor resection, thereby greatly contributing to surgical results. As an intraoperative histological information support device, intraoperative flow cytometry can evaluate the presence of tumor cells and malignancy based on the measurement of the amount of DNA in tumor cells, and also predict the prognosis intraoperatively. Photodynamic therapy has been shown to be effective in clinical trials as an intraoperative treatment support device, and it is covered by insurance. Therefore, it is expected to be established as a standard treatment method for primary glioblastoma patients.

Intraoperative surgery room management: A deep learning perspective.

The current study aimed to systematically review the literature addressing the use of deep learning (DL) methods in intraoperative surgery applications, focusing on the data collection, the objectives of these tools and, more technically, the DL-based paradigms utilized. A literature search with classic databases was performed: we identified, with the use of specific keywords, a total of 996 papers. Among them, we selected 52 for effective analysis, focusing on articles published after January 2015. The preliminary results of the implementation of DL in clinical setting are encouraging. Almost all the surgery sub-fields have seen the advent of artificial intelligence (AI) applications and the results outperformed the previous techniques in the majority of the cases. From these results, a conceptualization of an intelligent operating room (IOR) is also presented. This evaluation outlined how AI and, in particular, DL are revolutionizing the surgery field, with numerous applications, such as context detection and room management. This process is evolving years by years into the realization of an IOR, equipped with technologies perfectly suited to drastically improve the surgical workflow.

Real-Time Streaming of Surgery Performance and Intraoperative Imaging Data in the Hybrid Operating Room: Development and Usability Study.

BackgroundThe trend of quick evolution and increased digital data in today’s operating rooms (ORs) has led to the construction of hybrid ORs. There is often a main control room with monitors for integrating intraoperative data from multiple devices in the hybrid OR. However, there is no adequate solution for communicating the data with people outside the OR.ObjectiveThe objective of this study was to design an intelligent operating room (iOR) system, augmented onto the existing information technology (IT) infrastructure of hybrid ORs, to stream surgery performance and intraoperative imaging data.MethodsIn this study, an all-in-one device with synergetic encoder and decoder was used. The device was able to stream multiple sources to one display. The lossless video and images from specific surgical workflows were streamed outside the hybrid OR through network protocols and were further managed by a streaming server and wireless control system. The steps of this study included the following: (1) defining the requirements and feasibility of an iOR system in the hybrid OR, (2) connecting multiple sources, (3) setting up equipment across the hybrid OR and a conference room, (4) designing a video management system, and (5) real-time streaming under specific surgical workflows.ResultsThe wired streamed video was shown simultaneously on the display in the hybrid OR and the display in the conference room with near-zero latency. Additionally, an interactive video between the hybrid OR and the conference room was achieved through the bidirectional wireless control system. The functions of recording, archiving, and playback were successfully provided by the streaming server. The readily available hardware components and open-access programming reduced the cost required to construct this streaming system.ConclusionsThis flexible and cost-effective iOR system not only provided educational benefits, but also contributed to surgical telementoring.

Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos.

Real-time surgical tool tracking is a core component of the future intelligent operating room (OR), because it is highly instrumental to analyze and understand the surgical activities. Current methods for surgical tool tracking in videos need to be trained on data in which the spatial positions of the tools are manually annotated. Generating such training data is difficult and time-consuming. Instead, we propose to use solely binary presence annotations to train a tool tracker for laparoscopic videos. The proposed approach is composed of a CNN + Convolutional LSTM (ConvLSTM) neural network trained end to end, but weakly supervised on tool binary presence labels only. We use the ConvLSTM to model the temporal dependencies in the motion of the surgical tools and leverage its spatiotemporal ability to smooth the class peak activations in the localization heat maps (Lh-maps). We build a baseline tracker on top of the CNN model and demonstrate that our approach based on the ConvLSTM outperforms the baseline in tool presence detection, spatial localization, and motion tracking by over [Formula: see text], [Formula: see text], and [Formula: see text], respectively. In this paper, we demonstrate that binary presence labels are sufficient for training a deep learning tracking model using our proposed method. We also show that the ConvLSTM can leverage the spatiotemporal coherence of consecutive image frames across a surgical video to improve tool presence detection, spatial localization, and motion tracking.

Semi-supervised spatio-temporal CNN for recognition of surgical workflow

Robust and automated surgical workflow detection in real time is a core component of the future intelligent operating room. Based on this technology, it can help medical staff to automate and intelligently complete many routine activities during surgery. Recognition of surgical workflow based on traditional pattern recognition methods requires a large number of labeled surgical video data. However, the labeled surgical video data requires expert knowledge and it is difficult and time consuming to collect a sufficient number of labeled surgical video data in the medical field. Therefore, this paper proposes a semi-supervised spatio-temporal convolutional network for the recognition of surgical workflow based on convolutional neural networks and temporal-recursive networks. Firstly, we build a spatial convolutional extraction feature network based on unsupervised generative adversarial learning. Then, we build a bridge between low-level surgical video features and high-level surgical workflow semantics based on an unsupervised temporal-ordered network learning approach. Finally, we use the semi-supervised learning method to integrate the spatial model and the temporal model to fine-tune the network, and realize the intelligent recognition of the surgical workflow at a low cost to efficiently determine the progress of the surgical workflow. We performed some experiments for validating the mode based on m2cai16-workflow dataset. It shows that the proposed model can effectively extract the surgical feature and determine the surgical workflow. The Jaccard score of the model reaches 71.3%, and the accuracy of the model reaches 85.8%.

Intelligent operating room suite : From passive medical devices to the self-thinking cognitive surgical assistant

Modern operating room (OR) suites are mostly digitally connected but until now the primary focus was on the presentation, transfer and distribution of images. Device information and processes within the operating theaters are barely considered. Cognitive assistance systems have triggered afundamental rethinking in the automotive industry as well as in logistics. In principle, tasks in the OR, some of which are highly repetitive, also have great potential to be supported by automated cognitive assistance via a self-thinking system. This includes the coordination of the entire workflow in the perioperative process in both the operating theater and the whole hospital. With corresponding data from hospital information systems, medical devices and appropriate models of the surgical process, intelligent systems could optimize the workflow in the operating theater in the near future and support the surgeon. Preliminary results on the use of device information and automatically controlled OR suites are already available. Such systems include, for example the guidance of laparoscopic camera systems. Nevertheless, cognitive assistance systems that make use of knowledge about patients, processes and other pieces of information to improve surgical treatment are not yet available in the clinical routine but are urgently needed in order to automatically assist the surgeon in situation-related activities and thus substantiallyimprove patient care.

Closed-loop approach for situation awareness of medical devices and operating room infrastructure

Abstract In recent years, approaches for information and control integration in the digital operating room have emerged. A major step towards an intelligent operating room and a cooperative technical environment would be autonomous adaptation of medical devices and systems to the surgical workflow. The OR staff should be freed from information seeking and maintenance tasks. We propose a closed-loop concept integrating workflow monitoring, processing and (semi-)automatic interaction to bridge the gap between OR integration of medical devices and workflow-related information management. Four steps were identified for the implementation of workflow-driven assistance functionalities. The processing steps in the closed loop of workflow-driven assistance could either be implemented with centralized responsible components or in a cooperative agent-based approach. However, both strategies require a common framework and terminology to ensure interoperability between the components, the medical devices (actors) and the OR infrastructure.

An intelligent operating room of the future – an interview with the University of California Los Angeles Medical Center

Based on an interview with the University of California Los Angeles (UCLA) Medical Center, which is currently building a new facility to include an intelligent operating room (OR). UCLA’s goals for the intelligent operating room are to lead the development of the latest robotic and computer‐assisted medical and biomedical devices and technology, to have a flexible broad intelligence of space planning, and to have a paperless yet seamless data exchange of patient and doctor information. UCLA’s planning strategy, a multidisciplinary collaborative approach to patient care, is based on a partnership between academic engineers, practicing clinicians, practicing physicians, and manufacturing industry. By creating an intelligent OR, the UCLA Medical Center hopes to improve patient satisfaction, retain and attract the best surgeons, and eventually, through the continued incorporation of image‐guided surgery, be able to decrease surgeon variability.

A voice-controlled network for universal control of devices in the OR

SummaryTo achieve the benefits of an Intelligent Operating Room (OR), we envision an OR control center—a voice-controlled network of medical devices. This central controller allows doctors and nurses to control many smart devices in a minimally-invasive OR, in the same way the operating system of a personal computer allows users to control computer peripherals, such as a printer or modem. Surgeons in the OR may use voice commands to control surgical devices when their hands are busy. In addition, the OR staff may use a touch-sensitive display panel to control any device in the OR from a single, convenient location. As an example, this paper describes a new ‘operating system for the OR’, called HERMES. Because of the unique demands of the Intelligent OR, the design of such a voice-control network involves several challenges: safety, reliability and ease of use. This paper discusses the design strategies for the HERMES control network and presents findings from the use of commercial HERMES in operating rooms.