Automated Computer Vision Algorithm Research Articles

Automated surgical step recognition in transurethral bladder tumor resection using artificial intelligence: transfer learning across surgical modalities.

Automated surgical step recognition (SSR) using AI has been a catalyst in the "digitization" of surgery. However, progress has been limited to laparoscopy, with relatively few SSR tools in endoscopic surgery. This study aimed to create a SSR model for transurethral resection of bladder tumors (TURBT), leveraging a novel application of transfer learning to reduce video dataset requirements. Retrospective surgical videos of TURBT were manually annotated with the following steps of surgery: primary endoscopic evaluation, resection of bladder tumor, and surface coagulation. Manually annotated videos were then utilized to train a novel AI computer vision algorithm to perform automated video annotation of TURBT surgical video, utilizing a transfer-learning technique to pre-train on laparoscopic procedures. Accuracy of AI SSR was determined by comparison to human annotations as the reference standard. A total of 300 full-length TURBT videos (median 23.96 min; IQR 14.13-41.31 min) were manually annotated with sequential steps of surgery. One hundred and seventy-nine videos served as a training dataset for algorithm development, 44 for internal validation, and 77 as a separate test cohort for evaluating algorithm accuracy. Overall accuracy of AI video analysis was 89.6%. Model accuracy was highest for the primary endoscopic evaluation step (98.2%) and lowest for the surface coagulation step (82.7%). We developed a fully automated computer vision algorithm for high-accuracy annotation of TURBT surgical videos. This represents the first application of transfer-learning from laparoscopy-based computer vision models into surgical endoscopy, demonstrating the promise of this approach in adapting to new procedure types.

Keypoint-Based Automated Component Placement Inspection for Printed Circuit Boards

This study aims to develop novel automated computer vision algorithms and systems for component replacement inspection for printed circuit boards (PCBs). The proposed algorithms are able to identify the locations and sizes of different components. They are object detection algorithms based on key points of the target components. The algorithms can be implemented as neural networks consisting of two portions: frontend networks and backend networks. The frontend networks are used for the feature extractions of input images. The backend networks are adopted to produce component inspection results. Each component class can has its own frontend and backend networks. In this way, the neural model for the component class can be effectively reused for different PCBs. To reduce the computation time for the inference of the networks, different component classes can share the same frontend networks. A two-stage training process is proposed to effectively explore features of different components for accurate component inspection. The proposed algorithm has the advantages of simplicity in training for data collection, high accuracy in defect detection, and high reusability and flexibility for online inspection. The algorithm is an effective alternative for automated inspection in smart factories, with growing demand for product quality and diversification.

Automated Surface Defect Inspection Based on Autoencoders and Fully Convolutional Neural Networks

This study aims to develop a novel automated computer vision algorithm for quality inspection of surfaces with complex patterns. The proposed algorithm is based on both an autoencoder (AE) and a fully convolutional neural network (FCN). The AE is adopted for the self-generation of templates from test targets for defect detection. Because the templates are produced from the test targets, the position alignment issues for the matching operations between templates and test targets can be alleviated. The FCN is employed for the segmentation of a template into a number of coherent regions. Because the AE has the limitation that its capacities for the regeneration of each coherent region in the template may be different, the segmentation of the template by FCN is beneficial for allowing the inspection of each region to be independently carried out. In this way, more accurate detection results can be achieved. Experimental results reveal that the proposed algorithm has the advantages of simplicity for training data collection, high accuracy for defect detection, and high flexibility for online inspection. The proposed algorithm is therefore an effective alternative for the automated inspection in smart factories with a growing demand for the reliability for high quality production.

Urban Green Infrastructure Monitoring Using Remote Sensing from Integrated Visible and Thermal Infrared Cameras Mounted on a Moving Vehicle.

Climate change forecasts higher temperatures in urban environments worsening the urban heat island effect (UHI). Green infrastructure (GI) in cities could reduce the UHI by regulating and reducing ambient temperatures. Forest cities (i.e., Melbourne, Australia) aimed for large-scale planting of trees to adapt to climate change in the next decade. Therefore, monitoring cities’ green infrastructure requires close assessment of growth and water status at the tree-by-tree resolution for its proper maintenance and needs to be automated and efficient. This project proposed a novel monitoring system using an integrated visible and infrared thermal camera mounted on top of moving vehicles. Automated computer vision algorithms were used to analyze data gathered at an Elm trees avenue in the city of Melbourne, Australia (n = 172 trees) to obtain tree growth in the form of effective leaf area index (LAIe) and tree water stress index (TWSI), among other parameters. Results showed the tree-by-tree variation of trees monitored (5.04 km) between 2016–2017. The growth and water stress parameters obtained were mapped using customized codes and corresponded with weather trends and urban management. The proposed urban tree monitoring system could be a useful tool for city planning and GI monitoring, which can graphically show the diurnal, spatial, and temporal patterns of change of LAIe and TWSI to monitor the effects of climate change on the GI of cities.

Does agreement mean accuracy? Evaluating glance annotation in naturalistic driving data.

Naturalistic driving studies often make use of cameras to monitor driver behavior. To analyze the resulting video images, human annotation is often adopted. These annotations then serve as the ‘gold standard’ to train and evaluate automated computer vision algorithms, even though it is uncertain how accurate human annotation is. In this study, we provide a first evaluation of glance direction annotation by comparing instructed, actual glance direction of truck drivers with annotated direction. Findings indicate that while for some locations high annotation accuracy is achieved, for most locations accuracy is well below 50%. Higher accuracy can be obtained by clustering these locations, but this also leads to reduced detail of the annotation, suggesting that decisions to use clustering should take into account the purpose of the annotation. The data also show that high agreement between annotators does not guarantee high accuracy. We argue that the accuracy of annotation needs to be verified experimentally more often.

Vineyard pruning weight assessment by machine vision: towards an on-the-go measurement system

Aim: Pruning weight is an indicator of vegetative growth and vigour in grapevine. Traditionally, it is manually determined, which is time-consuming and labour-demanding. This study aims at providing a new, non-invasive and low-cost method for pruning weight estimation in commercial vineyards based on computer vision.Methods and results: The methodology relies on computer-based analysis of RGB images captured manually and on-the-go in a VSP Tempranillo vineyard. Firstly, the pruning weight estimation was evaluated using manually taken photographs using a controlled background. These images were analysed to generate a model of wood pruning weight estimation, resulting in a coefficient of determination (R2) of 0.91 (p<0.001) and a root-mean-square error (RMSE) of 87.7 g. After this, a mobile sensor platform (modified ATV) was used to take vine images automatically and on-the-go without background. These RGB images were analysed using a fully automated computer vision algorithm, resulting in R2 = 0.75 (p<0.001) and RMSE = 147.9 g. Finally, the mobile sensor platform was also used to sample a commercial VSP vineyard to map the spatial variability of wood pruning weight, and hereafter vine vigour.Conclusions: The results showed that the developed computer vision methodology was able to estimate the vine pruning weight in commercial vineyards and to map the spatial variation of the pruning weight across a vineyard.Significance and impact of the study: The presented methodology may become a valuable tool for the wine industry for rapid assessment and mapping of vine vigour. This information can be used to support decision making on pruning, fertilization and canopy management.

Technological Interventions for Medication Adherence in Adult Mental Health and Substance Use Disorders: A Systematic Review.

BackgroundMedication adherence is critical to the effectiveness of psychopharmacologic therapy. Psychiatric disorders present special adherence considerations, notably an altered capacity for decision making and the increased street value of controlled substances. A wide range of interventions designed to improve adherence in mental health and substance use disorders have been studied; recently, many have incorporated information technology (eg, mobile phone apps, electronic pill dispensers, and telehealth). Many intervention components have been studied across different disorders. Furthermore, many interventions incorporate multiple components, making it difficult to evaluate the effect of individual components in isolation.ObjectiveThe aim of this study was to conduct a systematic scoping review to develop a literature-driven, transdiagnostic taxonomic framework of technology-based medication adherence intervention and measurement components used in mental health and substance use disorders.MethodsThis review was conducted based on a published protocol (PROSPERO: CRD42018067902) in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses systematic review guidelines. We searched 7 electronic databases: MEDLINE, EMBASE, PsycINFO, the Cochrane Central Register of Controlled Trials, Web of Science, Engineering Village, and ClinicalTrials.gov from January 2000 to September 2018. Overall, 2 reviewers independently conducted title and abstract screens, full-text screens, and data extraction. We included all studies that evaluate populations or individuals with a mental health or substance use disorder and contain at least 1 technology-delivered component (eg, website, mobile phone app, biosensor, or algorithm) designed to improve medication adherence or the measurement thereof. Given the wide variety of studied interventions, populations, and outcomes, we did not conduct a risk of bias assessment or quantitative meta-analysis. We developed a taxonomic framework for intervention classification and applied it to multicomponent interventions across mental health disorders.ResultsThe initial search identified 21,749 results; after screening, 127 included studies remained (Cohen kappa: 0.8, 95% CI 0.72-0.87). Major intervention component categories include reminders, support messages, social support engagement, care team contact capabilities, data feedback, psychoeducation, adherence-based psychotherapy, remote care delivery, secure medication storage, and contingency management. Adherence measurement components include self-reports, remote direct visualization, fully automated computer vision algorithms, biosensors, smart pill bottles, ingestible sensors, pill counts, and utilization measures. Intervention modalities include short messaging service, mobile phone apps, websites, and interactive voice response. We provide graphical representations of intervention component categories and an element-wise breakdown of multicomponent interventions.ConclusionsMany technology-based medication adherence and monitoring interventions have been studied across psychiatric disease contexts. Interventions that are useful in one psychiatric disorder may be useful in other disorders, and further research is necessary to elucidate the specific effects of individual intervention components. Our framework is directly developed from the substance use disorder and mental health treatment literature and allows for transdiagnostic comparisons and an organized conceptual mapping of interventions.

Fetal Ultrasound Image Classification Using a Bag-of-words Model Trained on Sonographers’ Eye Movements

The acquisition of fetal biometric measurements via 2-D B-Mode ultrasound (US) scans is crucial for fetal monitoring. However, acquiring standardised head, abdominal and femoral image planes is challenging due to variable image quality. There remains a significant discrepancy between the way automated computer vision algorithms and human sonographers perform this task; this paper contributes to the attempt to bridge this gap by building knowledge of US image perception into a pipeline for classifying images obtained during 2-D fetal US scans. We record the eye movements of 10 participants performing 4 2-D US scans each, on a phantom fetal model at varying orientations. We analyse their eye movements to establish which high-level constraints and visual cues are used to localise the standardised abdominal plane. We then build a vocabulary of visual words trained on SURF descriptors extracted around eye fixations, and use the resulting bag of words model to classify head, abdominal and femoral image frames acquired during 10 clinical US scans and 10 further phantom US scans. On phantom data, we achieve classification accuracies of 89%, 87% and 85% for the head, abdominal and femoral images respectively. On clinical data, we achieve classification accuracies of 76%, 68% and 64% for the head, abdominal and femoral images respectively. This constitutes the first insight into image perception during real time US scanning, and a proof of concept for training bag of words models for US image analysis on human eye movements.

Performance of computer vision in vivo flow cytometry with low fluorescence contrast.

Detection and enumeration of circulating cells in the bloodstream of small animals are important in many areas of preclinical biomedical research, including cancer metastasis, immunology, and reproductive medicine. Optical in vivo flow cytometry (IVFC) represents a class of technologies that allow noninvasive and continuous enumeration of circulating cells without drawing blood samples. We recently developed a technique termed computer vision in vivo flow cytometry (CV-IVFC) that uses a high-sensitivity fluorescence camera and an automated computer vision algorithm to interrogate relatively large circulating blood volumes in the ear of a mouse. We detected circulating cells at concentrations as low as 20 cells/mL. In the present work, we characterized the performance of CV-IVFC with low-contrast imaging conditions with (1) weak cell fluorescent labeling using cell-simulating fluorescent microspheres with varying brightness and (2) high background tissue autofluorescence by varying autofluorescence properties of optical phantoms. Our analysis indicates that CV-IVFC can robustly track and enumerate circulating cells with at least 50% sensitivity even in conditions with two orders of magnitude degraded contrast than our previous in vivo work. These results support the significant potential utility of CV-IVFC in a wide range of in vivo biological models.

A computer vision approach to rare cell in vivo fluorescence flow cytometry

Noninvasive enumeration of rare circulating cell populations in small animals is of great importance in many areas of biomedical research. In this work, we describe a macroscopic fluorescence imaging system and automated computer vision algorithm that allows in vivo detection, enumeration and tracking of circulating fluorescently-labeled cells from multiple large blood vessels in the ear of a mouse. This imaging system uses a 660 nm laser and a high sensitivity electron-multiplied charge coupled device camera (EMCCD) to acquire fluorescence image sequences from relatively large (∼5 × 5 mm(2) ) imaging areas. The primary technical challenge was developing an automated method for identifying and tracking rare cell events in image sequences with substantial autofluorescence and noise content. To achieve this, we developed a two-step image analysis algorithm that first identified cell candidates in individual frames, and then merged cell candidates into tracks by dynamic analysis of image sequences. The second step was critical since it allowed rejection of >97% of false positive cell counts. Overall, our computer vision IVFC (CV-IVFC) approach allows single-cell detection sensitivity at estimated concentrations of 20 cells/mL of peripheral blood. In addition to simple enumeration, the technique recovers the cell's trajectory, which in the future could be used to automatically identify, for example, in vivo homing and docking events.

Development and validation of an automated computer vision algorithm for real-time embryo viability prediction

Time-lapse imaging can capture critical information to predict embryo viability (Wong et al, Nature Biotech, 2010). However, the manual processing of abundant image data poses a huge hurdle for clinical translation. Here we developed and validated a state-of-the-art computer vision algorithm that can automatically measure cell division timings and generate real-time embryo viability predictions.

Combining Wireless Neural Recording and Video Capture for the Analysis of Natural Gait.

Neural control of movement is typically studied in constrained environments where there is a reduced set of possible behaviors. This constraint may unintentionally limit the applicability of findings to the generalized case of unconstrained behavior. We hypothesize that examining the unconstrained state across multiple behavioral contexts will lead to new insights into the neural control of movement and help advance the design of neural prosthetic decode algorithms. However, to pursue electrophysiological studies in such a manner requires a more flexible framework for experimentation. We propose that head-mounted neural recording systems with wireless data transmission, combined with markerless computer-vision based motion tracking, will enable new, less constrained experiments. As a proof-of-concept, we recorded and wirelessly transmitted broadband neural data from 32 electrodes in premotor cortex while acquiring single-camera video of a rhesus macaque walking on a treadmill. We demonstrate the ability to extract behavioral kinematics using an automated computer vision algorithm without use of markers and to predict kinematics from the neural data. Together these advances suggest that a new class of "freely moving monkey" experiments should be possible and should help broaden our understanding of the neural control of movement.