Computer Vision Architectures Research Articles

Towards real-time monitoring of insect species populations

Insect biodiversity and abundance are in global decline, potentially leading to a crisis with profound ecological and economic consequences. Methods and technologies to monitor insect species to aid in preservation efforts are rapidly being developed yet their adoption has been slow and focused on specific use cases. We propose a computer vision model that works towards multi-objective insect species identification in real-time and on a large scale. We leverage an image data source with 16 million instances and a recent improvement in the YOLO computer vision architecture to present a quick and open-access method to develop visual AI models to monitor insect species across climatic regions.

MetaV: A Pioneer in feature Augmented Meta-Learning Based Vision Transformer for Medical Image Classification.

Image classification, a fundamental task in computer vision, faces challenges concerning limited data handling, interpretability, improved feature representation, efficiency across diverse image types, and processing noisy data. Conventional architectural approaches have made insufficient progress in addressing these challenges, necessitating architectures capable of fine-grained classification, enhanced accuracy, and superior generalization. Among these, the vision transformer emerges as a noteworthy computer vision architecture. However, its reliance on substantial data for training poses a drawback due to its complexity and high data requirements. To surmount these challenges, this paper proposes an innovative approach, MetaV, integrating meta-learning into a vision transformer for medical image classification. N-way K-shot learning is employed to train the model, drawing inspiration from human learning mechanisms utilizing past knowledge. Additionally, deformational convolution and patch merging techniques are incorporated into the vision transformer model to mitigate complexity and overfitting while enhancing feature representation. Augmentation methods such as perturbation and Grid Mask are introduced to address the scarcity and noise in medical images, particularly for rare diseases. The proposed model is evaluated using diverse datasets including Break His, ISIC 2019, SIPaKMed, and STARE. The achieved performance accuracies of 89.89%, 87.33%, 94.55%, and 80.22% for Break His, ISIC 2019, SIPaKMed, and STARE, respectively, present evidence validating the superior performance of the proposed model in comparison to conventional models, setting a new benchmark for meta-vision image classification models.

Encoding the Latent Posterior of Bayesian Neural Networks for Uncertainty Quantification.

Bayesian Neural Networks (BNNs) have long been considered an ideal, yet unscalable solution for improving the robustness and the predictive uncertainty of deep neural networks. While they could capture more accurately the posterior distribution of the network parameters, most BNN approaches are either limited to small networks or rely on constraining assumptions, e.g., parameter independence. These drawbacks have enabled prominence of simple, but computationally heavy approaches such as Deep Ensembles, whose training and testing costs increase linearly with the number of networks. In this work we aim for efficient deep BNNs amenable to complex computer vision architectures, e.g., ResNet-50 DeepLabv3+, and tasks, e.g., semantic segmentation and image classification, with fewer assumptions on the parameters. We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer. Our approach, called Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient (in terms of computation and memory during both training and testing) ensembles. LP-BNNs attain competitive results across multiple metrics in several challenging benchmarks for image classification, semantic segmentation, and out-of-distribution detection.

Multimodal integration of neuroimaging and genetic data for the diagnosis of mood disorders based on computer vision models

Mood disorders, particularly major depressive disorder (MDD) and bipolar disorder (BD), are often underdiagnosed, leading to substantial morbidity. Harnessing the potential of emerging methodologies, we propose a novel multimodal fusion approach that integrates patient-oriented brain structural magnetic resonance imaging (sMRI) scans with DNA whole-exome sequencing (WES) data. Multimodal data fusion aims to improve the detection of mood disorders by employing established deep-learning architectures for computer vision and machine-learning strategies. We analyzed brain imaging genetic data of 321 East Asian individuals, including 147 patients with MDD, 78 patients with BD, and 96 healthy controls. We developed and evaluated six fusion models by leveraging common computer vision models in image classification: Vision Transformer (ViT), Inception-V3, and ResNet50, in conjunction with advanced machine-learning techniques (XGBoost and LightGBM) known for high-dimensional data analysis. Model validation was performed using a 10-fold cross-validation. Our ViT ⊕ XGBoost fusion model with MRI scans, genomic Single Nucleotide polymorphism (SNP) data, and unweighted polygenic risk score (PRS) outperformed baseline models, achieving an incremental area under the curve (AUC) of 0.2162 (32.03% increase) and 0.0675 (+8.19%) and incremental accuracy of 0.1455 (+25.14%) and 0.0849 (+13.28%) compared to SNP-only and image-only baseline models, respectively. Our findings highlight the opportunity to refine mood disorder diagnostics by demonstrating the transformative potential of integrating diverse, yet complementary, data modalities and methodologies.

High-Throughput Plot-Level Quantitative Phenotyping Using Convolutional Neural Networks on Very High-Resolution Satellite Images

To ensure global food security, crop breeders conduct extensive trials across various locations to discover new crop varieties that grow more robustly, have higher yields, and are resilient to local stress factors. These trials consist of thousands of plots, each containing a unique crop variety monitored at intervals during the growing season, requiring considerable manual effort. In this study, we combined satellite imagery and deep learning techniques to automatically collect plot-level phenotypes from plant breeding trials in South Australia and Sonora, Mexico. We implemented two novel methods, utilising state-of-the-art computer vision architectures, to predict plot-level phenotypes: flowering, canopy cover, greenness, height, biomass, and normalised difference vegetation index (NDVI). The first approach uses a classification model to predict for just the centred plot. The second approach predicts per-pixel and then aggregates predictions to determine a value per-plot. Using a modified ResNet18 model to predict the centred plot was found to be the most effective method. These results highlight the exciting potential for improving crop trials with remote sensing and machine learning.

Developing the surgeon-machine interface: using a novel instance-segmentation framework for intraoperative landmark labelling.

The utilisation of artificial intelligence (AI) augments intraoperative safety, surgical training, and patient outcomes. We introduce the term Surgeon-Machine Interface (SMI) to describe this innovative intersection between surgeons and machine inference. A custom deep computer vision (CV) architecture within a sparse labelling paradigm was developed, specifically tailored to conceptualise the SMI. This platform demonstrates the ability to perform instance segmentation on anatomical landmarks and tools from a single open spinal dural arteriovenous fistula (dAVF) surgery video dataset. Our custom deep convolutional neural network was based on SOLOv2 architecture for precise, instance-level segmentation of surgical video data. Test video consisted of 8520 frames, with sparse labelling of only 133 frames annotated for training. Accuracy and inference time, assessed using F1-score and mean Average Precision (mAP), were compared against current state-of-the-art architectures on a separate test set of 85 additionally annotated frames. Our SMI demonstrated superior accuracy and computing speed compared to these frameworks. The F1-score and mAP achieved by our platform were 17% and 15.2% respectively, surpassing MaskRCNN (15.2%, 13.9%), YOLOv3 (5.4%, 11.9%), and SOLOv2 (3.1%, 10.4%). Considering detections that exceeded the Intersection over Union threshold of 50%, our platform achieved an impressive F1-score of 44.2% and mAP of 46.3%, outperforming MaskRCNN (41.3%, 43.5%), YOLOv3 (15%, 34.1%), and SOLOv2 (9%, 32.3%). Our platform demonstrated the fastest inference time (88ms), compared to MaskRCNN (90ms), SOLOV2 (100ms), and YOLOv3 (106ms). Finally, the minimal amount of training set demonstrated a good generalisation performance -our architecture successfully identified objects in a frame that were not included in the training or validation frames, indicating its ability to handle out-of-domain scenarios. We present our development of an innovative intraoperative SMI to demonstrate the future promise of advanced CV in the surgical domain. Through successful implementation in a microscopic dAVF surgery, our framework demonstrates superior performance over current state-of-the-art segmentation architectures in intraoperative landmark guidance with high sample efficiency, representing the most advanced AI-enabled surgical inference platform to date. Our future goals include transfer learning paradigms for scaling to additional surgery types, addressing clinical and technical limitations for performing real-time decoding, and ultimate enablement of a real-time neurosurgical guidance platform.

Lung and colon cancer detection from CT images using Deep Learning

Cancer is a deadly disease that has gained a reputation as a global health concern. Further, lung cancer has been widely reported as the most deadly cancer type globally, while colon cancer comes second. Meanwhile, early detection is one of the primary ways to prevent lung and colon cancer fatalities. To aid the early detection of lung and colon cancer, we propose a computer-aided diagnostic approach that employs a Deep Learning (DL) architecture to enhance the detection of these cancer types from Computed Tomography (CT) images of suspected body parts. Our experimental dataset (LC25000) contains 25000 CT images of benign and malignant lung and colon cancer tissues. We used weights from a pre-trained DL architecture for computer vision, EfficientNet, to build and train a lung and colon cancer detection model. EfficientNet is a Convolutional Neural Network architecture that scales all input dimensions such as depth, width, and resolution at the same time. Our research findings showed detection accuracies of 99.63%, 99.50%, and 99.72% for training, validation, and test sets, respectively.

Increasing depth of neural networks for life-long learning

Purpose:We propose a novel method for continual learning based on the increasing depth of neural networks. This work explores whether extending neural network depth may be beneficial in a life-long learning setting. Methods:We propose a novel approach based on adding new layers on top of existing ones to enable the forward transfer of knowledge and adapting previously learned representations. We employ a method of determining the most similar tasks for selecting the best location in our network to add new nodes with trainable parameters. This approach allows for creating a tree-like model, where each node is a set of neural network parameters dedicated to a specific task. The Progressive Neural Network concept inspires the proposed method. Therefore, it benefits from dynamic changes in network structure. However, Progressive Neural Network allocates a lot of memory for the whole network structure during the learning process. The proposed method alleviates this by adding only part of a network for a new task and utilizing a subset of previously trained weights. At the same time, we may retain the benefit of PNN, such as no forgetting guaranteed by design, without needing a memory buffer. Results:Experiments on Split CIFAR and Split Tiny ImageNet show that the proposed algorithm is on par with other continual learning methods. In a more challenging setup with a single computer vision dataset as a separate task, our method outperforms Experience Replay. Conclusion:It is compatible with commonly used computer vision architectures and does not require a custom network structure. As an adaptation to changing data distribution is made by expanding the architecture, there is no need to utilize a rehearsal buffer. For this reason, our method could be used for sensitive applications where data privacy must be considered.

A Review on Defect Detection of Electroluminescence-Based Photovoltaic Cell Surface Images Using Computer Vision

The past two decades have seen an increase in the deployment of photovoltaic installations as nations around the world try to play their part in dampening the impacts of global warming. The manufacturing of solar cells can be defined as a rigorous process starting with silicon extraction. The increase in demand has multiple implications for manual quality inspection. With automated inspection as the ultimate goal, researchers are actively experimenting with convolutional neural network architectures. This review presents an overview of the electroluminescence image-extraction process, conventional image-processing techniques deployed for solar cell defect detection, arising challenges, the present landscape shifting towards computer vision architectures, and emerging trends.

METHOD OF CONSTRUCTING HARDWARE ARCHITECTURE FOR COMPUTER VISION SYSTEM BASED ON FPGA

In this work, the methodology of hardware architecture development for computer vision systems based on programmable logic, in particular FPGA, is considered. In the work, the methods of developing architectures for computer vision are studied and the advantages of using FPGA compared to traditional general-purpose processors are established. The paper also considers the main aspects of hardware design on FPGA, in particular, the selection of a suitable development tool, logic design, synthesis, and validation of the developed hardware. The capabilities of FPGAs in providing high performance and efficiency of computer vision systems have been investigated, which makes them attractive and popular for use in visual systems. We discover that FPGA technology offers a high degree of flexibility and configurability, allowing for the creation of custom hardware architectures that can be tailored to specific computer vision applications. In addition, to studying the benefits of using FPGA technology for computer vision, we also consider the main aspects of hardware design on FPGA, including the selection of a suitable development tool, logic design, synthesis, and validation of the developed hardware. By carefully considering these aspects, we can ensure that the hardware architecture we develop is both efficient and effective. Our research shows that the capabilities of FPGAs in providing high performance and efficiency of computer vision systems are truly remarkable. This makes them an attractive choice for use in visual systems, particularly in scenarios where high speed and accuracy are critical. Overall, our work serves to shed light on the many benefits of FPGA technology for computer vision and lays the foundation for further research and development in this exciting field.

A Structure-Aware Convolutional Neural Network for Automatic Diagnosisof Fungal Keratitis with In Vivo Confocal Microscopy Images.

Fungal keratitis (FK) is a common and severe corneal disease, which is widely spread in tropical and subtropical areas. Early diagnosis and treatment are vital for patients, with confocal microscopy cornea imaging being one of the most effective methods for the diagnosis of FK. However, most cases are currently diagnosed by the subjective judgment of ophthalmologists, which is time-consuming and heavily depends on the experience of the ophthalmologists. In this paper, we introduce a novel structure-aware automatic diagnosis algorithm based on deep convolutional neural networks for the accurate diagnosis of FK. Specifically, a two-stream convolutional network is deployed, combining GoogLeNet and VGGNet, which are two commonly used networks in computer vision architectures. The main stream is used for feature extraction of the input image, while the auxiliary stream is used for feature discrimination and enhancement of the hyphae structure. Then, the features are combined by concatenating the channel dimension to obtain the final output, i.e., normal or abnormal. The results showed that the proposed method achieved accuracy, sensitivity, and specificity of 97.73%, 97.02%, and 98.54%, respectively. These results suggest that the proposed neural network could be a promising computer-aided FK diagnosis solution.

WaveNet: Wavelet Network with Knowledge Distillation for RGB-T Salient Object Detection.

In recent years, various neural network architectures for computer vision have been devised, such as the visual transformer and multilayer perceptron (MLP). A transformer based on an attention mechanism can outperform a traditional convolutional neural network. Compared with the convolutional neural network and transformer, the MLP introduces less inductive bias and achieves stronger generalization. In addition, a transformer shows an exponential increase in the inference, training, and debugging times. Considering a wave function representation, we propose the WaveNet architecture that adopts a novel vision task-oriented wavelet-based MLP for feature extraction to perform salient object detection in RGB (red-green-blue)-thermal infrared images. In addition, we apply knowledge distillation to a transformer as an advanced teacher network to acquire rich semantic and geometric information and guide WaveNet learning with this information. Following the shortest-path concept, we adopt the Kullback-Leibler distance as a regularization term for the RGB features to be as similar to the thermal infrared features as possible. The discrete wavelet transform allows for the examination of frequency-domain features in a local time domain and time-domain features in a local frequency domain. We apply this representation ability to perform cross-modality feature fusion. Specifically, we introduce a progressively cascaded sine-cosine module for cross-layer feature fusion and use low-level features to obtain clear boundaries of salient objects through the MLP. Results from extensive experiments indicate that the proposed WaveNet achieves impressive performance on benchmark RGB-thermal infrared datasets. The results and code are publicly available at https://github.com/nowander/WaveNet.

REVIEW AND ANALYSIS OF BASIC FEATURE DETECTION NETWORKS FOR CLASSIFICATION OF MRI IMAGES IN DEEP LEARNING MODELS

This paper presents an evaluation of modern deep learning models for the classification of MRI images of the knee joint. Among all the work related to this research, there have been several attempts to retrain the original MRNet model on more modern computer vision architectures. Also, no attempt has yet been reported to document the incremental improvement in MRNet prediction accuracy using newer computer vision architectures. This paper presents a comparative analysis of modern deep architectures of computer vision for extracting features from MRI images of the knee joint in the tasks of classification of injuries and anomalies of the knee. Such an analysis is needed, at least as a guide to creating applied architectures of machine learning models aimed at automated diagnosis of knee injuries in medical devices and systems. In the field of artificial intelligence, deep learning (DL) algorithms can be applied directly to many different musculoskeletal radiology tasks, including image reconstruction, synthetic imaging, tissue segmentation, and diagnosis and detection of musculoskeletal disease characteristics on radiographs, ultrasound , CT and MRI images. Ideally, such systems should also help radiologists focus on rare diseases as well as very complex abnormalities. At the same time, the task of automating the process of diagnosing typical injuries and anomalies is set. The level of confidence in the result of prediction should be similar to the conclusions of commissions of expert radiologists. To frame such a benchmarking analysis, this paper compares the performance of the basic MRNet architecture for the knee MRI image classification task, using various state-of-the-art computer vision architectures as framework networks for feature extraction. It also demonstrates a gradual increase in the prediction accuracy of these models in accordance with the evolution of the framework models themselves. A rather important aspect of the presented research is the fact that all machine learning models developed and trained in the considered experiment have a unified architecture, except for the feature extraction framework, and they were all trained from scratch using the same model parameters and training parameters. In addition, the model estimation strategies in this work use an additional metric that has not yet been measured and compared in any related work, namely Cohen’s Kappa metric. This metric is significant because the MRNet dataset used in this paper is not balanced.

Analysis of framework networks for sign detection in deep learning models

This paper analyzes and compares modern deep learning models for the classification of MRI images of the knee joint. An analysis of modern deep computer vision architectures for feature extraction from MRI images is presented. This analysis was used to create applied architectures of machine learning models. These models are aimed at automating the process of diagnosing knee injuries in medical devices and systems. This work is devoted to different types of feature detection framework networks for machine learning architectures that perform magnetic resonance imaging (MRI) image classification of the knee. The resulting models were evaluated on the MRNet validation dataset, calculating the metrics (ROC-AUC), prediction accuracy, F1 score, and Cohen’s K-Kappa. The results of this work also show that Cohen's Kappa metric is important for evaluating models on the MRNet architecture because it provides a deeper understanding of the classification decisions of each model.

Automation of Computer Vision Applications for Real-time Combat Sports Video Analysis

This study examines the potential applications of Human Action Recognition (HAR) in combat sports and aims to develop a prototype automation client that examines a video of a combat sports competition or training session and accurately classifies human movements. Computer Vision (CV) architectures that examine real-time video data streams are being investigated by integrating Deep Learning architectures into client-server systems for data storage and analysis using customised algorithms. The development of the automation client for training and deploying CV robots to watch and track specific chains of human actions is a central component of the project. Categorising specific chains of human actions allows for the comparison of multiple athletes' techniques as well as the identification of potential areas for improvement based on posture, accuracy, and other technical details, which can be used as an aid to improve athlete efficiency. The automation client will also be developed for the purpose of scoring, with a focus on the automation of the CV model to analyse and score a competition using a specific ruleset. The model will be validated by comparing performance and accuracy to that of combat sports experts. The primary research domains are CV, automation, robotics, combat sports, and decision science. Decision science is a set of quantitative techniques used to assist people to make decisions. The creation of a new automation client may contribute to the development of more efficient machine learning and CV applications in areas such as process efficiency, which improves user experience, workload management to reduce wait times, and run-time optimisation. This study found that real-time object detection and tracking can be combined with real-time pose estimation to generate performance statistics from a combat sports athlete's movements in a video.

Automation of Computer Vision Applications for Real-time Combat Sports Video Analysis

This study examines the potential applications of Human Action Recognition (HAR) in combat sports and aims to develop a prototype automation client that examines a video of a combat sports competition or training session and accurately classifies human movements. Computer Vision (CV) architectures that examine real-time video data streams are being investigated by integrating Deep Learning architectures into client-server systems for data storage and analysis using customised algorithms. The development of the automation client for training and deploying CV robots to watch and track specific chains of human actions is a central component of the project. Categorising specific chains of human actions allows for the comparison of multiple athletes' techniques as well as the identification of potential areas for improvement based on posture, accuracy, and other technical details, which can be used as an aid to improve athlete efficiency. The automation client will also be developed for the purpose of scoring, with a focus on the automation of the CV model to analyse and score a competition using a specific ruleset. The model will be validated by comparing performance and accuracy to that of combat sports experts. The primary research domains are CV, automation, robotics, combat sports, and decision science. Decision science is a set of quantitative techniques used to assist people to make decisions. The creation of a new automation client may contribute to the development of more efficient machine learning and CV applications in areas such as process efficiency, which improves user experience, workload management to reduce wait times, and run-time optimisation. This study found that real-time object detection and tracking can be combined with real-time pose estimation to generate performance statistics from a combat sports athlete's movements in a video.

Deep Learning Approach for Object Classification on Raw and Reconstructed GBSAR Data

The availability of low-cost microwave components today enables the development of various high-frequency sensors and radars, including Ground-based Synthetic Aperture Radar (GBSAR) systems. Similar to optical images, radar images generated by applying a reconstruction algorithm on raw GBSAR data can also be used in object classification. The reconstruction algorithm provides an interpretable representation of the observed scene, but may also negatively influence the integrity of obtained raw data due to applied approximations. In order to quantify this effect, we compare the results of a conventional computer vision architecture, ResNet18, trained on reconstructed images versus one trained on raw data. In this process, we focus on the task of multi-label classification and describe the crucial architectural modifications that are necessary to process raw data successfully. The experiments are performed on a novel multi-object dataset RealSAR obtained using a newly developed 24 GHz (GBSAR) system where the radar images in the dataset are reconstructed using the Omega-k algorithm applied to raw data. Experimental results show that the model trained on raw data consistently outperforms the image-based model. We provide a thorough analysis of both approaches across hyperparameters related to model pretraining and the size of the training dataset. This, in conclusion, shows how processing raw data provides overall better classification accuracy, it is inherently faster since there is no need for image reconstruction and it is therefore useful tool in industrial GBSAR applications where processing speed is critical.

A New Artificial Intelligence-Based Method for Identifying Mycobacterium Tuberculosis in Ziehl-Neelsen Stain on Tissue.

Mycobacteria identification is crucial to diagnose tuberculosis. Since the bacillus is very small, finding it in Ziehl–Neelsen (ZN)-stained slides is a long task requiring significant pathologist’s effort. We developed an automated (AI-based) method of identification of mycobacteria. We prepared a training dataset of over 260,000 positive and over 700,000,000 negative patches annotated on scans of 510 whole slide images (WSI) of ZN-stained slides (110 positive and 400 negative). Several image augmentation techniques coupled with different custom computer vision architectures were used. WSIs automatic analysis was followed by a report indicating areas more likely to present mycobacteria. Our model performs AI-based diagnosis (the final decision of the diagnosis of WSI belongs to the pathologist). The results were validated internally on a dataset of 286,000 patches and tested in pathology laboratory settings on 60 ZN slides (23 positive and 37 negative). We compared the pathologists’ results obtained by separately evaluating slides and WSIs with the results given by a pathologist aided by automatic analysis of WSIs. Our architecture showed 0.977 area under the receiver operating characteristic curve. The clinical test presented 98.33% accuracy, 95.65% sensitivity, and 100% specificity for the AI-assisted method, outperforming any other AI-based proposed methods for AFB detection.

Artificial Intelligence for Upper Gastrointestinal Endoscopy: A Roadmap from Technology Development to Clinical Practice.

Stomach cancer is the third deadliest type of cancer in the world (0.86 million deaths in 2017). In 2035, a 20% increase will be observed both in incidence and mortality due to demographic effects if no interventions are foreseen. Upper GI endoscopy (UGIE) plays a paramount role in early diagnosis and, therefore, improved survival rates. On the other hand, human and technical factors can contribute to misdiagnosis while performing UGIE. In this scenario, artificial intelligence (AI) has recently shown its potential in compensating for the pitfalls of UGIE, by leveraging deep learning architectures able to efficiently recognize endoscopic patterns from UGIE video data. This work presents a review of the current state-of-the-art algorithms in the application of AI to gastroscopy. It focuses specifically on the threefold tasks of assuring exam completeness (i.e., detecting the presence of blind spots) and assisting in the detection and characterization of clinical findings, both gastric precancerous conditions and neoplastic lesion changes. Early and promising results have already been obtained using well-known deep learning architectures for computer vision, but many algorithmic challenges remain in achieving the vision of AI-assisted UGIE. Future challenges in the roadmap for the effective integration of AI tools within the UGIE clinical practice are discussed, namely the adoption of more robust deep learning architectures and methods able to embed domain knowledge into image/video classifiers as well as the availability of large, annotated datasets.

Towards Automated Sample Collection and Return in Extreme Underwater Environments

In this report, we present the system design, operational strategy, and results of coordinated multivehicle field demonstrations of autonomous marine robotic technologies in searchfor- life missions within the Pacific shelf margin of Costa Rica and the Santorini-Kolumbo caldera complex, which serve as analogs to environments that may exist in oceans beyond Earth. This report focuses on the automation of remotely operated vehicle (ROV) manipulator operations for targeted biological sample-collection-and-return from the seafloor. In the context of future extraterrestrial exploration missions to ocean worlds, an ROV is an analog to a planetary lander, which must be capable of high-level autonomy. Our field trials involve two underwater vehicles, the SuBastian ROV and the Nereid Under Ice (NUI) hybrid ROV for mixed initiative (i.e., teleoperated or autonomous) missions, both equipped seven-degrees-of-freedom hydraulic manipulators.We describe an adaptable, hardware-independent computer vision architecture that enables high-level automated manipulation. The vision system provides a three-dimensional understanding of the workspace to inform manipulator motion planning in complex unstructured environments. We demonstrate the effectiveness of the vision system and control framework through field trials in increasingly challenging environments, including the automated collection and return of biological samples from within the active undersea volcano Kolumbo. Based on our experiences in the field, we discuss the performance of our system and identify promising directions for future research.