Automated Computer Vision Research Articles

High-fidelity Image Restoration of Large 3D Electron Microscopy Volume.

Volume electron microscopy (VEM) is an essential tool for studying biological structures. Due to the challenges of sample preparation and continuous volumetric imaging, image artifacts are almost inevitable. Such image artifacts complicate further processing both for automated computer vision methods and human experts. Unfortunately, the widely used contrast limited adaptive histogram equalization (CLAHE) can alter the essential relative contrast information about some biological structures. We developed an image-processing pipeline to remove the artifacts and enhance the images without CLAHE. We apply our method to VEM datasets of a Microwasp head. We demonstrate that our method restores the images with high fidelity while preserving the original relative contrast. This pipeline is adaptable to other VEM datasets.

Comparative analysis of personal protective equipment nonadherence detection: computer vision versus human observers.

Human monitoring of personal protective equipment (PPE) adherence among healthcare providers has several limitations, including the need for additional personnel during staff shortages and decreased vigilance during prolonged tasks. To address these challenges, we developed an automated computer vision system for monitoring PPE adherence in healthcare settings. We assessed the system performance against human observers detecting nonadherence in a video surveillance experiment. The automated system was trained to detect 15 classes of eyewear, masks, gloves, and gowns using an object detector and tracker. To assess how the system performs compared to human observers in detecting nonadherence, we designed a video surveillance experiment under 2 conditions: variations in video durations (20, 40, and 60 seconds) and the number of individuals in the videos (3 versus 6). Twelve nurses participated as human observers. Performance was assessed based on the number of detections of nonadherence. Human observers detected fewer instances of nonadherence than the system (parameter estimate -0.3, 95% CI -0.4 to -0.2, P < .001). Human observers detected more nonadherence during longer video durations (parameter estimate 0.7, 95% CI 0.4-1.0, P < .001). The system achieved a sensitivity of 0.86, specificity of 1, and Matthew's correlation coefficient of 0.82 for detecting PPE nonadherence. An automated system simultaneously tracks multiple objects and individuals. The system performance is also independent of observation duration, an improvement over human monitoring. The automated system presents a potential solution for scalable monitoring of hospital-wide infection control practices and improving PPE usage in healthcare settings.

Nitroglycerin challenge identifies microcirculatory target for improved resuscitation in patients with circulatory shock

BackgroundCirculatory shock and multi-organ failure remain major contributors to morbidity and mortality in critically ill patients and are associated with insufficient oxygen availability in the tissue. Intrinsic mechanisms to improve tissue perfusion, such as up-regulation of functional capillary density (FCD) and red blood cell velocity (RBCv), have been identified as maneuvers to improve oxygen extraction by the tissues; however, their role in circulatory shock and potential use as resuscitation targets remains unknown. To fill this gap, we examined the baseline and maximum recruitable FCD and RBCv in response to a topical nitroglycerin stimulus (FCDNG, RBCvNG) in patients with and without circulatory shock to test whether this may be a method to identify the presence and magnitude of a microcirculatory reserve capacity important for identifying a resuscitation target.MethodsSublingual handheld vital microscopy was performed after initial resuscitation in mechanically ventilated patients consecutively admitted to a tertiary medical ICU. FCD and RBCv were quantified using an automated computer vision algorithm (MicroTools). Patients with circulatory shock were retrospectively identified via standardized hemodynamic and clinical criteria and compared to patients without circulatory shock.Results54 patients (57 ± 14y, BMI 26.3 ± 4.9 kg/m2, SAPS 56 ± 19, 65% male) were included, 13 of whom presented with circulatory shock. Both groups had similar cardiac index, mean arterial pressure, RBCv, and RBCvNG. Heart rate (p < 0.001), central venous pressure (p = 0.02), lactate (p < 0.001), capillary refill time (p < 0.01), and Mottling score (p < 0.001) were higher in circulatory shock after initial resuscitation, while FCD and FCDNG were 10% lower (16.9 ± 4.2 and 18.9 ± 3.2, p < 0.01; 19.3 ± 3.1 and 21.3 ± 2.9, p = 0.03). Nitroglycerin response was similar in both groups, and circulatory shock patients reached FCDNG similar to baseline FCD found in patients without shock.ConclusionCritically ill patients suffering from circulatory shock were found to present with a lower sublingual FCD. The preserved nitroglycerin response suggests a dysfunction of intrinsic regulation mechanisms to increase the microcirculatory oxygen extraction capacity associated with circulatory shock and identifies a potential resuscitation target. These differences in microcirculatory hemodynamic function between patients with and without circulatory shock were not reflected in blood pressure or cardiac index.

Automated computer vision based individual salmon (Salmo salar) breathing rate estimation (SaBRE) for improved state observability

Salmon farming plays an important role in the Norwegian food industry, supplying a large portion of the world's salmon. Despite its economic importance, salmon farming faces unique biological challenges impacting the health and welfare of the fish. These factors limit the growth of salmon and the full economic potential of salmon farms, highlighting the need for advanced techniques in aquaculture to enhance productivity and sustainability. A key technical challenge is the unavailability of effective monitoring tools, leading to greater difficulties assessing the condition of salmon in water compared to the assessment of the animal stock on land-based farms. To improve the observability of salmon farms, we designed and created a computer-vision based approach for salmon breathing rate estimation (SaBRE), which allows the automatic monitoring of the respiration frequency of each individual salmon in a group of fish, seamlessly covering the entire workflow from video-stream input to the final data output (end-to-end). We thoroughly evaluated the capabilities of our method in two ways. Firstly, we performed a quantitative analysis of the constituent modules of SaBRE, revealing that all modules were highly accurate, including a salmon re-identification module that achieved an accuracy of 99.51 %. Secondly, we analyzed data from a salmon experiment with SaBRE, demonstrating that our algorithm provides high-quality respiration frequency information that can be compared with other types of experimental data to infer biological relationships. For the fish in our experiment, we observed that the ranking of the respiration frequencies in individual salmon remains relatively unchanged both in the short term and over longer periods (i.e., a salmon with a high breathing rate consistently remains among those with the highest rates, regardless of changes in the environment). Furthermore, a significant negative correlation (Pearson correlation coefficients with r values between −0.61 and −0.90 and p values below 0.01) was observed between our algorithm's estimated respiration frequency and the dissolved oxygen (dO2) content in the water. In addition, the average breathing rate of the salmon was observed to increase in response to incidents potentially causing stress.

Innovative cloud quantification: deep learning classification and finite-sector clustering for ground-based all-sky imaging

Abstract. Accurate cloud quantification is essential in climate change research. In this work, we construct an automated computer vision framework by synergistically incorporating deep neural networks and finite-sector clustering to achieve robust whole-sky image-based cloud classification, adaptive segmentation and recognition under intricate illumination dynamics. A bespoke YOLOv8 (You Only Look Once 8) architecture attains over 95 % categorical precision across four archetypal cloud varieties curated from extensive annual observations (2020) at a Tibetan highland station. Tailor-made segmentation strategies adapted to distinct cloud configurations, allied with illumination-invariant image enhancement algorithms, effectively eliminate solar interference and substantially boost quantitative performance even in illumination-adverse analysis scenarios. Compared with the traditional threshold analysis method, the cloud quantification accuracy calculated within the framework of this paper is significantly improved. Collectively, the methodological innovations provide an advanced solution to markedly escalate cloud quantification precision levels imperative for climate change research while offering a paradigm for cloud analytics transferable to various meteorological stations.

Dual-path lightweight convolutional neural network for automatic sorting of olive fruit based on cultivar and maturity

Identifying olive cultivars and maturity stages is crucial in the olive industry, as these traits significantly impact the nutritional and sensory properties of olive products and extracted oil. For this purpose, this study presents a novel automatic computer vision system that applies state-of-the-art deep learning technology to sort and classify two Iranian olive cultivars, Zard and Roghani, in five maturity stages, resulting in a total of ten distinct classes. The model was developed by evaluating multiple user-defined and standard structures. It was based on a dual-path lightweight convolutional neural network that uses both regular and dilated convolution operators. Dilated convolutions were used to extract more information and capture different properties by providing larger receptive fields. With a significantly lower number of trainable parameters than standard architectures, the lightweight nature of the model would enhance its potential for delivering fast responses in on-the-go applications. Four optimizers (RMSProp, SGD, Adam, and Nadam) were tested on the developed model to enhance its performance, and Nadam exhibited the greatest accuracy. The proposed model achieved a total classification accuracy of 95.79 % and a loss of 0.2214. The proposed model was completely accurate for some classes, and the classification metrics for all categories were high, ranging from 88 % to 100 % for precision, 83–100 % for recall, and 86–100 % for F1-score. The accuracy of classification within the Roghani cultivar classes stood at 98.28 %, while for the Zard cultivar classes, it achieved 97.76 %. The study found that the proposed model can be efficiently incorporated into an olive sorting system, facilitating the identification of olives with different cultivars and varying levels of maturity, thereby enhancing the production of post-harvest products and superior quality of oil.

Updating predictions in a complex repertoire of actions and its neural representation

Even though actions we observe in everyday life seem to unfold in a continuous manner, they are automatically divided into meaningful chunks, that are single actions or segments, which provide information for the formation and updating of internal predictive models. Specifically, boundaries between actions constitute a hub for predictive processing since the prediction of the current action comes to an end and calls for updating of predictions for the next action. In the current study, we investigated neural processes which characterize such boundaries using a repertoire of complex action sequences with a predefined probabilistic structure. Action sequences consisted of actions that started with the hand touching an object (T) and ended with the hand releasing the object (U). These action boundaries were determined using an automatic computer vision algorithm. Participants trained all action sequences by imitating demo videos. Subsequently, they returned for an fMRI session during which the original action sequences were presented in addition to slightly modified versions thereof. Participants completed a post-fMRI memory test to assess the retention of original action sequences. The exchange of individual actions, and thus a violation of action prediction, resulted in increased activation of the action observation network and the anterior insula. At U events, marking the end of an action, increased brain activation in supplementary motor area, striatum, and lingual gyrus was indicative of the retrieval of the previously encoded action repertoire. As expected, brain activation at U events also reflected the predefined probabilistic branching structure of the action repertoire. At T events, marking the beginning of the next action, midline and hippocampal regions were recruited, reflecting the selected prediction of the unfolding action segment. In conclusion, our findings contribute to a better understanding of the various cerebral processes characterizing prediction during the observation of complex action repertoires.

Automated Detection and Counting of Wild Boar in Camera Trap Images.

Camera traps are becoming widely used for wildlife monitoring and management. However, manual analysis of the resulting image sets is labor-intensive, time-consuming and costly. This study shows that automated computer vision techniques can be extremely helpful in this regard, as they can rapidly and automatically extract valuable information from the images. Specific training with a set of 1600 images obtained from a study where wild animals approaching wild boar carcasses were monitored enabled the model to detect five different classes of animals automatically in their natural environment with a mean average precision of 98.11%, namely 'wild boar', 'fox', 'raccoon dog', 'deer' and 'bird'. In addition, sequences of images were automatically analyzed and the number of wild boar visits and respective group sizes were determined. This study may help to improve and speed up the monitoring of the potential spread of African swine fever virus in areas where wild boar are affected.

Acceleration of flow-focused liquid jets in the presence of a strong electric field

This paper investigates the acceleration of flow-focused micro-jets under an applied external electric field. Such thin and fast jets are needed, for example, to deliver protein crystals in X-ray free electron laser experiments. This contribution focuses on analysing the acceleration of liquid jets produced from 50 % vol water and ethanol mixture. Such jets are focused with nitrogen gas and are subjected to 0 - 7 kV electric potential between a submerged electrode and an external electrode downstream of the nozzle. To gain insights into jet geometric properties an automatic image recognition and computer vision algorithm is utilised on already published data. Jet velocities and acceleration calculated for two jetting regimes show that application of external electric field accelerates the jets up to four orders of magnitude larger than the conventional micro-jets.

Demonstration of computer vision for void characterisation of 3D-printed continuous carbon fibre composites

Microstructural void formation during the processing of continuous fibre-reinforced polymer composites are a significant limitation of processes such as 3D printing, as voids inhibit resultant mechanical properties. Traditional optical microscopy approaches to void characterisation are tedious and are prone to errors due to non-ideal but realistic image conditions, including low contrast typical of microscopic images. This paper proposes a novel application of automated computer vision image processing techniques (i.e. a contrast-limited adaptive histogram equalisation (CLAHE) algorithm and Otsu's method) for detailed void characterisation of continuous fibre-reinforced composites. Microstructural void fraction was determined for both unidirectional and bi-axial laminates of a variety of thicknesses. The void characterisation by computer vision was shown to produce comparable results with manual contrast/brightness control, but with significantly less standard deviation. The computer vision software used in void characterisation is made freely available online at https://github.com/Xiao-ying/VoidDetector.

Automated surgical step recognition in transurethral bladder tumor resection using artificial intelligence: transfer learning across surgical modalities.

Automated surgical step recognition (SSR) using AI has been a catalyst in the "digitization" of surgery. However, progress has been limited to laparoscopy, with relatively few SSR tools in endoscopic surgery. This study aimed to create a SSR model for transurethral resection of bladder tumors (TURBT), leveraging a novel application of transfer learning to reduce video dataset requirements. Retrospective surgical videos of TURBT were manually annotated with the following steps of surgery: primary endoscopic evaluation, resection of bladder tumor, and surface coagulation. Manually annotated videos were then utilized to train a novel AI computer vision algorithm to perform automated video annotation of TURBT surgical video, utilizing a transfer-learning technique to pre-train on laparoscopic procedures. Accuracy of AI SSR was determined by comparison to human annotations as the reference standard. A total of 300 full-length TURBT videos (median 23.96 min; IQR 14.13-41.31 min) were manually annotated with sequential steps of surgery. One hundred and seventy-nine videos served as a training dataset for algorithm development, 44 for internal validation, and 77 as a separate test cohort for evaluating algorithm accuracy. Overall accuracy of AI video analysis was 89.6%. Model accuracy was highest for the primary endoscopic evaluation step (98.2%) and lowest for the surface coagulation step (82.7%). We developed a fully automated computer vision algorithm for high-accuracy annotation of TURBT surgical videos. This represents the first application of transfer-learning from laparoscopy-based computer vision models into surgical endoscopy, demonstrating the promise of this approach in adapting to new procedure types.

Using Computer Vision to Detect E-cigarette Content in TikTok Videos.

Previous research has identified abundant e-cigarette content on social media using primarily text-based approaches. However, frequently used social media platforms among youth, such as TikTok, contain primarily visual content, requiring the ability to detect e-cigarette-related content across large sets of videos and images. This study aims to use a computer vision technique to detect e-cigarette-related objects in TikTok videos. We searched 13 hashtags related to vaping on TikTok (eg, #vape) in November 2022 and obtained 826 still images extracted from a random selection of 254 posts. We annotated images for the presence of vaping devices, hands, and/or vapor clouds. We developed a YOLOv7-based computer vision model to detect these objects using 85% of extracted images (N = 705) for training and 15% (N = 121) for testing. Our model's recall value was 0.77 for all three classes: vape devices, hands, and vapor. Our model correctly classified vape devices 92.9% of the time, with an average F1 score of 0.81. The findings highlight the importance of having accurate and efficient methods to identify e-cigarette content on popular video-based social media platforms like TikTok. Our findings indicate that automated computer vision methods can successfully detect a range of e-cigarette-related content, including devices and vapor clouds, across images from TikTok posts. These approaches can be used to guide research and regulatory efforts. Object detection, a computer vision machine learning model, can accurately and efficiently identify e-cigarette content on a primarily visual-based social media platform by identifying the presence of vaping devices and evidence of e-cigarette use (eg, hands and vapor clouds). The methods used in this study can inform computational surveillance systems for detecting e-cigarette content on video- and image-based social media platforms to inform and enforce regulations of e-cigarette content on social media.

P153 Artificial Intelligence to predict interleukin-23 signalling activity from H&amp;E stained whole slide images of Inflammatory Bowel Disease

Abstract Background Interleukin23 (IL-23) is a cytokine that plays a crucial role in the pathogenesis of inflammatory bowel disease (IBD), making it a highly validated therapeutic target. Understanding the role of IL-23 in IBD at the histopathological level is crucial for determining effective treatment strategies, providing insights into IBD patients who fail to respond to targeted therapies, or predicting those who are likely to lose response. In this context, there is a surge in utilizing artificial intelligence (AI) for histopathological data in IBD and other disease indications. Here, we present an automated computer vision approach to predict IL-23 signalling activity directly from routinely stained Hematoxylin and Eosin (H&amp;E) images Methods A total of 1502 samples with matched clinical data and H&amp;E biopsy images were included from 991 Crohn’s disease (CD) and 511 ulcerative colitis (UC) samples. IL-23 signalling activity was calculated using gene set variation analysis on RNA-seq data collected from the same tissue biopsies. The data were obtained from the IBD Plexus program of the Crohn’s &amp; Colitis Foundation. The proposed approach is based on vision transformers (ViTs) which is a type of deep learning model. ViTs divide the input image into fixed-size patches, transform it into linear embedding, and analyze it with the self-attention mechanism. This enables ViTs to incorporate relationships between different patches of the input image to identify regions predictive of IL-23. Our approach was trained in a weakly supervised manner to automatically identify tissue regions that correlate with IL-23 signaling activity. The model produces interpretable heatmaps to interrogate model predictions and allow clinicians to visualize and interpret the significance of different tissue regions predictive of IL-23 Results We performed 5-fold cross-validation on the splits obtained at the patient level, retaining the data distribution of IL-23 signaling activity, biopsy location, and diagnosis. We separately validated the performance of the proposed model on both disease categories, including CD and UC. The proposed approach achieved an area under the curve (AUC) of 0.82 ± 0.04 on unseen data from CD and an AUC of 0.80 ± 0.02 for UC. The 5-fold results for both disease categories are shown below Conclusion The presented results highlight the significance of computational pathology algorithms to identify IL-23 signalling activity from H&amp;E images. Pathological interpretation from the heatmaps may help understand disease pathomechanism and optimize the treatment options for IBD patients by timely identification of IL-23 status. We are further validating the clinical utility of such heatmaps and expanding the use of H&amp;E to predict other patient-centric endpoints

Wavelet scattering‐ and object detection‐based computer vision for identifying dengue from peripheral blood microscopy

AbstractDengue fever infection is a global health concern. Early disease detection is crucial for averting complications and fatality. Characteristic morphological changes in lymphocytes can be observed on a peripheral blood smear (PBS) in cases of dengue infection. In this research, we have developed automated computer vision models for dengue detection on PBS images using two approaches: wavelet scattering transform (WST)‐based feature engineering and classification and You Only Look Once (YOLO)‐based deep transfer learning for object detection. In the former, Morlet wavelet scattering features extracted from lymphocytes were used as input for five shallow classifiers for image classification. Among these, the support vector machine achieved the best results of 98.7% accuracy using 10‐fold cross‐validation. In the latter, computer vision‐enabled object detection was implemented using five YOLOv8 scaled variants. Among these, YOLOv8s and YOLOv8l attained identical best mean accuracy of 99.3% ± 1.4% across five independent experiments. Our results confirmed the feasibility and excellent diagnostic accuracy for both WST‐ and YOLOv8‐enabled computer vision approaches for diagnosing dengue infection in PBS images. This research incorporates deep machine learning along with AI technology to enhance understanding and capabilities in automated Dengue diagnosis. The significance of this research extends to the broader domain of mosquito‐borne illnesses. However, it is important to note that the findings are limited to the dataset used by the researchers.

A Modified Xception Deep Learning Model for Automatic Sorting of Olives Based on Ripening Stages

Olive fruits at different ripening stages give rise to various table olive products and oil qualities. Therefore, developing an efficient method for recognizing and sorting olive fruits based on their ripening stages can greatly facilitate post-harvest processing. This study introduces an automatic computer vision system that utilizes deep learning technology to classify the ‘Roghani’ Iranian olive cultivar into five ripening stages using color images. The developed model employs convolutional neural networks (CNN) and transfer learning based on the Xception architecture and ImageNet weights as the base network. The model was modified by adding some well-known CNN layers to the last layer. To minimize overfitting and enhance model generality, data augmentation techniques were employed. By considering different optimizers and two image sizes, four final candidate models were generated. These models were then compared in terms of loss and accuracy on the test dataset, classification performance (classification report and confusion matrix), and generality. All four candidates exhibited high accuracies ranging from 86.93% to 93.46% and comparable classification performance. In all models, at least one class was recognized with 100% accuracy. However, by taking into account the risk of overfitting in addition to the network stability, two models were discarded. Finally, a model with an image size of 224 × 224 and an SGD optimizer, which had a loss of 1.23 and an accuracy of 86.93%, was selected as the preferred option. The results of this study offer robust tools for automatic olive sorting systems, simplifying the differentiation of olives at various ripening levels for different post-harvest products.

The looming threat of digital determinism and discrimination: Commentary on “The feudal glove of talent-selection decisions in sport – Strengthening the link between subjective and objective assessments” by Bar-Eli et al. (2023)

Recent technological developments, such as automatic broadcasting systems and computer vision, are enabling cost-effective data gathering, bringing big data analytics to youth sports. In a recent insightful article, Bar-Eli et al. (2023) discussed among other issues the subjective and objective talent selection approaches, advocating for the use of big data analytics to harness the benefits of both methods. My aim in this commentary is to complement that big data predictive tools also were found to be vulnerable to biases that render their decisions unfair. Therefore, artificial intelligence in the domain of youth talent identification should be approached with caution and in-depth consideration of ethical issues needs to be undertaken before these predictive tools can be widely adopted.

Lymphoma Cell Nuclei Classification using Color and Morphology Features

Abstract Diffuse large B-cell lymphoma (DLBCL) is the most common subtype of Non-Hodgkin’s Lymphoma, presenting a great challenge for treatment due to its highly heterogeneous nature. DLBCL is diagnosed based on microscopy images of patient tissue samples. To help gain a better understanding of DLBCL, we developed an automated computer vision method to analyze morphological and color-based information within patient biopsies. We analyzed a dataset of whole slide images of DLBCL by segmenting individual cells and representing cell morphologies through a set of engineered features. The features were evaluated using a variety of visualization and machine learning (ML) classification techniques. Current state-of-the-art deep learning methods use images as the input in classification tasks achieving high performance but lacking in interpretability. A big challenge lies in finding out what features the pixel-based deep learning methods utilize in prediction. Here, we present a technique that not only yields high prediction accuracy but also provides insights into which of the features are key for prediction. We show that the color-based features have the highest importance for cell classification, allowing for the accurate identification of various cell types with an accuracy of 84% in a multi-class and 91% in a binary classification. Our results provide valuable insights for exploring cell image datasets to gain an in-depth view of the tumor microenvironment.

Keypoint-Based Automated Component Placement Inspection for Printed Circuit Boards

This study aims to develop novel automated computer vision algorithms and systems for component replacement inspection for printed circuit boards (PCBs). The proposed algorithms are able to identify the locations and sizes of different components. They are object detection algorithms based on key points of the target components. The algorithms can be implemented as neural networks consisting of two portions: frontend networks and backend networks. The frontend networks are used for the feature extractions of input images. The backend networks are adopted to produce component inspection results. Each component class can has its own frontend and backend networks. In this way, the neural model for the component class can be effectively reused for different PCBs. To reduce the computation time for the inference of the networks, different component classes can share the same frontend networks. A two-stage training process is proposed to effectively explore features of different components for accurate component inspection. The proposed algorithm has the advantages of simplicity in training for data collection, high accuracy in defect detection, and high reusability and flexibility for online inspection. The algorithm is an effective alternative for automated inspection in smart factories, with growing demand for product quality and diversification.

The edge segmentation of grains in thin-section petrographic images utilising extinction consistency perception network

The first step in petrographic image analysis is to segment the mineral grains in the thin section petrographic image, allowing the petrographer to identify the rock based on the grain size and composition. The purpose of this research is to present an automated computer vision solution for obtaining accurate and efficient edge segmentation maps. The proposed model, termed the extinction consistency perception network, is made up of three sections. The multi-angle extinction consistency block, which is based on the extinction consistency of mineral grains, uses consecutive petrographic images to generate edge-enhanced features. Then they are processed by the multi-scale edge perception network to obtain rich expression of edge features at different levels. Afterward, a distance-map penalized compound loss function is introduced to guide the model to pay more attention to grains’ edges. The generated cross-polarized petrographic image dataset (CPPID) with meticulous annotations has been shared with the community. Experimental findings show that the proposed model is effective, which is evaluated on CPPID and scores 0.940 ODS and 0.941 OIS, outperforming seven classic edge detection models by a large margin.

"Automated Computer Vision Technology Algorithm for Wrist Joint Detection for Rheumatoid Arthritis"

"Automated Computer Vision Technology Algorithm for Wrist Joint Detection for Rheumatoid Arthritis"