Computer Vision Techniques Research Articles

Cobots at the Museum: The Role of Robotics in Digitisation

In the last decade, the Natural History Museum, UK (NHM), has been at the forefront of the digitisation of natural history collections, with almost six million of its 80 million specimens digitised. This momentous undertaking has led to numerous innovations on how to optimise digitisation workflows. One avenue that is currently being explored is the use of collaborative robots—cobots. Since acquiring a Techman TM5 900 robotic arm in 2023 (Scott et al. 2023), we have been experimenting with its capabilities. Experiments began with simple pick-and-place tasks, using artificial specimens. Next, we focused on two use-cases, based on the digitisation of shark teeth and pinned-insects. Both shark teeth and pinned insects are in abundance at the NHM, making their manual digitisation a tedious task. Currently, we have trained the cobot to pick up a specimen, move it elsewhere to photograph and scan the specimen, then move it back to its original position or to a new place. Thus far, this has all been coordinate-based. Focusing on pinned insect specimens, we have now begun training deep learning models to perform segmentation, classification, and tracking tasks on images and manually taken videos. Segmentation and classification tasks range from distinguishing specimens from one another within drawers, to classifying different pins, labels, and insects. Meanwhile, object tracking methods are utilised to track labels from videos taken around the specimen. By tracking different labels simultaneously from multiple frames, we can combine the views of the labels in order to obtain a full picture for each label (for example, using tools described in Salili-James et al. 2022). Thus far, our machine learning pipelines have proved successful, for example, with F1 scores of 96–98% to classify and segment insects and to locate pin heads from dorsal views. Soon, we will be establishing workflows that integrate computer vision (CV) and machine learning (ML) techniques directly with the robotic arm, with pipelines that could be applied to different datasets, and that can significantly enhance efficiency. Broadly, these pipelines can be split into four sections: Specimen Identification: CV/ML to locate individual specimens or certain parts of specimens e.g., pinheads within pinned insects. Handling: With custom grippers, the cobot can delicately pick up, move, and place specimens, to and from photography stations for high quality scanning. Imaging & Scanning: The cobot will scan and photograph specimens. As the camera moves around the specimen, ML is used to segment and track the specimen labels to image them from the optimal views. A built-in Optical Character Recognition process can also be integrated to perform automatic transcription from here. Identifiers: This step requires the cobot to attach identifier labels to specimens or drawers, after locating the optimal position to do this, using CV/ML. Specimen Identification: CV/ML to locate individual specimens or certain parts of specimens e.g., pinheads within pinned insects. Handling: With custom grippers, the cobot can delicately pick up, move, and place specimens, to and from photography stations for high quality scanning. Imaging & Scanning: The cobot will scan and photograph specimens. As the camera moves around the specimen, ML is used to segment and track the specimen labels to image them from the optimal views. A built-in Optical Character Recognition process can also be integrated to perform automatic transcription from here. Identifiers: This step requires the cobot to attach identifier labels to specimens or drawers, after locating the optimal position to do this, using CV/ML. In this talk, we will discuss the progress of the NHM’s cobot research and explore the future of robotics for the digitisation of natural history collections.

Classification of Coral Reef Species using Computer Vision and Deep Learning Techniques

Classification of Coral Reef Species using Computer Vision and Deep Learning Techniques

Digital Food Sensing and Ingredient Analysis Techniques to Facilitate Human-Food Interface Designs

Interactive technologies that shape the traditional human-food experiences are being explored under the emerging field of Human-Food Interaction (HFI). A key challenge in developing HFI technologies is the digital sensing of food, beverages, and their ingredients, commonly known as digital food sensing. Digital food sensing involves recognizing different food and beverages and their internal attributes, such as volume and ingredients (e.g., salt and sugar content). Contemporary research on interactive food applications, such as dietary assessment, primarily employs Computer Vision (CV) techniques to identify food; however, they are ineffective when (1) identifying food’s internal attributes, (2) discriminating visually similar food and beverages, and (3) seamlessly integrating with people’s natural interactions while consuming food. Thus, this article reviews potential food and beverage sensing technologies that can facilitate novel Human-Food Interfaces, primarily focusing on non-disruptive sensing techniques to analyze food and beverages during consumption. First, we review ten different digital food sensing techniques and their applications in four categories. Then, we discuss three main aspects to consider when adopting these food-sensing techniques for human-food interface designs. Finally, we suggest the future research requirements in digital food sensing methodologies, followed by potential applications of digital food sensing in future developments of Human-Food Interfaces.

Empowering Brain Tumor Diagnosis through Explainable Deep Learning

Brain tumors are among the most lethal diseases, and early detection is crucial for improving patient outcomes. Currently, magnetic resonance imaging (MRI) is the most effective method for early brain tumor detection due to its superior imaging quality for soft tissues. However, manual analysis of brain MRI scans is prone to errors, largely influenced by the radiologists’ experience and fatigue. To address these challenges, computer-aided diagnosis (CAD) systems are more significant. These advanced computer vision techniques such as deep learning provide accurate predictions based on medical images, enhancing diagnostic precision and reliability. This paper presents a novel CAD framework for multi-class brain tumor classification. The framework employs six pre-trained deep learning models as the base and incorporates comprehensive data preprocessing and augmentation strategies to enhance computational efficiency. To address issues related to transparency and interpretability in deep learning models, Gradient-weighted Class Activation Mapping (Grad-CAM) is utilized to visualize the decision-making processes involved in tumor classification from MRI scans. Additionally, a user-friendly Brain Tumor Detection System has been developed using Streamlit, demonstrating its practical applicability in real-world settings and providing a valuable tool for clinicians. All simulation results are derived from a public benchmark dataset, showing that the proposed framework achieves state-of-the-art performance, with accuracy approaching 99% in ResNet-50, Xception, and InceptionV3 models.

The suitability of AI in dermatology for enhanced skin care

This piece highlights the tremendous potential of Artificial Intelligence (AI) in the field of dermatology and its suitability in revolutionising patient care. The integration of AI technologies into dermatological practices has the power to significantly improve diagnostics, treatment decisions, and overall patient outcomes. AI algorithms have shown remarkable proficiency in analysing dermatological images with impressive accuracy, such as skin lesions, rashes and moles. By leveraging deep learning and computer vision techniques, AI models can recognise patterns, features, and characteristics of various skin conditions, thereby aiding in accurate diagnosis and assists dermatologists in formulating personalised treatment plans tailored to individual patients.

Transfer learning with YOLOV8 for real-time recognition system of American Sign Language Alphabet

Sign language serves as a sophisticated means of communication vital to individuals who are deaf or hard of hearing, relying on hand movements, facial expressions, and body language to convey nuanced meaning. American Sign Language (ASL) exemplifies this linguistic complexity with its distinct grammar and syntax. The advancement of real-time ASL gesture recognition has explored diverse methodologies, including motion sensors and computer vision techniques. This study specifically addresses the recognition of ASL alphabet gestures using computer vision through Mediapipe for hand movement tracking and YOLOv8 for training the deep learning model. The model achieved notable performance metrics: precision of 98%, recall rate of 98%, F1 score of 99%, mean Average Precision (mAP) of 98%, and mAP50-95 of 93%, underscoring its exceptional accuracy and sturdy capabilities.

Real-Time Object Detection and Tracking for Occupational Health and Safety (OHS) in Mining Operations Using Streamed CCTV Data

Abstract Occupational Safety and Health (OHS) is very important in the mining industry which is prone to accident risks. Automatic detection of violations through CCTV streaming with the concept of a perimeter area is needed to prevent accidents. The purpose of this research is to create a violation detection system without re-modeling training that can adapt to different objects and regions. The proposed violation detection model includes four computational steps, namely (i) data reading, (ii) data pre-processing, (iii) object detection, and (iv) object tracking and violation detection. The computer vision technique uses the SIFT (Scale Invariant Feature Transform) algorithm and heuristics implemented with OpenCV. We used five CCTV video recordings of mining operations as examples. The results showed an average precision of 61.14%, a precision of 80.41%, and a recall of 69.38%. The average computation time of the model is 0.3 seconds per frame with heuristics that allow the model to be used in real-time. The model can be widely used in various fields and locations without having to retrain the model to improve OHS in the mining industry and other sectors with similar risks. This study will contribute to the development of a flexible, fast, and real-time automatic detection system.

Identification of flange specification in real industrial settings with human reasoning assisted by augmented reality

Flange is an important component that connects pressure pipes in livelihood and industrial facilities. The specification information for a flange installed a long time ago is often unavailable or was never recorded. Precision measurement for determining the specification requires detaching the flange from connecting pipes, thus interrupting the facility’s operation. Computer vision techniques cannot guarantee reliable results for mounted flanges that are partially occluded or damaged in real environments. This study develops a new solution, augmented reality (AR)-based recognizer of flange specification (ARFS), to overcome these difficulties. This solution combines human spatial reasoning and computational intelligence in AR to distinguish multiple flanges of similar sizes in complex and uncertain on-site situations. We conducted evaluation experiments to design user interfaces for measuring key flange dimensions in AR. A usability test compares the measuring time and accuracy of the solution implemented on a handheld versus a head-mounted display device. Real-world testing confirms that deploying ARFS as a smartphone app provides an economic yet effective tool for smart asset management in the construction and infrastructure industries.

Multi-scale feature reconstruction network for industrial anomaly detection

Unsupervised anomaly detection techniques, which operate without prior knowledge of anomalies, have garnered significant attention in industrial inspection due to their adaptability and generalization. Therefore, knowledge-based computer vision techniques have been broadly applied to identify unusual image patterns. However, real-time industrial applications present challenges such as limited anomalous samples, inadequate defect knowledge, and complex background textures. These factors lead to difficulties in accurately identifying defect regions, and conventional auto-encoder networks often struggle to overcome these issues.To address these limitations, we propose a multi-scale feature reconstruction (MSFR) network specifically designed for domain shift scenarios. Our approach employs a pyramidal vision transformer network (PVTN) to reconstruct multi-scale feature maps, capturing discriminative features at various scales. Additionally, a pre-trained module extracts multi-level features at the same scale, and a dedicated feature matching module enhances accuracy by improving the alignment probability between features. The MSFR strategy surpasses conventional auto-encoders by filtering pixel-level information at multiple depths.Empirical evaluations were conducted using benchmark datasets such as MVTec AD and AeBAD-S. Furthermore, an extensive ablation study demonstrates the effectiveness and viability of the proposed MSFR approach for industrial anomaly detection tasks. The experimental results show that the proposed model significantly outperforms recent approaches, making it highly suitable for real-world industrial applications, particularly in manufacturing.

Deep-millet: a deep learning model for pearl millet disease identification to envisage precision agriculture

Abstract Plants are integral to human sustenance, serving as fundamental sources of sustenance, materials, and energy, crucial for economic prosperity. However, their productivity and yield are increasingly threatened by pests and diseases, exacerbated by shifting climatic conditions. Pearl millet, a vital crop in Africa and Asia, is particularly susceptible to a range of diseases including downy mildew, rust, ergot, smut, and blast, posing significant risks to crop yield and quality. Timely and accurate disease identification is paramount for effective management strategies. Traditional methods of disease detection relying on visual identification are laborious, costly, and often require specialized expertise, presenting formidable challenges for farmers. In this study, we propose a novel mobile application integrating a robust Deep Learning (DL) model for the automated identification of pearl millet leaf diseases, employing advanced computer vision techniques. A Convolutional Neural Network (CNN) architecture, named Deep Millet, was trained on a comprehensive dataset comprising 3441 field images depicting pearl millet leaves in both healthy and diseased states. It consists of fewer but more effective layers, which are optimized to extract the most pertinent features from the RGB images Comparative analysis against pre-trained models, including AlexNet, ResNet50, InceptionV3, Xception, NasNet mobile, VGG16, and VGG19, was conducted to evaluate the performance of the proposed model. Results demonstrate that Deep Millet achieved superior accuracy, completing training in a mere 240 s and yielding an impressive accuracy rating of 98.86%, surpassing current state-of-the-art models.

DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions

Fiducial markers are a computer vision tool used for object pose estimation and detection. These markers are highly useful in fields such as industry, medicine and logistics. However, optimal lighting conditions are not always available, and other factors such as blur or sensor noise can affect image quality. Classical computer vision techniques that precisely locate and decode fiducial markers often fail under difficult illumination conditions (e.g. extreme variations of lighting within the same frame). Hence, we propose DeepArUco++, a deep learning-based framework that leverages the robustness of Convolutional Neural Networks to perform marker detection and decoding in challenging lighting conditions. The framework is based on a pipeline using different Neural Network models at each step, namely marker detection, corner refinement and marker decoding. Additionally, we propose a simple method for generating synthetic data for training the different models that compose the proposed pipeline, and we present a second, real-life dataset of ArUco markers in challenging lighting conditions used to evaluate our system. The developed method outperforms other state-of-the-art methods in such tasks and remains competitive even when testing on the datasets used to develop those methods. Code available in GitHub: https://github.com/AVAuco/deeparuco/.

Advanced Computer Vision Techniques for Accurate Measurement in Unmanned Mobile Robots

Abstract For years, researchers have been studying computer vision, i.e. the ability of artificial intelligence (AI) systems to perceive and interpret visual data like humans. This study is gaining increasing attention as researchers aim to develop tools that automate visual tasks and replicate human visual awareness. However, the interpretation of images is very complex due to the vast amount of multi-resolution information they contain, making the development of AI technologies for visual recognition particularly challenging. This article provides an overview of digital image processing, highlighting the main concepts and introducing key algorithms. These methods are designed to capture, process, and interpret digital images and enable the extraction of important data from real-world environments. We conduct rigorous image processing tests and compare AI-driven recognition systems with human analysis. The results show that computer vision technology significantly outperforms human observation in terms of accuracy and consistency. These results highlight the potential of computer vision to revolutionize various industries by automating complex visual tasks and offer promising future applications in areas such as healthcare, security, and manufacturing. The paper provides valuable insights into current advances in digital image processing and the role of AI in improving visual recognition capabilities, paving the way for further innovation in this area.

A Computer Vision-based Architecture for Remote Physical Rehabilitation

The use of computer vision in healthcare is constantly growing and the application of these techniques in the context of physical rehabilitation can bring great benefits. In this work, a software architecture was proposed which, with the use of computer vision techniques, aims to assist in the treatment and remote diagnosis of patients undergoing physical rehabilitation. The architecture was developed to allow the system to be used on computers and mobile devices. In the proposed system, the user with a professional profile can register and prescribe exercises for their patients according to the treatment. Users with a patient profile can view and perform the exercises that were prescribed for them in the application, relying on the application’s help to visually assist them with proper execution. A field research and a qualitative assessment were carried out in order to verify the usability and effectiveness of the application from the users’ point of view, with a positive reception.

Mitigating Complex Cyber Threats: An Integrated Multimodal Deep Learning Framework for Enhanced Security

Abstract: The rapidly evolving landscape of cyber threats poses significant challenges to traditional cybersecurity methods, particularly in detecting and mitigating complex attacks that combine multiple data modalities. This article introduces a novel approach utilizing Multimodal Deep Learning (MDL) to enhance cybersecurity detection and prevention capabilities. By fusing Natural Language Processing (NLP) and computer vision techniques, our proposed MDL framework demonstrates superior performance in analyzing both textual and visual data associated with potential threats. Through a series of experiments and case studies, we show that this integrated approach significantly improves detection accuracy, reduces false positives, and exhibits remarkable adaptability to emerging attack vectors. Our results indicate a 37% increase in threat detection efficiency and a 42% reduction in false positives compared to traditional unimodal methods. Furthermore, we explore the potential applications of this technology in email security, social media monitoring, and advanced malware detection. This article not only contributes to the growing body of knowledge in AI-driven cybersecurity but also provides a roadmap for future developments, including the integration of audio analysis and the application of Explainable AI (XAI) to enhance trust and transparency in MDL-based security systems.

Clinical Significance of Pose Estimation Methods Compared with Radiographic Parameters in Adolescent Patients with Idiopathic Scoliosis.

Human pose estimation, a computer vision technique that identifies body parts and constructs human body representations from images and videos, has recently demonstrated high performance through deep learning. However, its potential application in clinical photography remains underexplored. This study aimed to establish photographic parameters for patients with adolescent idiopathic scoliosis (AIS) using pose estimation and to determine correlations between these photographic parameters and corresponding radiographic measures. We conducted a study involving 42 patients with AIS who had undergone spinal correction surgery and conservative treatment. Preoperative photographs were captured using an iPhone 13 Pro mounted on a tripod positioned at the head of an X-ray tube. From the outputs of pose estimation, we derived five photographic parameters and subsequently conducted a statistical analysis to assess their correlations with relevant conventional radiographic parameters. In the sagittal plane, we identified significant correlations between photographic and radiographic parameters measuring trunk tilt angles. In the coronal plane, significant correlations were found between photographic parameters measuring shoulder height and trunk tilt and corresponding radiographic measurements. The results suggest that pose estimation, achievable with common mobile devices, offers potential for AIS screening, early detection, and continuous posture monitoring, effectively mitigating the need for X-ray radiation exposure. Level of Evidence: 3.

Real-Time Table Tennis Scoring and Player Position Tracking System Using Computer Vision

This study focuses on developing a system that uses state-of-the-art computer vision techniques to improve the accuracy and efficiency of table tennis tracking. The table tennis ball's trajectory and movement are tracked automatically by the system using a mobile camera array, allowing for real-time score updates without the need for human participation. This method guarantees precise and seamless scoring, freeing players from worrying about maintaining score and allowing them to concentrate fully on the game. The system uses player pose estimation technologies in addition to real-time scoring to track and record player positions and motions during the match. The system may provide a thorough study of player motions, including tracking posture and finding important motion patterns, by gathering and analyzing skeletal data.

A dual-channel network based on occlusion feature compensation for human pose estimation

Human pose estimation is an important technique in computer vision. Existing methods perform well in ideal environments, but there is room for improvement in occluded environments. The specific reasons are that the ambiguity of the features in the occlusion area makes the network pay insufficient attention to it, and the inadequate expressive ability of the features in the occlusion part cannot describe the true keypoint features. To address the occlusion issue, we propose a dual-channel network based on occlusion feature compensation. The dual channels are occlusion area enhancement channel based on convolution and occlusion feature compensation channel based on graph convolution, respectively. In the convolution channel, we propose an occlusion handling enhanced attention mechanism (OHE-attention) to improve the attention to the occlusion area. In the graph convolution channel, we propose a node feature compensation module that eliminates the obstacle features and integrates the shared and private attributes of the keypoints to improve the expressive ability of the node features. We conduct experiments on the COCO2017 dataset, COCO-Wholebody dataset, and CrowdPose dataset, achieving accuracy of 78.7%, 66.4%, and 77.9%, respectively. In addition, a series of ablation experiments and visualization demonstrations verify the performance of the dual-channel network in occluded environments.

Performance Analysis of YOLO and Detectron2 Models for Detecting Corn and Soybean Pests Employing Customized Dataset

One of the most challenging aspects of agricultural pest control is accurate detection of insects in crops. Inadequate control measures for insect pests can seriously impact the production of corn and soybean plantations. In recent years, artificial intelligence (AI) algorithms have been extensively used for detecting insect pests in the field. In this line of research, this paper introduces a method to detect four key insect species that are predominant in Brazilian agriculture. Our model relies on computer vision techniques, including You Only Look Once (YOLO) and Detectron2, and adapts them to lightweight formats—TensorFlow Lite (TFLite) and Open Neural Network Exchange (ONNX)—for resource-constrained devices. Our method leverages two datasets: a comprehensive one and a smaller sample for comparison purposes. With this setup, the authors aimed at using these two datasets to evaluate the performance of the computer vision models and subsequently convert the best-performing models into TFLite and ONNX formats, facilitating their deployment on edge devices. The results are promising. Even in the worst-case scenario, where the ONNX model with the reduced dataset was compared to the YOLOv9-gelan model with the full dataset, the precision reached 87.3%, and the accuracy achieved was 95.0%.

Monocular 3D Multi-Person Pose Estimation for On-Site Joint Flexion Assessment: A Case of Extreme Knee Flexion Detection.

Work-related musculoskeletal disorders (WMSDs) represent a significant health challenge for workers in construction environments, often arising from prolonged exposure to ergonomic risks associated with manual labor, awkward postures, and repetitive motions. These conditions not only lead to diminished worker productivity but also incur substantial economic costs for employers and healthcare systems alike. Thus, there is an urgent need for effective tools to assess and mitigate these ergonomic risks. This study proposes a novel monocular 3D multi-person pose estimation method designed to enhance ergonomic risk assessments in construction environments. Leveraging advanced computer vision and deep learning techniques, this approach accurately captures and analyzes the spatial dynamics of workers' postures, with a focus on detecting extreme knee flexion, a critical indicator of work-related musculoskeletal disorders (WMSDs). A pilot study conducted on an actual construction site demonstrated the method's feasibility and effectiveness, achieving an accurate detection rate for extreme flexion incidents that closely aligned with supervisory observations and worker self-reports. The proposed monocular approach enables universal applicability and enhances ergonomic analysis through 3D pose estimation and group pose recognition for timely interventions. Future efforts will focus on improving robustness and integration with health monitoring to reduce WMSDs and promote worker health.

Seismic damage assessment of bonded versus unbonded laminated rubber bearings: A deep learning perspective

Laminated Rubber Bearings (LRBs), widely employed in highway bridges worldwide, are vulnerable to shear failure, squeezing or displacing during seismic events, potentially compromising bridge safety. Establishing a reliable approach for assessing the damage of LRBs is crucial for evaluating the seismic performance of bridges and preventing serious damage. To address this concern, this paper proposes a novel approach for LRBs seismic damage assessment based on computer vision (CV) technique. First, quasi-static loading tests are conducted on both bonded and unbonded LRBs to effectively enhance the understanding of their nonlinear properties. Subsequently, a large number of bearing samples are investigated to develop a comprehensive image database for a CV-based damage assessment model. A deep convolutional network (CNN) is designed to segment damage pixels, which are then combined with damage indicators to quantify the impact of various influencing factors on seismic damage using the interpretable machine learning technique, Shapley value. The results demonstrate that the CNN model achieves high-precision pixel segmentation and accurate predictions of seismic damage to LRBs, with RMSE consistently below 0.009 and R2 exceeding 95 %. The primary factor influencing LRB damage is the type of constraint, which interacts with loading characteristics and geometric dimensions to produce various coupled effects. This high-precision CV-based approach shows significant potential for estimating seismic damage states of critical bridge components and building seismic protection.