Computer Vision Research Articles

Editor’s Note: Special Issue on Advances in Visual Computing

Editor’s Note: Special Issue on Advances in Visual Computing

A computer vision based real-time warpage monitoring and detection system in fused deposition modeling

A computer vision based real-time warpage monitoring and detection system in fused deposition modeling

Cultural big data: nineteenth to twenty-first century panoramic visualization

From the nineteenth-century panorama to the emergence of the digital panoramic format in the 1990's, the visualization of large images frequently relies on panoramic viewing strategies. Originally rendered in the form of epic painted canvases, these strategies are now amplified through gigapixel imaging, computer vision and machine learning. Whether for scientific analysis, dissemination, or to visualize cultural big data, panoramic strategies pivot on the illusion of immersion. The latter is achieved through human-centered design situated within a large-scale environment combined with a multi-sensory experience spanning sight, sound, touch, and smell. In this article, we present the original research undertaken to realize a digital twin of the 1894 panorama of the battle of Murten. Following a brief history of the panorama, the methods and technological framework systems developed for Murten panorama's visualization are delineated. Novel visualization methodologies are further discussed, including how to create the illusion of immersion for the world's largest image of a single physical object and its cultural big data. We also present the visualization strategies developed for the augmentation of the layered narratives and histories embedded in the final interactive viewing experience of the Murten panorama. This article offers researchers in heritage big data new schemas for the visualization and augmentation of gigapixel images in digital panoramas.

Ensemble Pretrained Convolutional Neural Networks for the Classification of Insulator Surface Conditions

Overhead transmission line insulators are non-conductive materials that separate conductors from grounded transmission towers. Once in operation, they frequently experience environmental pollution and electrical or mechanical stress. Since adverse operational conditions can lead to insulation failure, regular inspections are essential to prevent power outages. To this end, this paper proposes a novel technique based on deep convolutional neural networks (CNNs) to classify high-voltage insulator surface conditions based on their image. Successful applications of CNNs in computer vision have led to several pretrained architectures for image classification. To use these pretrained models, a practitioner typically fine-tunes and selects one final model via a model selection stage and discards all other models. In contrast with many existing studies that use such a “winner-takes-all” approach, here, we identify the best subset of seven popular pretrained CNN architectures that are combined by soft voting to form an ensemble classifier. From a machine learning (ML) perspective, this focus is warranted because the convolutional base of each pretrained architecture operates as a feature extractor and an ensemble of them works as a combination of various feature extraction rules. Our numerical experiments demonstrate the advantage of the identified ensemble model to individual pretrained architectures.

Attention-enhanced multimodal feature fusion network for clothes-changing person re-identification

Clothes-Changing Person Re-Identification is a challenging problem in computer vision, primarily due to the appearance variations caused by clothing changes across different camera views. This poses significant challenges to traditional person re-identification techniques that rely on clothing features. These challenges include the inconsistency of clothing and the difficulty in learning reliable clothing-irrelevant local features. To address this issue, we propose a novel network architecture called the Attention-Enhanced Multimodal Feature Fusion Network (AE-Net). AE-Net effectively mitigates the impact of clothing changes on recognition accuracy by integrating RGB global features, grayscale image features, and clothing-irrelevant features obtained through semantic segmentation. Specifically, global features capture the overall appearance of the person; grayscale image features help eliminate the interference of color in recognition; and clothing-irrelevant features derived from semantic segmentation enforce the model to learn features independent of the person’s clothing. Additionally, we introduce a multi-scale fusion attention mechanism that further enhances the model’s ability to capture both detailed and global structures, thereby improving recognition accuracy and robustness. Extensive experimental results demonstrate that AE-Net outperforms several state-of-the-art methods on the PRCC and LTCC datasets, particularly in scenarios with significant clothing changes. On the PRCC and LTCC datasets, AE-Net achieves Top-1 accuracy rates of 60.4% and 42.9%, respectively.

A New Way to Identify Mastitis in Cows Using Artificial Intelligence

Mastitis is a disease that is considered an obstacle in dairy farming. Some methods of diagnosing mastitis have been used effectively over the years, but with an associated relative cost that reduces the producer’s profit. In this context, this sector needs tools that offer an early, safe, and non-invasive diagnosis and that direct the producer to apply resources to confirm the clinical picture, minimizing the cost of monitoring the herd. The objective of this study was to develop a predictive methodology based on sequential knowledge transfer for the automatic detection of bovine subclinical mastitis using computer vision. The image bank used in this research consisted of 165 images, each with a resolution of 360 × 360 pixels, sourced from a database of 55 animals diagnosed with subclinical mastitis, all of which were not exhibiting clinical symptoms at the time of imaging. The images utilized in the sequential learning transfer were those of MammoTherm, which is used for the detection of breast cancer in women. The optimized model demonstrated the most optimal network performance, achieving 92.1% accuracy, in comparison to the model with manual search (86.1%). The proposed predictive methodologies, based on knowledge transfer, were effective in accurately classifying the images. This significantly enhanced the automatic detection of both healthy animals and those diagnosed with subclinical mastitis using thermal images of the udders of dairy cows.

Near real‐time monitoring of wading birds using uncrewed aircraft systems and computer vision

Wildlife population monitoring over large geographic areas is increasingly feasible due to developments in aerial survey methods coupled with the use of computer vision models for identifying and classifying individual organisms. However, aerial surveys still occur infrequently, and there are often long delays between the acquisition of airborne imagery and its conversion into population monitoring data. Near real‐time monitoring is increasingly important for active management decisions and ecological forecasting. Accomplishing this over large scales requires a combination of airborne imagery, computer vision models to process imagery into information on individual organisms, and automated workflows to ensure that imagery is quickly processed into data following acquisition. Here we present our end‐to‐end workflow for conducting near real‐time monitoring of wading birds in the Everglades, Florida, USA. Imagery is acquired as frequently as weekly using uncrewed aircraft systems (aka drones), processed into orthomosaics (using Agisoft metashape), converted into individual‐level species data using a Retinanet‐50 object detector, post‐processed, archived, and presented on a web‐based visualization platform (using Shiny). The main components of the workflow are automated using Snakemake. The underlying computer vision model provides accurate object detection, species classification, and both total and species‐level counts for five out of six target species (White Ibis, Great Egret, Great Blue Heron, Wood Stork, and Roseate Spoonbill). The model performed poorly for Snowy Egrets due to the small number of labels and difficulty distinguishing them from White Ibis (the most abundant species). By automating the post‐survey processing, data on the populations of these species is available in near real‐time (<1 week from the date of the survey) providing information at the time scales needed for ecological forecasting and active management.

An Efficient Dense Reconstruction Algorithm from LiDAR and Monocular Camera

Dense reconstruction have been studied for decades in the fields of computer vision and robotics, in which LiDAR and camera are widely used. However, vision-based methods are sensitive to illumination variation and lack direct depth, and LiDAR-based methods are limited by sparse LiDAR measurement and lacking color and texture information. In this paper, we propose a novel 3D reconstruction algorithm based on LiDAR and a monocular camera, which realizes dense reconstruction. In the algorithm, a LiDAR odometry is used to get accurate poses and poses calculated by the odometry module are used in the calculation of depth maps and fusion of depth maps, and then mesh and texture mapping are implemented. In addition, a semantic segmentation network and a depth completion network are used to obtain dense and accurate depth maps. The concept of symmetry is utilized to generate 3D models of objects or scenes; that is, the reconstruction and camera imaging of these objects or scenes are symmetrical. Experimental results on public dataset show that the proposed algorithm achieves higher accuracy, efficiency and completeness than existing methods.

Research on Target Hybrid Recognition and Localization Methods Based on an Industrial Camera and a Depth Camera in Complex Scenes

In order to improve the target visual recognition and localization accuracy of robotic arms in complex scenes with similar targets, hybrid recognition and localization methods based on an industrial camera and depth camera are proposed. First, according to the speed and accuracy requirements of target recognition and localization, YOLOv5s is introduced as the basic algorithm model for target hybrid recognition and localization. Then, in order to improve the accuracy of target recognition and coarse localization based on an industrial camera (eye-to-hand), the AFPN feature fusion module, simple and parameter-free attention module (SimAM), and soft non-maximum suppression (Soft NMS) are introduced. In order to improve the accuracy of target recognition and fine localization based on a depth camera (eye-in-hand), the SENetV2 backbone network structure, dynamic head module, deformable attention mechanism, and chain-of-thought prompted adaptive enhancer network are introduced. After that, on the basis of constructing a dual camera platform for target hybrid recognition and localization, the hand–eye calibration, collection and production of image datasets required for model training are completed. Finally, for the docking of the oil filling port, the hybrid recognition and localization experimental tests are completed in sequence. The test results show that in target recognition and coarse localization based on the industrial camera, the recognition accuracy of the designed model reaches 99%, and the average localization errors in the horizontal and vertical directions are 2.22 mm and 3.66 mm, respectively. In target recognition and fine localization based on the depth camera, the recognition accuracy of the designed model reaches 98%, and the average errors in depth, horizontal, and vertical directions are 0.12 mm, 0.28 mm, and 0.16 mm, respectively. These not only verify the effectiveness of the target hybrid recognition and localization methods based on dual cameras, but also demonstrate that they meet the high-precision recognition and localization requirements in complex scenes.

Attention is All They Need: Exploring the Media Archaeology of the Computer Vision Research Paper

Research papers, in addition to textual documents, are a designed interface through which researchers communicate. Recently, rapid growth has transformed that interface in many fields of computing. In this work, we examine the effects of this growth from a media archaeology perspective, through the changes to figures and tables in research papers. Specifically, we study these changes in computer vision over the past decade, as the deep learning revolution has driven unprecedented growth in the discipline. We ground our investigation through interviews with veteran researchers spanning computer vision, graphics, and visualization. Our analysis focuses on the research attention economy: how research paper elements contribute towards advertising, measuring, and disseminating an increasingly commodified "contribution." Through this work, we seek to motivate future discussion surrounding the design of both the research paper itself as well as the larger sociotechnical research publishing system, including tools for finding, reading, and writing research papers.

Advances in Emerging Non-Destructive Technologies for Detecting Raw Egg Freshness: A Comprehensive Review

Eggs are a rich food source of proteins, fats, vitamins, minerals, and other nutrients. However, the egg industry faces some challenges such as microbial invasion due to environmental factors, leading to damage and reduced usability. Therefore, detecting the freshness of raw eggs using various technologies, including traditional and non-destructive methods, can overcome these challenges. As the traditional methods of assessing egg freshness are often subjective and time-consuming, modern non-destructive technologies, including near-infrared (NIR) spectroscopy, Raman spectroscopy, fluorescence spectroscopy, computer vision (color imaging), hyperspectral imaging, electronic noses, and nuclear magnetic resonance, have offered objective and rapid results to address these limitations. The current review summarizes and discusses the recent advances and developments in applying non-destructive technologies for detecting raw egg freshness. Some of these technologies such as NIR spectroscopy, computer vision, and hyperspectral imaging have achieved an accuracy of more than 96% in detecting egg freshness. Therefore, this review provides an overview of the current trends in the state-of-the-art non-destructive technologies recently utilized in detecting the freshness of raw eggs. This review can contribute significantly to the field of emerging technologies in this research track and pique the interests of both food scientists and industry professionals.

A review of deep learning-based stereo vision techniques for phenotype feature and behavioral analysis of fish in aquaculture

The industrialization, high-density, and greener aquaculture requires a more precise and intelligent aquaculture management. Phenotypic and behavioral information of fish, which can reflect fish growth and welfare status, play a crucial role in aquaculture management. Stereo vision technology, which simulates parallax perception of the human eye, can obtain the three-dimensional phenotypic characteristics and movement trajectories of fish through different types of sensors. It can overcome the limitations in dealing with fish deformation, frequent occlusions and understanding three-dimension scenes compared to the traditional two-dimensional computer vision techniques. With the deep learning development and application in aquaculture, stereo vision has become a super computer vision technology that can provide more precise and interpretable information for intelligent aquaculture management, such as size estimation, counting and behavioral analysis of fish. Hence, it is very beneficial for researchers, managers, and entrepreneurs to possess a thorough comprehension about the fast-developing stereo vision technology for modern aquaculture. This study provides a critical review of relevant topics, including the four-layer application structure of stereo vision technology in aquaculture, various deep learning-based technologies used, and specific application scenarios. The review contributes to research development by identifying the current challenges and provide valuable suggestions for future research directions. This review can serve as a useful resource for developing future studies and applications of stereo vision technology in smart aquaculture, focusing on phenotype feature extraction and behavioral analysis of fish.

GETr: A Geometric Equivariant Transformer for Point Cloud Registration

AbstractAs a fundamental problem in computer vision, 3D point cloud registration (PCR) aims to seek the optimal transformation to align point cloud pairs. Meanwhile, the equivariance lies at the core of matching point clouds at arbitrary pose. In this paper, we propose GETr, a geometric equivariant transformer for PCR. By learning the point‐wise orientations, we decouple the coordinate to the pose of the point clouds, which is the key to achieve equivariance in our framework. Then we utilize attention mechanism to learn the geometric features for superpoints matching, the proposed novel self‐attention mechanism encodes the geometric information of point clouds. Finally, the coarse‐to‐fine manner is used to obtain high‐quality correspondence for registration. Extensive experiments on both indoor and outdoor benchmarks demonstrate that our method outperforms various existing state‐of‐the‐art methods.

Unveiling Urban River Visual Features Through Immersive Virtual Reality: Analyzing Youth Perceptions with UAV Panoramic Imagery

The visual evaluation and characteristic analysis of urban rivers are pivotal for advancing our understanding of urban waterscapes and their surrounding environments. Unmanned aerial vehicles (UAVs) offer significant advantages over traditional satellite remote sensing, including flexible aerial surveying, diverse perspectives, and high-resolution imagery. This study centers on the Haihe River, South Canal, and North Canal in Tianjin China, employing UAVs to capture continuous panoramic image data. Through immersive virtual reality (VR) technology, visual evaluations of these panoramic images were obtained from a cohort of young participants. These evaluations encompassed assessments of scenic beauty, color richness, vitality, and historical sense. Subsequently, computer vision techniques were utilized to quantitatively analyze the proportions of various landscape elements (e.g., trees, grass, buildings) within the images. Clustering analysis of visual evaluation results and semantic segmentation outcomes from different study points facilitated the effective identification and grouping of river visual features. The findings reveal significant differences in scenic beauty, color richness, and vitality among the Haihe River, South Canal, and North Canal, whereas the South and North Canals exhibited a limited sense of history. Six landscape elements—water bodies, buildings, trees, etc.—comprised over 90% of the images, forming the primary visual characteristics of the three rivers. Nonetheless, the uneven spatial distribution of these elements resulted in notable variations in the visual features of the rivers. This study demonstrates that the visual feature analysis method based on UAV panoramic images can achieve a quantitative evaluation of multi-scene urban 3D landscapes, thereby providing a robust scientific foundation for the optimization of urban river environments.

FusionGCN: Multi-focus image fusion using superpixel features generation GCN and pixel-level feature reconstruction CNN

In recent years, convolutional neural networks have demonstrated significant advancements in the domain of computer vision, effectively addressing numerous previously challenging issues. An increasing number of researchers are focusing their investigations on this field, proposing innovative network architectures. However, many existing networks necessitate intricate module designs and a substantial number of parameters to achieve satisfactory fusion outcomes, which poses challenges for lightweight devices with constrained computational resources. To mitigate this concern, the present study introduces a novel methodology that integrates block segmentation with pixel optimization. Specifically, we initially employ graph convolutional networks to execute flexible convolutions on large-scale, irregular regions generated through superpixel clustering, thereby achieving coarse segmentation at the block level. Subsequently, we utilize parallel lightweight convolutional networks to provide pixel-level guidance, ultimately resulting in a more accurate decision map. Furthermore, to leverage the strengths of both networks and facilitate the optimization of feature generation from the graph convolutional network for non-Euclidean data, we meticulously design a superpixel-based graph decoder alongside a pixel-based convolutional neural network extraction block to enhance feature acquisition and propagation. In comparison to numerous state-of-the-art methodologies, our approach demonstrates commendable competitiveness in both qualitative and quantitative analyses, as well as in efficiency evaluations. The code can be downloaded at https://github.com/ouyangbaicai/FusionGCN.

Artificial intelligence correctly classifies developmental stages of monarch caterpillars enabling better conservation through the use of community science photographs

Rapid technological advances and growing participation from amateur naturalists have made countless images of insects in their natural habitats available on global web portals. Despite advances in automated species identification, traits like developmental stage or health remain underexplored or manually annotated, with limited focus on automating these features. As a proof-of-concept, we developed a computer vision model utilizing the YOLOv5 algorithm to accurately detect monarch butterfly caterpillars in photographs and classify them into their five developmental stages (instars). The training data were obtained from the iNaturalist portal, and the photographs were first classified and annotated by experts to allow supervised training of models. Our best trained model demonstrates excellent performance on object detection, achieving a mean average precision score of 95% across all five instars. In terms of classification, the YOLOv5l version yielded the best performance, reaching 87% instar classification accuracy for all classes in the test set. Our approach and model show promise in developing detection and classification models for developmental stages for insects, a resource that can be used for large-scale mechanistic studies. These photos hold valuable untapped information, and we’ve released our annotated collection as an open dataset to support replication and expansion of our methods.

Constructing a Classification Scheme - and its Consequences: A Field Study of Learning to Label Data for Computer Vision in a Hospital Intensive Care Unit

Research on data annotation for artificial intelligence (AI) has demonstrated that biases, power, and culture impact the ways that annotators apply labels to data and subsequently affect downstream AI systems. However, annotators can only apply labels that are available to them in the annotation classification scheme. Drawing on a 3-year ethnographic study of an R&D collaboration between medical and AI researchers, we argue that the construction of the classification schema itself -- decisions about what kinds of data can and cannot be collected, what activities can and cannot be detected in the data, what the possible annotation classes ought to be, and the rules by which an item ought to be classified into each class -- dramatically shape the annotation process, and through it, the AI. We draw on Bowker and Star's [9] classification theory to detail how the creation of a training data codebook for a computer vision algorithm in hospital intensive care units (ICUs) evolved from its original, clinically-driven goal of classifying complex clinical activities into a narrower goal of identifying physical objects and simpler activities in the ICU. This work reinforces how trade-offs and decisions made long before annotators begin labeling data are highly consequential to the resulting AI system.

Enhancing Video Anomaly Detection with Improved UNET and Cascade Sliding Window Technique

Computer vision video anomaly detection still needs to be improved, especially when identifying images with unusual motions or objects. Current approaches mainly concentrate on reconstruction and prediction methods, and unsupervised video anomaly detection faces difficulties because there are not enough tagged abnormalities, which reduces accuracy. This paper presents a novel framework called the Improved UNET (I-UNET), designed to counteract overfitting by addressing the need for complex models that can extract subtle information from video anomalies. Video frame noise can be eliminated by preprocessing the frames with a Weiner filter. Moreover, the system uses Convolution Long Short-Term Memory (ConvLSTM) layers to smoothly integrate temporal and spatial data into its encoder and decoder portions, improving the accuracy of anomaly identification. The Cascade Sliding Window Technique (CSWT) is used post-processing to identify anomalous frames and generate anomaly scores. Compared to baseline approaches, experimental results on the UCF, UCSDped1, and UCSDped2 datasets demonstrate notable performance gains, with 99% accuracy, 90.8% Area Under Curve (AUC), and 10.9% Equal Error Rate (EER). This study provides a robust and accurate framework for video anomaly detection with the highest accuracy rate.

Optimizing LLM Strategies for Playing Mendikot using Prompt Engineering

This study investigates the integration of a large language model (LLM) enhanced by prompt engineering and game theory to effectively engage in the strategic card game Mendikot. By refining complex prompts and leveraging a tailored visual understanding of game dynamics, we significantly bolster the decision-making prowess of the LLM. Our methodology involved the systematic simplification of game prompts to facilitate deeper learning and faster response times, coupled with the implementation of a visual recognition system to interpret and react to game states dynamically. The results illustrate that the adapted LLM outperforms traditional AI approaches in strategic decision-making tasks, underscoring a substantial improvement in both the accuracy and efficiency of game-play. This research not only demonstrates a viable model for enhancing AI interaction in recreational gaming but also opens avenues for deploying advanced AI strategies in complex strategic environments, offering insights into the broader application of AI in leisure and competitive arenas. The findings suggest that AI can transcend conventional gaming roles, potentially transforming strategic gameplay in digital and physical platforms.

Vision transformers for glioma classification using T1 magnetic resonance imaging

Automated image analysis and classification have increasingly advanced in recent decades owing to machine learning and computer vision. In particular, deep learning (DL) architectures have become popular in resource-limited and labor-restricted environments such as the health-care sector. Transformer architecture, a DL method with self-attention mechanism, excels in natural language processing; however, its application in image-based diagnosis in health-care sector remains limited. Herein, the feasibility, bottlenecks, and performance of transformers in magnetic resonance imaging (MRI)-based brain tumor classification were investigated. To this end, a vision transformer (ViT) model was trained and tested using the popular Brain Tumor Segmentation (BraTS) 2015 dataset for glioma classification. Owing to limited data availability, domain adaptation techniques were used to pretrain the ViT model and the BraTS 2015 dataset was used for its fine-tuning. With the model only trained for 100 epochs, the confusion matrix for the two-class problem of tumor and nontumor classification showed an overall classification accuracy of 81.8%. In conclusion, although convolutional neural networks are traditionally used for DL-based medical image classification owing to their attention mechanism and long-range dependency-capturing capability, ViTs can outperform them in MRI-based brain tumor classification.