Computer Vision Research Articles

Advancing human pose estimation with transformer models: An experimental approach

This paper explores the integration of Transformer architectures into human pose estimation, a critical task in computer vision that involves detecting human figures and predicting their poses by identifying body joint positions. With applications ranging from enhancing interactive gaming experiences to advancing biomechanical analyses, human pose estimation demands high accuracy and flexibility, particularly in dynamic and partially occluded scenes. This study hypothesizes that Transformers, renowned for their ability to manage long-range dependencies and focus on relevant data parts through self-attention mechanisms, can significantly outperform existing deep learning methods such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). We introduce the PoseTransformer, a hybrid model that combines the precise feature extraction capabilities of CNNs with the global contextual awareness of Transformers, aiming to set new standards for accuracy and adaptability in pose estimation tasks. The model's effectiveness is demonstrated through rigorous testing on benchmark datasets, showing substantial improvements over traditional approaches, especially in complex scenarios.

Centralised visual processing center for remote sensing target detection

Target detection in satellite images is an essential topic in the field of remote sensing and computer vision. Despite extensive research efforts, accurate and efficient target detection in remote sensing images remains unsolved due to the large target scale span, dense distribution, and overhead imaging and complex backgrounds, which result in high target feature similarity and serious occlusion. In order to address the above issues in a comprehensive manner, within this paper, we first propose a Centralised Visual Processing Center (CVPC), this structure is a parallel visual processing center for Transformer encoder and CNN, employing a lightweight encoder to capture broad, long-range interdependencies. Pixel-level Learning Center (PLC) module is used to establish pixel-level correlations and improve the depiction of detailed features. CVPC effectively improves the detection efficiency of remote sensing targets with high feature similarity and severe occlusion. Secondly, we propose a centralised feature cross-layer fusion pyramid structure to fuse the results with the CVPC in a top-down manner to enhance the detailed feature representation capability at each layer. Ultimately, we present a Context Enhanced Adaptive Sparse Convolutional Network (CEASC), which improves the accuracy while ensuring the detection efficiency. Based on the above modules, we designed and conducted a series of experiments. These experiments are conducted on three challenging public datasets, DOTA-v1.0, DIOR, and RSDO, showing that our proposed 3CNet achieves a more advanced detection accuracy while balancing the detection speed (78.62% mAP for DOTA-v1.0, 79.12% mAP for DIOR, and 95.50% mAP for RSOD).

Streamlining electronic reporting of serious adverse events (SAEs) using the REDCap data collection system: the eSAE Project

BackgroundIt is essential that electronic data collection (EDC) systems are both compliant with regulations and the principles of Good Clinical Practice (GCP) to allow for the timely and accurate reporting of data including safety data. For clinical trials of investigational medicinal products (CTIMPs), investigators must immediately report to the sponsor any serious adverse event (SAE) that occurs in a site for which they are responsible. It is therefore expected that sponsors provide systems for timely review and reporting should a SAE be classified as a suspected unexpected serious adverse reaction (SUSAR). Challenges arise when data related to adverse events (AEs) needs to be re-entered for SAEs; this can be prone to error and may delay reporting. Additionally, recognising what has changed from an initial SAE report when an investigator responds to queries raised can cause errors.MethodA multi-disciplinary working group came together from a UK academic clinical trials unit (CTU) to establish if an electronic system could be created in the unit’s open-source EDC system—REDCap, to manage SAEs in an efficient way.ResultsA module has been created in REDCap to facilitate electronic SAE reporting: enabling an AE form to automatically trigger an SAE form for any AE which is also a SAE, prepopulating relevant fields of the SAE form, reducing the risk of delay and error when entering data into the SAE form. The system has also been developed with an embedded code to allow for instant visual recognition of any data updated following reporting to allow the sponsor to immediately review and resolve SAEs in a timely manner, complying with UK regulatory reporting. This functionality ‘The eSAE Project’ is now an active project for all of our new trials where data collection is undertaken using the REDCap system.ConclusionThe eSAE Project coded into REDCap offers a unique way of populating SAE forms with information already entered in the initial AE forms as applicable, coupled with highlighting any updates during the lifetime of the SAE for sponsors to identify any new information that needs to be reassessed to process and report the SAE.

Learning real-world heterogeneous noise models with a benchmark dataset

Noise modeling is an important research field in computer vision; the design of an accurate model for imaging sensor noise depends on not only a comprehensive benchmark dataset of the real world, but also a precise design of the noise modeling algorithm. However, due to the inaccurate estimation method of noise-free images and limited shooting scenes, the current realistic datasets could not describe the diverse noise properties sufficiently. Moreover, popular parametric noise models are not sophisticated enough to characterize the real-world noise exactly. In this work, we first construct a more comprehensive dataset of the real world by capturing more indoor and outdoor scenes under different lighting conditions using diverse smartphones, then we propose a non-parametric noise estimation method capable of modeling the spatial heterogeneity of real-world noise patterns. Specifically, in order to characterize the spatial heterogeneity of real-world noise, we assume a non-i.i.d Gaussian distribution and propose a deep convolutional neural network (DCNN)-based approach for learning pixel-wise noise variance maps. To learn the pixel-wise variance map, we have constructed a variance estimation network mapping from the conditional signals (clean image, ISO, and camera model) to surrogate labels obtained from the nonlocal search of similar patches from the clean-noisy image pair. Finally, we conducted denoising and classification experiments using different kinds of simulated noisy images, compared to the Poisson-Gaussian and Noise Flow noise models, the proposed method achieves denoising performance improvements (PSNR) of 1.13 dB and 2.51 dB respectively on the proposed real-world test dataset, denoising and classification results on the real noisy data captured by mobile phones have verified that our approach is more accurate than current noise modeling methods.

Dynamic coordinated air supply for moving individuals in industrial settings: Effectiveness evaluation and demonstration

Tracking occupant movement and reducing pollution exposure in industrial settings utilizing intelligent ventilation modes without human intervention is a challenge. This work proposes a dynamic coordinated air supply (DCAS) system incorporating static and dynamic ventilation modules, which is capable of switching air supply modes and adjusting air supply parameters in response to changing occupant positions with the assistance of computer vision technology and Internet of Things (IoT). Air movement and pollutant dispersion in an experimental chamber equipped with a DCAS system during occupant movement are analyzed through computational fluid dynamics with overset mesh method. The feasibility of the dynamic simulation method is verified by a walking experiment in the chamber. Three assessment indicators, namely, inhalation exposure, protection factor, and effective protective ratio, are identified to comprehensively evaluate the pollutant removal effectiveness of DCAS. Results show that the static ventilation module can remarkably reduce the occupant’s exposure risk within its protective range. The proposed sliding and rotating modes for the dynamic ventilation module are effective in reducing occupant inhalation exposure during movement if the advance operation strategies are specified to eliminate the negative effects of protection lag. In addition, the feasibility of the IoT architecture-based DCAS system is demonstrated in a schematic laboratory. This work helps drive industrial ventilation development toward smart, healthy purposes.

HDConv: Heterogeneous kernel-based dilated convolutions

Dilated convolution has been widely used in various computer vision tasks due to its ability to expand the receptive field while maintaining the resolution of feature maps. However, the critical challenge is the gridding problem caused by the isomorphic structure of the dilated convolution, where the holes filled in the dilated convolution destroy the integrity of the extracted information and cut off the relevance of neighboring pixels. In this work, a novel heterogeneous dilated convolution, called HDConv, is proposed to address this issue by setting independent dilation rates on grouped channels while keeping the general convolution operation. The heterogeneous structure can effectively avoid the gridding problem while introducing multi-scale kernels in the filters. Based on the heterogeneous structure of the proposed HDConv, we also explore the benefit of large receptive fields to feature extraction by comparing different combinations of dilated rates. Finally, a series of experiments are conducted to verify the effectiveness of some computer vision tasks, such as image segmentation and object detection. The results show the proposed HDConv can achieve a competitive performance on ADE20K, Cityscapes, COCO-Stuff10k, COCO, and a medical image dataset UESTC-COVID-19. The proposed module can readily replace conventional convolutions in existing convolutional neural networks (i.e., plug-and-play), and it is promising to further extend dilated convolution to wider scenarios in the field of image segmentation.

Real-Time Testing of AI Enabled Automatic Emergency Braking System for ADAS Vehicle using 3D Point cloud and Precise Depth Information

At the forefront of automobile safety technology, Automatic Emergency Braking (AEB) represents a major advancement in collision avoidance systems. The cutting-edge technology provides an additional layer of security at pivotal times, making it a vital part of the changing landscape of vehicle safety. This research introduces an effective system for automatic emergency braking in ADAS-equipped or Autonomous vehicles using a combination of 3d lidar and stereo vision camera for a swift and robust system in the vehicle that is faster than human drivers in cases of unexpected emergencies. Utilizing the power of clustering algorithms on 3d point clouds and state-of-the-art computer vision algorithms on an RGB image mapped to a depth frame from the stereo vision camera, the system as a whole provides a comprehensive system adding to the safety of the vehicle and the passengers. Further, the efficiency of the system is studied based on various parameters. The data from an external inertial measurement unit is also utilized to derive results that support the claims of the study. The system has been developed and implemented on a passenger car that has been modified into an electric vehicle and further tested in real-world traffic conditions in autonomous driving mode. The findings of the study were having exceptionally good precision in split-second decision-making in emergency maneuvers.

A Critical Review of Emerging Technologies for Flash Flood Prediction: Examining Artificial Intelligence, Machine Learning, Internet of Things, Cloud Computing, and Robotics Techniques

There has been growing interest in the application of smart technologies for hazard management. However, very limited studies have reviewed the trends of such technologies in the context of flash floods. This study reviews innovative technologies such as artificial intelligence (AI)/machine learning (ML), the Internet of Things (IoT), cloud computing, and robotics used for flash flood early warnings and susceptibility predictions. Articles published between 2010 and 2023 were manually collected from scientific databases such as Google Scholar, Scopus, and Web of Science. Based on the review, AI/ML has been applied to flash flood susceptibility and early warning prediction in 64% of the published papers, followed by the IoT (19%), cloud computing (6%), and robotics (2%). Among the most common AI/ML methods used in susceptibility and early warning predictions are random forests and support vector machines. However, further optimization and emerging technologies, such as computer vision, are required to improve these technologies. AI/ML algorithms have demonstrated very accurate prediction performance, with receiver operating characteristics (ROC) and areas under the curve (AUC) greater than 0.90. However, there is a need to improve on these current models with large test datasets. Through AI/ML, IoT, and cloud computing technologies, early warnings can be disseminated to targeted communities in real time via electronic media, such as SMS and social media platforms. In spite of this, these systems have issues with internet connectivity, as well as data loss. Additionally, Al/ML used a number of topographical variables (such as slope), geological variables (such as lithology), and hydrological variables (such as stream density) to predict susceptibility, but the selection of these variables lacks a clear theoretical basis and has inconsistencies. To generate more reliable flood risk assessment maps, future studies should also consider sociodemographic, health, and housing data. Considering future climate change impacts, susceptibility or early warning studies may be projected under different climate change scenarios to help design long-term adaptation strategies.

Leveraging camera traps and artificial intelligence to explore thermoregulation behaviour.

Behavioural thermoregulation has critical ecological and physiological consequences that profoundly influence individual fitness and species distributions, particularly in the context of climate change. However, field monitoring of this behaviour remains labour-intensive and time-consuming. With the rise of camera-based surveys and artificial intelligence (AI) approaches in computer vision, we should try to build better tools for characterizing animals' behavioural thermoregulation. In this study, we developed a deep learning framework to automate the detection and classification of thermoregulation behaviour. We used lizards, the Rough-tail rock agama (Laudakia vulgaris), as a model animal for thermoregulation. We colour-marked the lizards and curated a diverse dataset of images captured by trail cameras under semi-natural conditions. Subsequently, we trained an object-detection model to identify lizards and image classification models to determine their microclimate usage (activity in sun or shade), which may indicate thermoregulation preferences. We then evaluated the performance of each model and analysed how the classification of thermoregulating lizards performed under different solar conditions (sun or shade), times of day and marking colours. Our framework's models achieved high scores in several performance metrics. The behavioural thermoregulation classification model performed significantly better on sun-basking lizards, achieving the highest classification accuracy with white-marked lizards. Moreover, the hours of activity and the microclimate choices (sun vs shade-seeking behaviour) of lizards, generated by our framework, are closely aligned with manually annotated data. Our study underscores the potential of AI in effectively tracking behavioural thermoregulation, offering a promising new direction for camera trap studies. This approach can potentially reduce the labour and time associated with ecological data collection and analysis and help gain a deeper understanding of species' thermal preferences and risks of climate change on species behaviour.

Monitoring the lactation-related behaviors of sows and their piglets in farrowing crates using deep learning

Pig farming is a major sector of livestock production. The preweaning stage is a critical period in the pig farming process, where lactation-related behaviors between sows and their piglets directly influence the preweaning survivability of the piglets. Lactation-related behaviors are mutual interactions that require the combined monitoring of both the sow and her piglets. Conventional naked-eye observation is discontinuous and labor-intensive and may result in undetected abnormal behavior and economic losses. Thus, this study proposed to monitor the lactation-related behaviors of sows and their piglets simultaneously and continuously using computer vision. Videos were recorded from farrowing crates using embedded systems equipped with regular RGB cameras. The sow posture recognition model (SPRM), comprising a convolutional neural network (CNN) of the architecture EfficientNet and a long short-term memory network, was trained to identify seven postures of sows. The piglet localization and tracking model (PLTM), comprising a CNN of the architecture YOLOv7 and a simple online and realtime tracking algorithm, was trained to localize and track piglets in the farrowing crate. The sow posture information was then combined with the piglet activity to detect unfed piglets. The trained SPRM and PLTM reached an accuracy of 91.36% and a multiple object tracking accuracy of 94.6%. The performance of the proposed unfed piglet detection achieved a precision of 98.4% and a recall of 90.7%. A long-term experiment was conducted to monitor lactation-related behaviors of sows and their piglets from the birth of the piglets to day 15. The overall mean daily percentages ± standard deviations (SDs) of sow postures were 6.8% ± 2.9% for feeding, 8.8% ± 6.6% for standing, 11.8% ± 4.5% for sitting, 20.6% ± 16.3% for recumbency, 14.1% ± 6.5% for lying, and 38.1% ± 7.5% for lactating. The overall mean daily percentages ± SDs of piglet activities were 38.1% ± 7.5% for suckling, 22.2% ± 5.4% for active, and 39.7% ± 10.5% for rest. The proposed approach provides a total solution for the automatic monitoring of sows and their piglets in the farrowing house. This automatic detection of abnormal lactation-related behaviors can help in preventing piglet preweaning mortality and therefore aid pig farming efficiency.

Application of Gaussian Model and Deep Learning Encoder-Decoder Algorithm for Single-Image Reflection Removal

Images of target scenes shot through clear, reflective materials like glass are frequently interfered by unwanted reflection scenes which often overlaid on top of the targeted scenes. This, has constantly degrades the quality of the captured images and affects their subsequent analyses. While cognitively, distinguishing a recognizable object from its reflection in a picture is not difficult for humans, it is highly difficult and more complex in computer vision due to the ill-posed nature of the problem. In this research an enhanced single-image reflection removal model was developed by combining Gaussian filter and deep learning encoder-decoder for effective performance. While the Gaussian filter denoises the reflection-contaminated image, the encoder-decoder network learns the features of the image to produce reflection-free image. The proposed network is an end-to-end trained network with three losses. The experimental findings showed that the proposed model out-performed several state-of-the-art methods both qualitatively and quantitatively on five different datasets.

SMART BABY MONITORING SYSTEM USING YOLO V8 ALGORITHM

The Smart Baby Monitoring System using the YOLO V8 algorithm is designed to enhance infant monitoring by leveraging advanced computer vision techniques. This project utilizes YOLO (You Only Look Once) version 8, a state-of-the-art object detection algorithm, implemented with Python and frameworks like Tensor Flow or PyTorch, to detect and track objects in real-time video feeds. The system incorporates features for facial recognition to identify known caregivers and alert mechanisms for unusual activities or emergencies. The user interface provides real-time alerts, visualizations, and historical data analysis for caregivers via a web or mobile application. By leveraging YOLO V8's efficiency in object detection and Python's capabilities for data processing and integration, this system aims to enhance safety, improve care giving efficiency, and provide peace of mind to parents and caregivers.

Lane and Object Detection using YOLO: Indian Roads Scenario

This research explores the feasibility of You Only Look Once (YOLO), a deep learning object detection algorithm, for lane and object detection in challenging Indian road environments. Traditional methods struggle with faded lane markings, dense and diverse traffic, and unpredictable scenarios. YOLO's speed and accuracy make it suitable for real-time ADAS applications. The methodology leverages Roboflow, a platform for computer vision tasks, to explore data acquisition, model selection, training, and evaluation. This research aims to contribute to developing safer and more reliable ADAS systems for Indian roads.

EXCOR membrane motion analyzer (EMMA) to quantify and assess hemodynamic performance of the EXCOR pediatric heart assist device.

Pediatric heart failure is associated with high mortality rates and is a current clinical burden. There is only one FDA approved pediatric VAD, Berlin Heart EXCOR, for treatment. Thrombo-embolic complications are a significant clinical challenge, which can lead to devastating complications such as stroke and impair efficient EXCOR function. Currently, clinicians perform largely qualitative periodic assessment of EXCOR operation by observing the motion of a rapidly moving membrane, which can be prone to human error and can lead to missing out on crucial information. In this study, we design and implement a quantitative early warning system for accurate and quantitative assessment of the EXCOR membrane, named EXCOR Membrane Motion Analyzer (EMMA). Using a combination of image analysis, computer vision and custom designed algorithm, we perform rigorous frame by frame analysis of EXCOR membrane video data. We developed specialized metrics to identify relative smoothness between successive peaks, time between peaks and overall smoothness indicators to quantify and compare between multiple cases. Our results demonstrate that EMMA can successfully identify the motion and wrinkles on each video frame and quantify the smoothness and identify the phases of each cardiac cycle. Moreover, EMMA can obtain the smoothness of each frame and the temporal evolution of membrane smoothness across all image frames for the video sequence. EMMA allows for a fast, accurate, quantitative assessment to be completed and reduces user error. This enables EMMA to be used effectively as an early warning system to rapidly identify device abnormalities.

Advancements and Integration in Computer Vision and Graphics Systems

Advancements and Integration in Computer Vision and Graphics Systems

TTFDNet: Precise Depth Estimation from Single-Frame Fringe Patterns

This work presents TTFDNet, a transformer-based and transfer learning network for end-to-end depth estimation from single-frame fringe patterns in fringe projection profilometry. TTFDNet features a precise contour and coarse depth (PCCD) pre-processor, a global multi-dimensional fusion (GMDF) module and a progressive depth extractor (PDE). It utilizes transfer learning through fringe structure consistency evaluation (FSCE) to leverage the transformer’s benefits even on a small dataset. Tested on 208 scenes, the model achieved a mean absolute error (MAE) of 0.00372 mm, outperforming Unet (0.03458 mm) models, PDE (0.01063 mm) and PCTNet (0.00518 mm). It demonstrated precise measurement capabilities with deviations of ~90 μm for a 25.4 mm radius ball and ~6 μm for a 20 mm thick metal part. Additionally, TTFDNet showed excellent generalization and robustness in dynamic reconstruction and varied imaging conditions, making it appropriate for practical applications in manufacturing, automation and computer vision.

Artificial intelligence model for automated surgical instrument detection and counting: an experimental proof-of-concept study

BackgroundRetained surgical items (RSI) are preventable events that pose a significant risk to patient safety. Current strategies for preventing RSIs rely heavily on manual instrument counting methods, which are prone to human error. This study evaluates the feasibility and performance of a deep learning-based computer vision model for automated surgical tool detection and counting.MethodsA novel dataset of 1,004 images containing 13,213 surgical tools across 11 categories was developed. The dataset was split into training, validation, and test sets at a 60:20:20 ratio. An artificial intelligence (AI) model was trained on the dataset, and the model’s performance was evaluated using standard object detection metrics, including precision and recall. To simulate a real-world surgical setting, model performance was also evaluated in a dynamic surgical video of instruments being moved in real-time.ResultsThe model demonstrated high precision (98.5%) and recall (99.9%) in distinguishing surgical tools from the background. It also exhibited excellent performance in differentiating between various surgical tools, with precision ranging from 94.0 to 100% and recall ranging from 97.1 to 100% across 11 tool categories. The model maintained strong performance on a subset of test images containing overlapping tools (precision range: 89.6–100%, and recall range 97.2–98.2%). In a real-time surgical video analysis, the model maintained a correct surgical tool count in all non-transition frames, with a median inference speed of 40.4 frames per second (interquartile range: 4.9).ConclusionThis study demonstrates that using a deep learning-based computer vision model for automated surgical tool detection and counting is feasible. The model’s high precision and real-time inference capabilities highlight its potential to serve as an AI safeguard to potentially improve patient safety and reduce manual burden on surgical staff. Further validation in clinical settings is warranted.

Boosting Sports Card Sales: Leveraging Visual Display and Machine Learning in Online Retail

Trading cards are a fast-growing industry. However, previous research in sports merchandise has largely overlooked the role of cards’ visual appeal in online-commerce. This study addresses this gap by analyzing over 7000 samples from a leading sports card trading platform. Using computer vision algorithms (Mask R–CNN) and a machine learning algorithm (CatBoost), we unveil the importance of 12 image display attributes and their relationship with the card premium rate. Moreover, we identify inverted U-shaped relationships with attributes such as warm hue, saturation, and brightness. The findings offer valuable insights for card dealers to enhance product image display effectiveness.

Pig Weight Estimation Method Based on a Framework Combining Mask R-CNN and Ensemble Regression Model

Using computer vision technology to estimate pig live weight is an important method to realize pig welfare. But there are two key issues that affect pigs’ weight estimation: one is the uneven illumination, which leads to unclear contour extraction of pigs, and the other is the bending of the pig body, which leads to incorrect pig body information. For the first one, Mask R-CNN was used to extract the contour of the pig, and the obtained mask image was converted into a binary image from which we were able to obtain a more accurate contour image. For the second one, the body length, hip width and the distance from the camera to the pig back were corrected by XGBoost and actual measured information. Then we analyzed the rationality of the extracted features. Three feature combination strategies were used to predict pig weight. In total, 1505 back images of 39 pigs obtained using Azure kinect DK were used in the numerical experiments. The highest prediction accuracy is XGBoost, with an MAE of 0.389, RMSE of 0.576, MAPE of 0.318% and R2 of 0.995. We also recommend using the Mask R-CNN + RFR method because it has fairly high precision in each strategy. The experimental results show that our proposed method has excellent performance in live weight estimation of pigs.

A computer vision approach to vehicle detection, classification, and tracking from UAV data for Indian traffic analysis

Traffic surveillance is crucial for road safety and efficiency. This study examines object detection techniques tailored for Indian traffic, highlighting the challenges faced. The aim is to track vehicles, pedestrians, and cyclists using drone-captured data in Indian cities. Urban areas in India grapple with traffic problems due to rapid growth and inadequate infrastructure. Effective monitoring can address these. Recent advancements in object detection, a critical computer vision task, show promise for this application. Employing the YOLOv8 model, trained on our drone-collected dataset, which improved detection accuracy for the Indian Traffic Scenario, is shown. This dataset labelling of vehicle bounding boxes underwent preprocessing using Gaussian Filter, resizing, normalization, and augmentation. Testing showcased the model's real-world applicability in areas like traffic management and autonomous driving. The proposed model thus enhances vehicle detection systems, fostering better safety and decision-making, yielding promising results, as evidenced by 0.86 mAP50 after training and testing the model under own UAV Dataset.