Computer Vision Tools Research Articles

Dynamic modulation of social gaze by sex and familiarity in marmoset dyads.

Social communication relies on the ability to perceive and interpret the direction of others' attention, and is commonly conveyed through head orientation and gaze direction in humans and nonhuman primates. However, traditional social gaze experiments in nonhuman primates require restraining head movements, significantly limiting their natural behavioral repertoire. Here, we developed a novel framework for accurately tracking facial features and three-dimensional head gaze orientations of multiple freely moving common marmosets ( Callithrix jacchus ). By combining deep learning-based computer vision tools with triangulation algorithms, we were able to track the facial features of marmoset dyads within an arena. This method effectively generates dynamic 3D geometrical facial frames while overcoming common challenges like occlusion. To detect the head gaze direction, we constructed a virtual cone, oriented perpendicular to the facial frame. Using this pipeline, we quantified different types of interactive social gaze events, including partner-directed gaze and joint gaze to a shared spatial location. We observed clear effects of sex and familiarity on both interpersonal distance and gaze dynamics in marmoset dyads. Unfamiliar pairs exhibited more stereotyped patterns of arena occupancy, more sustained levels of social gaze across social distance, and increased social gaze monitoring. On the other hand, familiar pairs exhibited higher levels of joint gazes. Moreover, males displayed significantly elevated levels of gazes toward females' faces and the surrounding regions, irrespective of familiarity. Our study reveals the importance of two key social factors in driving the gaze behaviors of a prosocial primate species and lays the groundwork for a rigorous quantification of primate behaviors in naturalistic settings.

Automated identification of earthen berms in Western US rangelands from LiDAR‐based digital elevation models

AbstractEarthworks such as earthen berms have been constructed across the western US since the late 1800s to mitigate erosion in landscapes where water is both the dominant driver of erosion and the limiting resource for biota. Berms alter hydrologic, geomorphic and ecologic processes by intercepting runoff and altering patterns of water availability in the landscape. Understanding site‐specific changes in process dynamics requires accurate mapping of berm locations and knowledge of their condition. This paper presents an automated, object‐based framework for identifying earthen berms from 1 m LiDAR‐derived digital elevation models in the western US rangelands. Geomorphon, a computer vision tool, was used to classify landforms and identify berm‐like landforms, including summits and ridges. Ten geomorphic and geometric attributes associated with each potential berm object were used to develop a machine‐learning model for distinguishing berms from natural summits and ridges. The model was trained and applied to independent test sites to identify and map berms. The mapped berms were compared with manually identified reference berms for accuracy assessment. The identification results achieved 79% to 87% recall, 82% to 92% precision and 81% to 89% F‐measure. We also explored the influence of training sample selection on model performance and conducted an analysis of attribute relative importance. The automated framework has the potential to be scaled up to larger areas in semi‐arid environments.

DeepArUco++: Improved detection of square fiducial markers in challenging lighting conditions

Fiducial markers are a computer vision tool used for object pose estimation and detection. These markers are highly useful in fields such as industry, medicine and logistics. However, optimal lighting conditions are not always available, and other factors such as blur or sensor noise can affect image quality. Classical computer vision techniques that precisely locate and decode fiducial markers often fail under difficult illumination conditions (e.g. extreme variations of lighting within the same frame). Hence, we propose DeepArUco++, a deep learning-based framework that leverages the robustness of Convolutional Neural Networks to perform marker detection and decoding in challenging lighting conditions. The framework is based on a pipeline using different Neural Network models at each step, namely marker detection, corner refinement and marker decoding. Additionally, we propose a simple method for generating synthetic data for training the different models that compose the proposed pipeline, and we present a second, real-life dataset of ArUco markers in challenging lighting conditions used to evaluate our system. The developed method outperforms other state-of-the-art methods in such tasks and remains competitive even when testing on the datasets used to develop those methods. Code available in GitHub: https://github.com/AVAuco/deeparuco/.

Deep Hair Phenomics: Implications in Endocrinology, Development, and Aging

Hair quality is an important indicator of health in humans and other animals. Current approaches to assess hair quality are generally non-quantitative or are low throughput due to technical limitations of ‘splitting hairs’. We developed a deep learning-based computer vision approach for the high throughput quantification of individual hair fibers at a high resolution. Our innovative computer vision tool can distinguish and extract overlapping fibers for quantification of multivariate features including length, width, and color to generate single-hair phenomes (shPhenome) of diverse conditions across the lifespan of mice. Using our tool, we explored the effects of hormone signaling, genetic modifications, and aging on hair follicle output. Our analyses revealed hair phenotypes resultant of endocrinological, developmental, and aging-related alterations in the fur coats of mice. These results demonstrate the efficacy of our deep hair phenomics tool for characterizing factors that modulate the hair follicle and developing new diagnostic methods for detecting disease through the hair fiber. Finally, we have generated a searchable, interactive web tool for the exploration of our hair fiber data at skinregeneration.org.

An automated lightweight approach for detecting dead fish in a recirculating aquaculture system

Accurate methods for detecting dead fish are of great significance for fisheries because of their potential to improve production and reduce water pollution. Small-scale fisheries typically apply manual observation, but this approach is labor-intensive and time-consuming, which limits its application on large-scale farms. Advances in computer vision tools have provided the foundation for revolutionary methods of automated identification of individual-animal behavior. However, computer vision systems often suffer from unstable accuracy, low efficiency, and limited detection capability. Thus, the aim of this study was to design a deep learning system, designated “Deadfish-YOLO”, to detect dead fish based solely on standard underwater camera images with no additional hardware. First, eight selected rearing tanks were monitored by a self-developed computer version system. From these tank images, a dataset containing 18,114 frames was built by manual labeling. Next, a lightweight backbone network was generated with YOLOv4 to ensure fast computations, and an attention mechanism was introduced into the model to suppress unimportant features. Finally, the ReLU-memristor-like activation function was adopted to improve neural-network performance. The accuracy and processing speed of Deadfish-YOLO were superior to those of other state-of-the-art single- and two-stage detection models; Deadfish-YOLO ran at 85 frames per second with a mean average precision of 0.946 and an average ratio of 0.924 between intersection and union. These results demonstrate that Deadfish-YOLO may be used to automatically monitor dead fish in real circulating aquaculture systems. In addition, the results of this study should facilitate the wider application of artificial-intelligence-based animal monitoring in aquaculture.

Benchmarking analysis of computer vision algorithms on edge devices for the real-time detection of digital dermatitis in dairy cows

Digital dermatitis (DD) is a bovine claw disease responsible for ulcerative lesions on the coronary band of the foot. It causes significant animal welfare and economic losses to the cattle industry. Early detection of DD can lead to prompt treatment and decrease lameness. Current detection and staging methods require a trained individual to evaluate the interdigital space on each foot for clinical signs of DD. Computer vision (CV), a type of artificial intelligence for image analysis, has demonstrated promising results on object detection tasks. However, farms require robust solutions that can be deployed in harsh conditions including dust, debris, humidity, precipitation, other equipment issues. The study aims to train, deploy, and benchmark DD detection models on edge devices. Images were collected from commercial dairy farms with the camera facing the interdigital space on the plantar surface of the foot. Images were scored for M-stages of DD by a trained investigator using the M-stage DD classification system. Models were trained to detect and score DD lesions and embedded on an edge device. The Tiny YOLOv4 model deployed on a CV specific integrated camera module connected to a single board computer achieved a mean average precision (mAP) of 0.895, an overall prediction accuracy of 0.873, and a Cohen’s kappa of 0.830 for agreement between the computer vision model and the trained investigator. The model reached a final inference speed of 40 frames per second (FPS) and ran stably without any interruptions. The CV model was able to detect DD lesions on an edge device with high performance and speed. The CV tool can be used for early detection and prompt treatment of DD in dairy cows. Real-time detection of DD on edge device will improve health outcomes, while simultaneously decreasing labor costs. We demonstrate that the deployed model can be a low-power and portable solution for real-time detection of DD on dairy farms. This result is a step towards applying CV algorithms to veterinary medicine and implementing real-time detection of health outcomes in precision farming.

IoT-Integrated Smart Attendance and Attention Monitoring System For Primary and Secondary School Classroom Management

The monitoring of student attendance is a crucial aspect of the assessment of academic performance. The conventional methods for monitoring student attendance have inherent limitations in terms of both time efficiency and accuracy. Consequently, there is a clear need for a more expedient and precise attendance system. The objective of this research is to present the design of a real-time attendance recording and monitoring system for students from elementary school to senior high school, which will be implemented using the concept of the Internet of Things (IoT). The proposed system employs biometric technology in the form of face recognition. The methodology commences with the capture of images of students who leave the classroom during the instructional period via an active camera positioned on the classroom door. The system employs a Convolutional Neural Network (CNN) algorithm and a powerful computer vision tool, OpenCV, to perform real-time face recognition. Teachers will be informed of student absences and returns, as well as at the 15th and 30th minutes. An absence exceeding 30 minutes is classified as truancy. The integration of sophisticated technologies, such as machine learning and image processing, not only enhances the precision of attendance records but also equips educators with an efficient and automated system for streamlining classroom attendance management. This not only optimizes the learning environment but also facilitates more advanced and efficient pedagogical practices.

Computer vision tools for early post-disaster assessment: Enhancing generalizability

Remote sensing data, particularly satellite imagery, have made early, post-hazard aerial damage assessment possible due to its fast availability and extensive coverage. Despite breakthroughs in using deep computer vision with satellite image inputs, achieving high generalizability across diverse hazards and locations remains the main obstacle to effectively deploying early assessment tools in real-world scenarios, as the same hazard can manifest differently across various landscapes and urban textures. This challenge may be overlooked when working with curated datasets with minimal (test-to-train) shifts in urban textures and hazard damage features (e.g., due to undersized study regions), leaving models ill-prepared for real-world scenarios. The primary objective of this study was to understand non-trivial generalizability challenges by taking the 2023 Turkiye earthquake and the Maui wildfire incidents as “training” and “unseen test” events that simulated a demanding scenario. Subsequently, strategies such as augmenting image channels with damage proxy maps, data fusion, deep ensemble learning, and Test Time Augmentation were exclusively designed and implemented to address those challenges. These measures significantly improved the damage detection, with or without severity classification, with F1 scores increased from 0.71 to 0.82 and 0.40 to 0.87, respectively. Furthermore, and through data fusion, the proposed framework accommodates estimating socioeconomic loss metrics at the individual building level, supporting both response and recovery phases. This research has the potential to enhance the effectiveness of rapid aerial damage assessment models, ultimately aiding in more efficient and targeted disaster response and recovery efforts. Data, models, and codes are available at https://github.com/TRG-AI4Good/Lahaina_Generalizability.

A toolkit for the dynamic study of air sacs in siamang and other elastic circular structures.

Biological structures are defined by rigid elements, such as bones, and elastic elements, like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments provide insights into complex time-varying dynamics of biological motion. Conversely, the elastic soft-tissues of organisms, like the nose of elephant seals, or the buccal sac of frogs, are poorly studied and no computer vision methods have been proposed. This leaves major gaps in different areas of biology. In primatology, most critically, the function of air sacs is widely debated; many open questions on the role of air sacs in the evolution of animal communication, including human speech, remain unanswered. To support the dynamic study of soft-tissue structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit contains unsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting DeepLabCut) methodology to track inflation of laryngeal air sacs or other biological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis, we show that air sac inflation correlates with acoustic markers that likely inform about body size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of siamang (Symphalangus syndactylus) singing. This toolkit (https://github.com/WimPouw/AirSacTracker) aims to revitalize the study of non-skeletal morphological structures across multiple species.

Advancing Artistic Swimming Officiating and Performance Assessment: A Computer Vision Study Using MediaPipe

Abstract Artistic Swimming (AS) requires complete execution and synchronization of movements for performance evaluation. The interest in objective and subjective performance analysis worldwide in sports via valid and reliable Artificial Intelligence (AI) tools is spreading depending on the required analysis parameters to design a novel system. This study investigated a novel application of the MediaPipe-based computer vision tool validation by examining biomechanical aspects and the objective performance impact in ballet leg and barracuda AS techniques. Twenty experienced AS athletes participated and executed these techniques under controlled conditions. Thirty-six recorded video trials were captured and analyzed via computer vision using MediaPipe, Kinovea, and AutoCAD (gold standard), with correlations calculated to assess the reliability of measurements and tools. The results indicated a non-significant difference (p<0.05) among the software tools, supported by one-way ANOVA and Bland-Altman tests. Notably, in ballet leg technique, maintaining alignment between the upper body trunk and knee in a line had a small correlation with other leg deviations; however, this aspect had a moderate negative correlation in scoring. Overall, this study suggests MediaPipe efficiency in computer vision for AS officiating and performance analysis, offering a reliable, real-time alternative to traditional methods and providing perceptions of AS techniques.

Artificial Neural Networks for Mapping Coastal Lagoon of Chilika Lake, India, Using Earth Observation Data

This study presents the environmental mapping of the Chilika Lake coastal lagoon, India, using satellite images Landsat 8-9 OLI/TIRS processed using machine learning (ML) methods. The largest brackish water coastal lagoon in Asia, Chilika Lake, is a wetland of international importance included in the Ramsar site due to its rich biodiversity, productivity, and precious habitat for migrating birds and rare species. The vulnerable ecosystems of the Chilika Lagoon are subject to climate effects (monsoon effects) and anthropogenic activities (overexploitation through fishing and pollution by microplastics). Such environmental pressure results in the eutrophication of the lake, coastal erosion, fluctuations in size, and changes in land cover types in the surrounding landscapes. The habitat monitoring of the coastal lagoons is complex and difficult to implement with conventional Geographic Information System (GIS) methods. In particular, landscape variability, patch fragmentation, and landscape dynamics play a crucial role in environmental dynamics along the eastern coasts of the Bay of Bengal, which is strongly affected by the Indian monsoon system, which controls the precipitation pattern and ecosystem structure. To improve methods of environmental monitoring of coastal areas, this study employs the methods of ML and Artificial Neural Networks (ANNs), which present a powerful tool for computer vision, image classification, and analysis of Earth Observation (EO) data. Multispectral satellite data were processed by several ML image classification methods, including Random Forest (RF), Support Vector Machine (SVM), and the ANN-based MultiLayer Perceptron (MLP) Classifier. The results are compared and discussed. The ANN-based approach outperformed the other methods in terms of accuracy and precision of mapping. Ten land cover classes around the Chilika coastal lagoon were identified via spatio-temporal variations in land cover types from 2019 until 2024. This study provides ML-based maps implemented using Geographic Resources Analysis Support System (GRASS) GIS image analysis software and aims to support ML-based mapping approach of environmental processes over the Chilika Lake coastal lagoon, India.

Unleashing the power of Convolutional Neural Networks in license plate recognition and beyond

This article explores the application of Convolutional Neural Networks (CNN) in the field of license plate recognition. It begins by introducing the architecture of CNN, which consists of three key layers: Convolutional Layers, Pooling Layers, and Fully Connected Layers. The article then references three relevant papers that demonstrate how CNNs are applied in license plate recognition. The first paper utilizes TensorFlow to construct a CNN model and integrates it with an STM32MP157 embedded chip for license plate recognition. The second paper presents a real-time car license plate detection and recognition method called Multi-Task Light CNN, emphasizing robustness. The third paper employs the ResNet+FPN feature extraction network of the Mask R-CNN model and annotates a license plate dataset. The article highlights the promising future of CNNs in various fields beyond license plate recognition, emphasizing their potential for further development and industrial applications. CNNs have proven to be versatile and powerful tools in computer vision, offering solutions to a wide range of problems. Their adaptability and effectiveness make them a key player in the ongoing advancement of artificial intelligence and automation technologies.

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

The ApeTI dataset was built with the aim of retrieving physiological signals such as heart rate, breath rate, and cognitive load from thermal images of great apes. We want to develop computer vision tools that psychologists and animal behavior researchers can use to retrieve physiological signals noninvasively. Our goal is to increase the use of a thermal imaging modality in the community and avoid using more invasive recording methods to answer research questions. The first step to retrieving physiological signals from thermal imaging is their spatial segmentation to then analyze the time series of the regions of interest. For this purpose, we present a thermal imaging dataset based on recordings of chimpanzees with their face and nose annotated using a bounding box and nine landmarks. The face and landmarks’ locations can then be used to extract physiological signals. The dataset was acquired using a thermal camera at the Leipzig Zoo. Juice was provided in the vicinity of the camera to encourage the chimpanzee to approach and have a good view of the face. Several computer vision methods are presented and evaluated on this dataset. We reach mAPs of 0.74 for face detection and 0.98 for landmark estimation using our proposed combination of the Tifa and Tina models inspired by the HRNet models. A proof of concept of the model is presented for physiological signal retrieval but requires further investigation to be evaluated. The dataset and the implementation of the Tina and Tifa models are available to the scientific community for performance comparison or further applications.

Scalable semantic 3D mapping of coral reefs with deep learning

Abstract Coral reefs are among the most diverse ecosystems on our planet, and essential to the livelihood of hundreds of millions of people who depend on them for food security, income from tourism and coastal protection. Unfortunately, most coral reefs are existentially threatened by global climate change and local anthropogenic pressures. To better understand the dynamics underlying deterioration of reefs, monitoring at high spatial and temporal resolution is key. However, conventional monitoring methods for quantifying coral cover and species abundance are limited in scale due to the extensive manual labor required. Although computer vision tools have been employed to aid in this process, in particular structure‐from‐motion (SfM) photogrammetry for 3D mapping and deep neural networks for image segmentation, analysis of the data products creates a bottleneck, effectively limiting their scalability. This paper presents a new paradigm for mapping underwater environments from ego‐motion video, unifying 3D mapping systems that use machine learning to adapt to challenging conditions under water, combined with a modern approach for semantic segmentation of images. The method is exemplified on coral reefs in the northern Gulf of Aqaba, Red Sea, demonstrating high‐precision 3D semantic mapping at unprecedented scale with significantly reduced required labor costs: given a trained model, a 100 m video transect acquired within 5 min of diving with a cheap consumer‐grade camera can be fully automatically transformed into a semantic point cloud within 5 min. We demonstrate the spatial accuracy of our method and the semantic segmentation performance (of at least 80% total accuracy), and publish a large dataset of ego‐motion videos from the northern Gulf of Aqaba, along with a dataset of video frames annotated for dense semantic segmentation of benthic classes. Our approach significantly scales up coral reef monitoring by taking a leap towards fully automatic analysis of video transects. The method advances coral reef transects by reducing the labor, equipment, logistics, and computing cost. This can help to inform conservation policies more efficiently. The underlying computational method of learning‐based Structure‐from‐Motion has broad implications for fast low‐cost mapping of underwater environments other than coral reefs.

Novel computer vision tools applied to marine recreational fisheries spatial planning

Successful marine spatial planning relies on understanding patterns of human use, with accurate, detailed, and up-to-date information about the spatial distribution of fishing effort. In commercial vessels, tracking systems like the vessel monitoring system (VMS) or the automatic identification system (AIS) have helped to maintain and enhance the biodiversity of areas by generating large sources of positional data that served for commercial marine spatial planning. However, there is no regulation regarding location systems such as VMS or AIS for marine recreational fishing boats. Obtaining spatial data on marine recreational fishing can be difficult and time-intensive given the widespread and variable nature of the fleet. Remote cameras and computer vision systems are increasingly used to overcome the cost limitations of these conventional methods. Here we show a novel high-resolution and low-cost tracking system based on photo time-lapses and state-of-the-art computer vision algorithms, including deep learning, to automatically classify and obtain precise trajectories of fishing and cruising boats in coastal areas. Our method contributes to the automatic surveillance of marine protected areas by providing an image-based tool for automatic, real-time monitoring. Our method also allows for determining the intensity and spatial-temporal distribution of recreational fishing effort, important to defining the sustainability of the activity and coastal areas. We finally discuss the opportunities and limitations of computer vision tools applied to marine recreational fisheries spatial planning.

Method of comparing and transforming images obtained using UAV

The subject matter of this article involves reviewing and developing methods for the comparison and transformation of images obtained using UAV via Computer Vision tools. The goal is to improve methods for image comparison and transformation. Various image-processing methods were employed to achieve the goal of this study,thereby contributing to the development of practical algorithms and approaches for image analysis and comparison. The tasks can be described as follows: 1) development of image comparison methods: design tools for the comparison of images from UAV that efficiently detect differences using algorithms such as cv2.absdiff and the PIL module; 2) Image transformation: implement transformation methods for images from UAV, including perspective transformation and thresholding, to enhance the quality and accuracy of image analysis. The methods used were algorithm development, image transformation methods, statistical analysis, experimental testing, and performance evaluation. The metrics used in this article are response time and accuracy. Algorithms for image comparison have also been refined, particularly those transformed through Global Threshold Value, Adaptive Mean Thresholding, and Adaptive Gaussian Thresholding. A novel change filtering method was introduced to enhance the precision of image comparison by filtering out insignificant alterations following image transformation. Comprehensive investigation of image comparison involving edge detection methods has been systematically presented. The results contain the development of practical algorithms and approaches for image analysis and comparison applicable in diverse areas such as military, security, and agriculture. Possibilities of applying our methods and algorithms in the context of drones were also considered, which is particularly relevant in tasks related to computer vision in unmanned aerial vehicles, where limited resources and the need for real-time processing of a large volume of data create unique challenges. Conclusions. The results contain OpenCV and PIL image comparison methods. OpenCV pixel-by-pixel comparison algorithm showed a better response time with the same accuracy. OpenCV method has 92,46% response time improvement compared with PIL and is 276ms. As for image thresholding with comparison, a method based on Global Threshold Value showed the shortest response time (266ms) and the lowest accuracy. The highest accuracy and response time (366ms) were obtained using the Adaptive Gaussian Thresholding method.

Sizing mudsnails: Applying superpixels to scale growth detection under ocean warming

Abstract The expansion of scientific image data holds great promise to quantify individuals, size distributions and traits. Computer vision tools are especially powerful to automate data mining of images and thus have been applied widely across studies in aquatic and terrestrial ecology. Yet marine benthic communities, especially infauna, remain understudied despite their dominance of marine biomass, biodiversity and playing critical roles in ecosystem functioning. Here, we disaggregated infauna from sediment cores taken throughout the spring transition (April–June) from a near‐natural mesocosm setup under experimental warming (Ambient, +1.5°C, +3.0°C). Numerically abundant mudsnails were imaged in batches under stereomicroscopy, from which we automatically counted and sized individuals using a superpixel‐based segmentation algorithm. Our segmentation approach was based on clustering superpixels, which naturally partition images by low‐level properties (e.g., colour, shape and edges) and allow instance‐based segmentation to extract all individuals from each image. We demonstrate high accuracy and precision for counting and sizing individuals, through a procedure that is robust to the number of individuals per image (5–65) and to size ranges spanning an order of magnitude (<750 μm to 7.4 mm). The segmentation routine provided at least a fivefold increase in efficiency compared with manual measurements. Scaling this approach to a larger dataset tallied >40k individuals and revealed overall growth in response to springtime warming. We illustrate that image processing and segmentation workflows can be built upon existing open‐access R packages, underlining the potential for wider adoption of computer vision tools among ecologists. The image‐based approach also generated reproducible data products that, alongside our scripts, we have made freely available. This work reinforces the need for next‐generation monitoring of benthic communities, especially infauna, which can display differential responses to average warming.

Deep-Learning-Based Segmentation of Keyhole in In-Situ X-ray Imaging of Laser Powder Bed Fusion.

In laser powder bed fusion processes, keyholes are the gaseous cavities formed where laser interacts with metal, and their morphologies play an important role in defect formation and the final product quality. The in-situ X-ray imaging technique can monitor the keyhole dynamics from the side and capture keyhole shapes in the X-ray image stream. Keyhole shapes in X-ray images are then often labeled by humans for analysis, which increasingly involves attempting to correlate keyhole shapes with defects using machine learning. However, such labeling is tedious, time-consuming, error-prone, and cannot be scaled to large data sets. To use keyhole shapes more readily as the input to machine learning methods, an automatic tool to identify keyhole regions is desirable. In this paper, a deep-learning-based computer vision tool that can automatically segment keyhole shapes out of X-ray images is presented. The pipeline contains a filtering method and an implementation of the BASNet deep learning model to semantically segment the keyhole morphologies out of X-ray images. The presented tool shows promising average accuracy of 91.24% for keyhole area, and 92.81% for boundary shape, for a range of test dataset conditions in Al6061 (and one AliSi10Mg) alloys, with 300 training images/labels and 100 testing images for each trial. Prospective users may apply the presently trained tool or a retrained version following the approach used here to automatically label keyhole shapes in large image sets.

Deep Attentional Guided Image Filtering.

Guided filter is a fundamental tool in computer vision and computer graphics, which aims to transfer structure information from the guide image to the target image. Most existing methods construct filter kernels from the guidance itself without considering the mutual dependency between the guidance and the target. However, since there typically exist significantly different edges in two images, simply transferring all structural information from the guide to the target would result in various artifacts. To cope with this problem, we propose an effective framework named deep attentional guided image filtering, the filtering process of which can fully integrate the complementary information contained in both images. Specifically, we propose an attentional kernel learning module to generate dual sets of filter kernels from the guidance and the target and then adaptively combine them by modeling the pixelwise dependency between the two images. Meanwhile, we propose a multiscale guided image filtering module to progressively generate the filtering result with the constructed kernels in a coarse-to-fine manner. Correspondingly, a multiscale fusion strategy is introduced to reuse the intermediate results in the coarse-to-fine process. Extensive experiments show that the proposed framework compares favorably with the state-of-the-art methods in a wide range of guided image filtering applications, such as guided super-resolution (SR), cross-modality restoration, and semantic segmentation. Moreover, our scheme achieved the first place in the real depth map SR challenge held in ACM ICMR 2021. The codes can be found at https://github.com/zhwzhong/DAGF.

Rapid-Motion-Track: Markerless tracking of fast human motion with deep learning

Human movement patterns reflect central nervous system function. Small deficits in repetitive fast movements, such as slightly slowed finger-tapping or mildly irregular rhythm of stepping, are often an early sign of a neurological disorder. Accessible tools that precisely measure the individual components of fast movements would thus enhance disease detection, monitoring and research. Deep learning-based computer vision methods applied to digital video-recordings hold promise but current state-of-the-art tools, including DeepLabCut (DLC) and other advanced models, fail to accurately track the fastest range of human movements, primarily due to image blur. To solve this, we developed a new end-to-end, Rapid-Motion-Track (RMT) computer vision tool. This study aimed to evaluate the accuracy of RMT compared to DLC and other advanced computer vision tools. 220 finger-tapping tests were performed at frequencies between 0.5Hz and 6Hz and recorded simultaneously with a standard 30 frames/sec 2D laptop camera and a high-speed 250 frames/sec 3D motion tracking system (ground-truth). Bland-Altman plots and paired Welch's t-test were used to quantify the validity of movement features extracted by computer vision methods with the ground-truth. The movement features extracted by RMT (e.g. frequency, speed, variance) exhibited high concurrent validity across all tapping-frequencies. RMT outperformed other computer vision methods for very fast movements >4Hz. RMT also robustly tracked other fast motions including sit-to-stand, head-turning, foot-tapping, and leg agility. This new tool provides an accurate method to precisely and automatically measure even the fastest and finest human movements. It holds potential of wide reach as digital cameras are ubiquitous in homes, clinics and research centres.