Computer Vision Approach Research Articles

Aesthetics of hotel photos and its impact on consumer engagement: A computer vision approach

Visual content has been an essential marketing approach to exerting enjoyable virtual experiences and inducing consumer engagement. However, despite the widespread acknowledgment of the importance of visual content, how would the exerted aesthetic perception affect consumer engagement is still unexplored. Adopting a deep convolutional neural network model, this study quantifies the aesthetics of thumbnail images and managerial photos of hotels and thereby explores the impact of hotel photo aesthetics on consumer engagement. Aesthetically enjoyable photos are shown effective not only to encourage more consumers to engage in word-of-mouth discussions but also to promote consumer ratings. Such impact is also significantly moderated by hotel price. This study illustrates a viable approach to probing the perceived aesthetics of visual content in the hospitality field, and the uncovered significant role of aesthetics highlights the necessity of further attention to the cognitive perceptions of visual marketing content in both theoretical research and practical management.

Assessment of CNN-Based Models for Odometry Estimation Methods with LiDAR

The problem of simultaneous localization and mapping (SLAM) in mobile robotics currently remains a crucial issue to ensure the safety of autonomous vehicles’ navigation. One approach addressing the SLAM problem and odometry estimation has been through perception sensors, leading to V-SLAM and visual odometry solutions. Furthermore, for these purposes, computer vision approaches are quite widespread, but LiDAR is a more reliable technology for obstacles detection and its application could be broadened. However, in most cases, definitive results are not achieved, or they suffer from a high computational load that limits their operation in real time. Deep Learning techniques have proven their validity in many different fields, one of them being the perception of the environment of autonomous vehicles. This paper proposes an approach to address the estimation of the ego-vehicle positioning from 3D LiDAR data, taking advantage of the capabilities of a system based on Machine Learning models, analyzing possible limitations. Models have been used with two real datasets. Results provide the conclusion that CNN-based odometry could guarantee local consistency, whereas it loses accuracy due to cumulative errors in the evaluation of the global trajectory, so global consistency is not guaranteed.

K-Means Clustering으로 분류한 닭 깃털색 표현형의 분석

Chickens are a species of vertebrate with varying colors. Various colors of chickens must be classified to find color-related genes. In the past, color scoring was performed based on human visual observation. Therefore, chicken colors have not been measured with precise standards. In order to solve this problem, a computer vision approach was used in this study. Image quantization based on k-means clustering for all pixels of RGB values can objectively distinguish inherited colors that are expressed in various ways. This study was also conducted to determine whether plumage color differences exist in the reciprocal cross lines between two breeds: black Yeonsan Ogye (YO) and White Leghorn (WL). Line B is a crossbred line between YO males and WL females while Line L is a reciprocal crossbred line between WL males and YO females. One male and ten females were selected for each F₁ line, and full-sib mating was conducted to generate 883 F₂ birds. The results indicate that the distribution of light and dark colors of k-means clustering converged to 7:3. Additionally, the color of Line B was lighter than that of Line L (<i>P</i>&lt;0.01). This study suggests that the genes underlying plumage colors can be identified using quantification values from the computer vision approach described in this study.

A Combined Computer Vision and Deep Learning Approach for Rapid Drone-Based Optical Characterization of Parabolic Troughs

Abstract Optical accuracy is a primary driver of parabolic trough concentrating solar power (CSP) plant performance, but can be damaged by wind loads, gravity, error during installation, and regular plant operation. Collecting and analyzing optical measurements over an entire operating parabolic trough plant is difficult, given the large scale of typical installations. Distant Observer, a software tool developed at the National Renewable Energy Laboratory, uses images of the absorber tube reflected in the collector mirror to measure both surface slope in the parabolic mirror and offset of the absorber tube from the ideal focal point. This technology has been adapted for fast data collection using low-cost commercial drones, but until recently still required substantial human labor to process large amounts of data. A new method leveraging advanced deep learning and computer vision tools can drastically reduce the time required to process images. This new method addresses the primary analysis bottleneck, identifying featureless, reflective mirror corner points to a high degree of accuracy. Recent work has shown promising results using computer vision methods. The combined deep learning and computer vision approach presented here proved highly effective and has the potential to further automate data collection and analysis, making the tool more robust. The method presented in this paper automatically identified 74.3% of mirror corners within 2 pixels of their manually marked counterparts and 91.9% within 3 pixels. This level of accuracy is sufficient for practical Distant Observer analysis within a target uncertainty. A commercial drone collected video of over 100 parabolic trough modules at an operating CSP plant to demonstrate the deep learning and computer vision method’s usefulness in processing large amounts of data. These troughs were successfully analyzed using Distant Observer, paired with the new deep learning and computer vision algorithm, and can provide plant operators and trough designers with valuable insight about plant performance, operating strategies, and plant-wide optical error trends.

Dynamic identification of crane load fall zone: A computer vision approach

Crane activities pose significant safety risks on construction sites. Thus, numerous studies have applied computer vision techniques to identify crane load fall zones or areas where the crane load could fall during an accident. However, past studies typically assumed that the fall zone directly below the load was static and had a fixed size. In reality, the fall zone is dynamic because its size changes with different crane load types and heights. Thus, this study proposes and validates a vision-based method that dynamically identifies the fall zone directly below the load. The proposed dynamic fall zone detection and tracking method first detects the key points of the load, and then the fall zone is estimated by projecting the area of the lifted load to the construction floor. Once the fall zone is detected and tracked, coupled with a previously established approach for worker detection and tracking, workers entering the fall zone can be automatically detected. Data were collected from a public housing project in Singapore to train and evaluate the proposed method. The proposed method achieved precision and recall of 0.86 and 0.83, respectively. The main contribution of this study is the development of a novel method to detect and track the crane load fall zone. Besides providing real-time safety alerts to crane operators and site supervisors, the proposed method can accumulate statistics on unsafe behaviour and situations to facilitate safety management.

Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: Killing two birds with one stone?

Background and objectiveThe videofluoroscopic swallowing study (VFSS) is a gold-standard imaging technique for assessing swallowing, but analysis and rating of VFSS recordings is time consuming and requires specialized training and expertise. Researchers have recently demonstrated that it is possible to automatically detect the pharyngeal phase of swallowing and to localize the bolus in VFSS recordings via computer vision approaches, fostering the development of novel techniques for automatic VFSS analysis. However, training of algorithms to perform these tasks requires large amounts of annotated data that are seldom available. In this paper, we demonstrate that the challenges of pharyngeal phase detection and bolus localization can be solved together using a single approach. MethodsWe propose a deep-learning framework that jointly tackles pharyngeal phase detection and bolus localization in a weakly-supervised manner, requiring only the initial and final frames of the pharyngeal phase as ground truth annotations for the training. Our approach stems from the observation that bolus presence in the pharynx is the most prominent visual feature upon which to infer whether individual VFSS frames belong to the pharyngeal phase. We conducted extensive experiments with multiple convolutional neural networks (CNNs) on a dataset of 1245 bolus-level clips from 59 healthy subjects. ResultsWe demonstrated that the pharyngeal phase can be detected with an F1-score higher than 0.9. Moreover, by processing the class activation maps of the CNNs, we were able to localize the bolus with promising results, obtaining correlations with ground truth trajectories higher than 0.9, without any manual annotations of bolus location used for training purposes. ConclusionsOnce validated on a larger sample of participants with swallowing disorders, our framework will pave the way for the development of intelligent tools for VFSS analysis to support clinicians in swallowing assessment.

Vision-Based Finger Tapping Test in Patients With Parkinson's Disease via Spatial-Temporal 3D Hand Pose Estimation.

Finger tapping test is crucial for diagnosing Parkinson's Disease (PD), but manual visual evaluations can result in score discrepancy due to clinicians' subjectivity. Moreover, applying wearable sensors requires making physical contact and may hinder PD patient's raw movement patterns. Accordingly, a novel computer-vision approach is proposed using depth camera and spatial-temporal 3D hand pose estimation to capture and evaluate PD patients' 3D hand movement. Within this approach, a temporal encoding module is leveraged to extend A2J's deep learning framework to counter the pose jittering problem, and a pose refinement process is utilized to alleviate dependency on massive data. Additionally, the first vision-based 3D PD hand dataset of 112 hand samples from 48 PD patients and 11 control subjects is constructed, fully annotated by qualified physicians under clinical settings. Testing on this real-world data, this new model achieves 81.2% classification accuracy, even surpassing that of individual clinicians in comparison, fully demonstrating this proposition's effectiveness. The demo video can be accessed at https://github.com/ZhilinGuo/ST-A2J.

DEVELOPMENT AND RESEARCH OF COMPUTER VISION ALGORITHMS FOR IMAGE ANALYSIS OF DEFORMED ERYTHROCYTES

The paper presents the results of the development and research of algorithms for analyzing images obtained by blood microscopy using computer vision technologies. The objects of the study are images of erythrocytes deformed in the shear flow. The deformability of erythrocytes largely determines the nature of blood microcirculation and therefore is directly related to the diagnosis and treatment of many diseases. At the current moment image analysis of the evaluation of blood cells deformations is usually performed visually by medical technicians. To automate the image processing process various technologies were investigated. As a result, a new computer vision approach for rapid and accurate recognition of erythrocytes deformed in shear flow is presented. The developed algorithms use image binarization and a neural network based on the U-Net architecture for separated erythrocytes, and a neural network based on the StarDist architecture for their conglomerates. We present the results of the algorithms on real lood microscopy images, compare their performance and discuss the practical applications in medical diagnostics. Evaluation of the distribution of erythrocytes by deformability will provide additional diagnostic and scientific information for medical research. Automation of image analysis of deformed images will increase the accuracy and simultaneously reduce the time and cost of tests, leading to a better patient treatment.

Computer Vision Approaches for Segmentation of Nanoscale Precipitates in Nickel-Based Superalloy IN718

Computer Vision Approaches for Segmentation of Nanoscale Precipitates in Nickel-Based Superalloy IN718

Unmanned Aerial Vehicles: A Literature Review

In recent years, Unmanned Aerial Vehicles (UAVs) have grown and increased in applications because of computational simplicity and adaptive control capacity with strong support from both civilian and military sectors. The applications of UAVs in various military, commercial and civilian areas have led to sustainable results. The application areas include but are not limited to oil &amp; gas, cargo transport, geographic mapping, aerial photography, health care, and disaster management. The success of the UAV application missions is completely dependent on the accuracy in control provided by the flight controllers. Thus, there is a need for accurate, robust, and adaptive flight controllers. UAV dynamics modeling and identification and control of these vehicles are still major active areas of research and development. They pose severe challenges due to the vehicle's complex design, inherently nonlinear, and time-varying dynamics. The main goal of this paper is to identify the past research trends and recent improvements in UAVs. Furthermore, this paper discusses a comprehensive literature review according to the optimized objectives, solution techniques, and applications of UAVs such as Cargo Transport, Disaster Management, etc. According to the literature review, aerial photography is one of the applications of smart UAVs. The reliability of image matching across multiple camera perspectives, angles, and positions encourages computer vision approaches for UAV navigation, opening the way for future researchers to develop vision applications. This article presents a comprehensive literature review discussing the importance of UAV applications related to cost-effectiveness and versatility. Furthermore, a detailed survey of system modeling identification and control techniques is presented.

FEATURE MATCHING ENHANCEMENT USING THE GRAPH NEURAL NETWORK (GNN-RANSAC)

Abstract. Improving the performance of feature matching plays a key role in computers vision and photogrammetry applications, such as fast image recognition, Structure from Motion (SFM), aerial triangulation, Visual Simultaneous Localization and Mapping (VSLAM), etc., where the RANSAC algorithm is frequently used for outlier detection; note that RANSAC is the most widely used robust approach in photogrammetry and computer vision for outlier detection. It is known that the outlier ratio used in RANSAC primarily determines the number of trial runs needed, which eventually, determines the computation time. Over time, different methods have been proposed to reject the false-positive correspondences and improve RANSAC, such as GR_RANSAC, SuperGlue, and LPRANSAC. The specific objective of this study is to propose a filtering algorithm based on Graph Neural Networks (GNN), as a pre-processing step before RANSAC, which can result in improvements for rejecting the outliers. The research is based on the idea that descriptors of corresponding points, as well as their spatial relationship, should be similar in image sequences. In graph representation, built by the adjacency matrix of data (nodes features), there should be similarity for corresponding points that are close to each other in the image domain. From the many GNNs techniques, Graph Attention Networks (GATs) were selected for this study as they assign different importance to each neighbour’s contribution as anisotropic operations, so the features of neighbour nodes are not considered in the same way, unlike other GNNs techniques. In our approach, we build a graph in each image, because the similarity of the two-dimensional spatial relationships between points in the image domain of consecutive images should be similar. Then during processing, points with any significantly different neighbours are considered as outliers. Next, the points can be updated in the GNN layer. GNN-RANSAC is tested experimentally on real image pairs. Clearly, the proposed pre-filtering increases the inlier ratio and results in faster convergence compared to ordinary RANSAC, making it attractive for real-time applications. Furthermore, there is no need to learn the features.

The effects of urban greenway environment on recreational activities in tropical high-density Singapore: A computer vision approach

The urban greenway has been increasingly recognised as an important type of green infrastructure especially for land-scarce, densely-populated cities to efficiently provide their residents with continuous public spaces close to nature for recreation. Nevertheless, empirical studies on urban greenways and their recreational use rarely focus on high-density environment. Moreover, most research endeavours in this field are also largely confined to the subtropical climate, whereas much of the world’s future urban growth is projected to occur in the form of high-density mega-cities in much of tropical South and Southeast Asia. In view of these gaps, this study proposes a new approach that employs Computer Vision tools to examine the effects of the greenway’s physical environment on recreational activities, taking tropical Singapore as the test bed. The semantic segmentation model, PSPNet and the action detection model, ACAM are adapted and applied in conjunction with geographical information system tools to measure the greenway’s physical environment and people’s recreational activity at the human scale, and analyse their relationships. The result reveals a pattern that sees the clustering of different types of recreational activities at different time periods. It also reveals the relationships between recreational activities and specific environmental features, which were observed to have influenced the overall spatial distributions of the recreational activities. The finding also corroborates the design strategies for Singapore’s future urban greenways and offers a reference for engaging community groups to participate in the maintenance of urban greenways.

GlucoseML Mobile Application for Automated Dietary Assessment of Mediterranean Food.

Over the years and with the help of technology, the daily care of type 1 diabetes has been improved significantly. The increased adoption of continuous glucose monitoring, the continuous subcutaneous insulin injection and the accurate behavioral monitoring mHealth solutions have contributed to this phenomenon. In this study we present a mobile application for automated dietary assessment of Mediterranean food images as part of the GlucoseML system. Based on short-term predictive analysis of the glucose trajectory, GlucoseML is a type-1 diabetes self-management system. A computer vision approach is used as main part of the GlucoseML dietary assessment system calculating food carbohydrates, fats and proteins, relying on: (i) a deep learning subsystem for food image classification, and (ii) a 3D food image reconstruction subsystem for the volume estimation of food. The deep learning subsystem achieves 82.4% and 97.5% top-1 and top-5 accuracy, respectively, for food image classification while the subsystem for volume estimation of food achieves a mean absolute percentage error 10.7% for the four main categories of MedGRFood dataset.

An Automated Hyperparameter Tuning Recurrent Neural Network Model for Fruit Classification

Automated fruit classification is a stimulating problem in the fruit growing and retail industrial chain as it assists fruit growers and supermarket owners to recognize variety of fruits and the status of the container or stock to increase business profit and production efficacy. As a result, intelligent systems using machine learning and computer vision approaches were explored for ripeness grading, fruit defect categorization, and identification over the last few years. Recently, deep learning (DL) methods for classifying fruits led to promising performance that effectively extracts the feature and carries out an end-to-end image classification. This paper introduces an Automated Fruit Classification using Hyperparameter Optimized Deep Transfer Learning (AFC-HPODTL) model. The presented AFC-HPODTL model employs contrast enhancement as a pre-processing step which helps to enhance the quality of images. For feature extraction, the Adam optimizer with deep transfer learning-based DenseNet169 model is used in which the Adam optimizer fine-tunes the initial values of the DenseNet169 model. Moreover, a recurrent neural network (RNN) model is utilized for the identification and classification of fruits. At last, the Aquila optimization algorithm (AOA) is exploited for optimal hyperparameter tuning of the RNN model in such a way that the classification performance gets improved. The design of Adam optimizer and AOA-based hyperparameter optimizers for DenseNet and RNN models show the novelty of the work. The performance validation of the presented AFC-HPODTL model is carried out utilizing a benchmark dataset and the outcomes report the promising performance over its recent state-of-the-art approaches.

Marked crosswalks in US transit-oriented station areas, 2007–2020: A computer vision approach using street view imagery

Improving the built environment to support walking is a popular strategy to increase urban sustainability and walkability. In the past decade alone, many US cities have implemented crosswalk visibility enhancement programs as part of road safety improvements and active transportation plans. However, there are no systematic ways of measuring and monitoring the presence of key built environment attributes that influence the safety and walkability of an area, such as marked crosswalks. Furthermore, little is known about how these attributes change over time at a national scale. In this paper, we introduce an innovative approach using a deep learning-based computer vision model on Street View images to identify changes in intersection-level marked crosswalks around more than 4,000 US transit stations over a 14-year period. We found an increase in the overall number of marked crosswalks at intersections. Furthermore, high-visibility crosswalks became more common, as they replaced existing parallel-line crosswalks. We further examine crosswalks around transit stations in New York City and San Francisco to illustrate geographic variations and compare associations with other characteristics of the built environment as reported in the Smart Location Database. Areas with increases in high-visibility crosswalks focused on high density residential areas and areas with a higher percent of zero-vehicle households. However, geographic variations exist. For example, in San Francisco, transit station areas outside downtown or major corridors (South and Southwest of the city) had the lower prevalence of marked crosswalks. This analysis confirms important gaps in crosswalk visibility that call for safety enhancements and opens the door for additional research involving these data. We conclude by discussing the limitations and future research opportunities using computer vision to automatically detect large-scale transportation infrastructure changes at a relatively low cost.

Monitoring and automatic characterization of cracks in strain-hardening cementitious composite (SHCC) through intelligent interpretation of photos

This paper presents an intelligent photo interpretation approach to automatically monitor and characterize dense interconnected microcracks in strain-hardening cementitious composite (SHCC) featuring unique crack patterns in terms of crack number and crack width. The presented approach employs a stereo vision system that integrates binocular and monocular cameras for automatic detection, ranging, and quantification of cracks as well as characterization of crack patterns. The presented approach was implemented into evaluation of SHCC in flexural tests and direct tension tests. Dense microcracks were detected and ranged by the stereo vision system, segmented by an encoder-decoder approach, and quantified by an efficient computer vision approach. Evolution of the cracks was traced throughout the loading process until failure, and a statistical analysis revealed that the crack width was retained while the crack number monotonically increased. The interpretation time was shorter than 0.4 s for each photo, making the approach promising for monitoring of SHCC. The proposed system can be deployed for automated assessment of cementitious composites with complex crack patterns in material research and engineering structures.

Machine learning techniques to characterize functional traits of plankton from image data.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab‐based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

WaferSegClassNet - A light-weight network for classification and segmentation of semiconductor wafer defects

As the integration density and design intricacy of semiconductor wafers increase, the magnitude and complexity of defects in them are also on the rise. Since the manual inspection of wafer defects is costly, an automated artificial intelligence (AI) based computer-vision approach is highly desired. The previous works on defect analysis have several limitations, such as low accuracy and the need for separate models for classification and segmentation. For analyzing mixed-type defects, some previous works require separately training one model for each defect type, which is non-scalable. In this paper, we present WaferSegClassNet (WSCN), a novel network based on encoder-decoder architecture. WSCN performs simultaneous classification and segmentation of both single and mixed-type wafer defects. WSCN uses a “shared encoder” for classification, and segmentation, which allows training WSCN end-to-end. We use N-pair contrastive loss to first pretrain the encoder and then use BCE-Dice loss for segmentation, and categorical cross-entropy loss for classification. Use of N-pair contrastive loss helps in better embedding representation in the latent dimension of wafer maps. WSCN has a model size of only 0.51MB and performs only 0.2 M FLOPS. Thus, it is much lighter than other state-of-the-art models. Also, it requires only 150 epochs for convergence, compared to 4000 epochs needed by a previous work. We evaluate our model on the MixedWM38 dataset, which has 38,015 images. WSCN achieves an average classification accuracy of 98.2% and a dice coefficient of 0.9999. We are the first to show segmentation results on the MixedWM38 dataset. The source code can be obtained from https://github.com/ckmvigil/WaferSegClassNet.

Study on the Application of Visual Communication Design in APP Interface Design in the Context of Deep Learning.

Visual communication concepts enable linguistics or semiotics to the teaching of visual communication designs, creating graphic designs into an innovative and scientific discipline. The use of storyline techniques in visual communication not only inspires the imagination of designer but also arouses the visual memory of the audience. Besides, improving cultural heritage such as historical images is important to protect cultural diversity. Recently, the developments of deep learning (DL) and computer vision (CV) approaches make it possible for the automatic colorization of grayscale images into color images. Also, the usage of visual communication design in APP interface design has increased. With this motivation, this work introduces the enhanced deep learning-based automated historical image colorization (EDL-AHIC) technique for wireless network-enabled visual communication. The proposed EDL-AHIC technique intends to effectually convert the grayscale images into color images. The presented EDL-AHIC technique extracts the local as well as global features. For global feature extraction, the enhanced capsule network (ECN) model is applied. Finally, the fusion layer and decoding unit are employed to determine the output, i.e., chrominance component of the input image. A comprehensive experimental validation process is performed to ensure the betterment of the EDL-AHIC technique. The comparison study reported the supremacy of the EDL-AHIC technique over the other recent methods.

Graph neural network-enabled manufacturing method classification from engineering drawings

While millions of scanned engineering drawings are received every year, the online quotation companies for custom mechanical parts have experienced a surging need to increase their processing efficiency by replacing the currently manual inspection process with an automatic system. Previous work has used traditional, and data-driven computer-vision approaches to detect symbols and text information from the drawings. However, there lacks a unified framework to determine the associated manufacturing processes as a critical step for realizing an automatic quoting system. In this paper, we propose a computational framework to automatically determine the manufacturing method appropriate to produce each queried engineering drawing, such as lathing, sheet metal bending, and milling. We present a data-driven framework that directly processes the raster images with a series of pre-processing steps and accurately determines the corresponding manufacturing methods for the queried part with a graph neural network. We propose a novel line tracing algorithm to transform complex geometries in engineering drawings into vectorized line segments with minimal information loss. To extract the shape contours, we use an efficient image segmentation network to remove the information tables, followed by a sequential graph neural network to detect and eliminate dimension lines. Finally, we propose a novel graph neural network with updated graph connections to hierarchically distill graph descriptors and classify the engineering drawing by its appropriate manufacturing method. Our framework has been validated on industry datasets. We verify that our framework can effectively classify the engineering drawings with an accuracy of 90.78%. We further assess our framework by comparing it against state-of-the-art image classification algorithms(+17.20%).