Computer Vision-based Techniques Research Articles

Drowsiness Detection using EAR (Eye Aspect Ratio) by Machine Learning

Drowsiness detection is critical in many sectors, including transportation, healthcare, and workplace safety, since it may have a substantial influence on human performance and safety. Traditional sleepiness detection approaches are frequently subjective, time intensive, and unsuitable for real-time applications. In recent years, computer vision-based techniques that use eye-related characteristics to identify tiredness have shown promise. The eye aspect ratio, a geometric measure determined from ocular landmarks that indicates the openness or closure of the eyes, is one such trait. We present a sleepiness detection method in this research that blends eye aspect ratio computation with machine learning techniques to provide real-time and accurate drowsiness evaluation. We offer a thorough technique that includes calculating the eye-aspect ratio, extracting features, and classifying them with machine learning methods. The ability of our proposed technique is evaluated using a dataset of eye pictures acquired from individuals under various sleepiness situations. Our experimental findings show that our technique is successful in detecting sleepiness with good accuracy, sensitivity, and specificity. Our suggested method includes applications such as sleepy driving detection, tiredness monitoring in hospital settings, and workplace safety. This research advances the area of sleepiness detection by combining the ocular aspect ratio and machine learning to measure tiredness in real time. KEYWORDS: Drowsiness detection, Eye aspect ratio, Machine learning, Eye tracking, Blink rate, Pupil dilation, Real-time monitoring, Non-invasive, Safety, Transportation

On Automated Object Grasping for Intelligent Prosthetic Hands Using Machine Learning.

Prosthetic technology has witnessed remarkable advancements, yet challenges persist in achieving autonomous grasping control while ensuring the user's experience is not compromised. Current electronic prosthetics often require extensive training for users to gain fine motor control over the prosthetic fingers, hindering their usability and acceptance. To address this challenge and improve the autonomy of prosthetics, this paper proposes an automated method that leverages computer vision-based techniques and machine learning algorithms. In this study, three reinforcement learning algorithms, namely Soft Actor-Critic (SAC), Deep Q-Network (DQN), and Proximal Policy Optimization (PPO), are employed to train agents for automated grasping tasks. The results indicate that the SAC algorithm achieves the highest success rate of 99% among the three algorithms at just under 200,000 timesteps. This research also shows that an object's physical characteristics can affect the agent's ability to learn an optimal policy. Moreover, the findings highlight the potential of the SAC algorithm in developing intelligent prosthetic hands with automatic object-gripping capabilities.

A comprehensive review on helmet detection and number plate recognition approaches

Motorbikes serve as the primary mode of transportation in most countries as they are cost effective and appropriate for a nuclear family. But it has been observed that more than 70% of the users do not prefer to wear safety helmets for various reasons jeopardizing their lives and falling prey to accidents. The most prevalent method for ensuring this right now is traffic police manually monitoring the motorcyclists. But due to excess traffic and limited traffic personnels, many violators go unrecognized and continue to practice the same. Thus, it is important to eliminate the human intervention and automate the monitoring system using deep learning and computer vision-based techniques. Our proposed system implements this by extracting number plate of helmet violators and generates an e-challan on the registered mobile number. We propose using a custom trained YOLO-v8 model for violation detection and YOLO-v8 + EasyOCR for number plate detection and extraction. Canny Edge Detection is a preprocessing step that can be used to enhance the edges of objects in an image, making them more distinguishable. This system holds great potential for enhancing safety-related policies and ensuring strict enforcement of traffic regulations. Additionally, it contributes to the advancement of traffic management through the implementation of an AI-based automated traffic violation and ticketing system.

Sports video retrieval and classification using focus u-net based squeeze excitation and residual mapping deep learning model

Sports videos are widely used by athletes and coaches for training and match analysis purposes outside the mainstream audience. Sports videos should be effectively classified into different genres to easily retrieve and index them from large video datasets. Manual labelling classification methods may cause errors and have low accuracy. Classification based on video content analysis is challenging for computer vision-based techniques. This work introduces an improved focus-net deep learning (DL) model called the Convolutional squeeze U-Net based encoder-decoder for sports video retrieval and classification. First, the keyframes are extracted from the input sports video using a clustering and optical flow analysis method. In the next stage, the frames are pre-processed using a smoothed shock filtering technique to remove the noise. The process of image segmentation is carried out using a Convolutional squeeze U-Net based encoder-decoder model. Finally, the sports video can be classified using the softmax classifier. A CNN (convolutional neural network) is utilized at the encoder section for extracting the features and fed to the decoder for video classification. The experiments are performed in the UCF101 dataset, and the proposed model achieved an overall accuracy of 99.68%. Hence, it is proven that the proposed focus-net model can be efficiently utilized in sports video classification.

Vigour testing for the rice seed with computer vision-based techniques.

Rice is the staple food for approximately half of the world's population. Seed vigour has a crucial impact on the yield, which can be evaluated by germination rate, vigor index and etc. Existing seed vigour testing methods heavily rely on manual inspections that are destructive, time-consuming, and labor-intensive. To address the drawbacks of existing rice seed vigour testing, we proposed a multispectral image-based non-destructive seed germination testing approach. Specifically, we collected multispectral data in 19 wavebands for six rice varieties. Furthermore, we designed an end-to-end pipeline, denoted as MsiFormer (MisFormer cod3e will be available at https://github.com/LiaoYun0x0/MisFormer) by integrating a Yolo-based object detector (trained Yolo v5) and a vision transformer-based vigour testing model, which effectively improved the automation and efficiency of existing techniques. In order to objectively evaluate the performance of the proposed method in this paper, we conduct a comparison between MisFormer and other 3 deep learning methods. The results showed that, MisFormer performed much better with the accuracy of 94.17%, which was 2.5%-18.34% higher than the other 3 deep learning methods. Besides MsiFormer, possibilities of CIELab mediated image analysis of TTC (tetrazolium chloride) staining in rice seed viability and nCDA (normalized canonical discriminant analysis) in rice seed vigour were also discussed, where CIELab L* of TTC staining were negatively correlated with vigor index and germination rate, with Pearson's correlation coefficient of -0.9874, -0.9802 respectively, and CIELab A* of TTC staining were and positively correlated with vigor index and germination rate, with Pearson's correlation coefficient of 0.9624, 0.9544 respectively, and CIELab A* of nCDA had Pearson's correlation coefficient of -0.8866 and -0.9340 with vigor index and germination rate, respectively. Besides testing methods, vigour results within and among variety(ies) showed that, there were great variations among the 6 rice varieties, and mean coefficient of variation (CV) of vigor index of individual seed within a variety reached 64.87%, revealing the high risk of conventional methods in random sampling. Vigour variations had close relationship with wavelengths of 780 nm-970 nm, indicating their value in future research.

REALITY-ENHANCED SYNTHETIC IMAGE TRAINING DATASET FOR COMPUTER VISION CONSTRUCTION MONITORING

Recent growth in the availability of aerial and ground-level images from construction sites has created a surge in computer vision-based techniques for construction monitoring applications. To address the need for large volumes of visual data with ground truth for training the underlying machine learning models, the application of synthetic data, particularly from 3D/4D BIM, has gained traction in the research community. However, the existing domain gap between real and synthetic images, such as rendering noise from repeated material texture patterns, randomized positional camera parameters that collect the ground truth of BIM elements, and the poorly visibility of BIM elements to the camera viewpoints within these environments, has negatively impacted the potential use of these synthetic datasets as scale. To address these limitations, this paper presents a new synthetic image generation pipeline that optimizes and integrates camera extrinsic parameters with a novel synthetic image appearance enhancement technique to generate high volume of quality synthetic data with respective ground-truth. The use of synthetic datasets for training progress monitoring models is validated through several real-data segmentation cases that incorporate: 1) the automatic collection of synthetic images and ground-truth annotations from high-LoD BIM model disciplines, 2) optimization of positional camera parameters using element visibility metrics, and 3) the enhancement of realism of synthetic images using a patch-based generative approach. The benefits and the current limitations to automated progress monitoring, as well as robotic path automation and optimization for progress motoring real data collection are discussed.

A novel glaucoma detection model using Unet++-based segmentation and ResNet with GRU-based optimized deep learning

Glaucoma is a dangerous eye disease and it affects the vision in people. It mainly damages the eye nerves and the severe cause of glaucoma leads to permanent sight loss. The target of the glaucoma disease detection approach is lowering the intraocular pressure and reducing inflammation. The detection of glaucoma disease takes more time. This method functioned based on the availability of energy and sample resources. However, the existing approaches have the chance for human error and misclassification results. Many computer vision-based techniques are used for glaucoma detection but, none of them are detected glaucoma disease in the early stage. Then, they focused on the Convolutional Neural Network (CNN) of segmentation and feature extraction for detecting glaucoma disease. In this research, a new glaucoma detection model with an ensemble learning architecture is implemented with the heuristic strategy. Then, the collected images are given into histogram equalization technique and Contrast Limited Adaptive Histogram Equalization (CLAHE) for pre-processing the retinal fundus images. The pre-processed images are separately considered for performing the disc and cup segmentation, where both the segmentation is done through U-Shape Network (Unet++) technique. The segmented disc and cup images preceded towards the glaucoma detection, in which the hybrid deep learning technique of Residual Network (ResNet) with Gated Recurrent Units (ResNet-GRU) is used for detecting the glaucoma disease. Here, certain parameters in the deep learning techniques for glaucoma detection are tuned with a Modified Density Factor-based HBA (MDF-HBA). The detected output through disc-segmented and cup-segmented images are undergone averaging to finalize the optimal detection output.

Interpreting the variation in particle size of ground spice by high-resolution visual and spectral imaging: A ginger case study

High-resolution (HR) visual imaging and spectral imaging are common computer vision-based techniques used for food quality analysis and/or authentication based on the interaction of light and material surface and/or composition. The particle size of ground spices is an important morphological feature that affects the physico-chemical properties of food products containing such particles. This study aimed to interpret the impact of particle size of ground spice on its HR visual profile and spectral imaging profile using ginger powder as a representative spice powder model. The results revealed an increase in the light reflection with the decrease of particle size of ginger powder, which was manifested by the lighter colour (higher percentage of the colour code with lighter yellow colour) of the HR visual image and stronger reflection with spectral imaging. The study also revealed that, in spectral imaging, the influence of the particle size of ginger powder increased with rising wavelengths. Finally, the results indicated a relationship between spectral wavelengths, ginger particle size, and other natural variables of the products which might be generated from cultivation to processing. Ultimately, the impact of natural variables arising during the food production process on the physico-chemical properties of the product should be fully considered or even additionally evaluated prior to the application of specific food quality and/or authentication analytical techniques.

Addressing practical challenge of using autopilot drone for asphalt surface monitoring: Road detection, segmentation, and following

According to the recent world bank report, around 80% of a life-cycle cost of a road is devoted to maintenance which includes monitoring and repair processes. To more effectively keep road serviceability, knowing the current status of road health is crucial. Typically, the monitoring processes are handled by human-intensive procedures. So, automating this task could lead to saving time and cost and also, improve efficiency. Although, this task has been optimized by Ground Vehicles further. Yet it lacks the disadvantages of human-intensive procedure. As they are still semi-manual, creates traffic issues, and not being eco-friendly or cost efficient. In recent years, UAVs have been successfully utilized to handle a wide range of labor-based tasks including road assessments. This paper presents a drone-based solution to automate road monitoring and segmentation as well as addressing the practical challenges of using drones for this purpose. To do so, a platform is developed that controls a drone through a road monitoring flight using computer vision-based techniques. The platform, rather than sending maneuvering commands to the drone during a flight starting from takeoff to landing, firstly could detect road boundaries by finding vanishing points, and secondly, could identify the dash lines and the center of the road. Finally, the captured road is segmented and labeled with the temporal and geographical information supplied by the Inertial Measurement Unit (IMU) of the drone. It has been tried to optimize the platform in order to handle all the processes in real-time while the UAV is following the road during a flight. To evaluate the proposed idea, the developed platform is tested in urban areas. The achieved results demonstrate how effectively could detect and segment a road in different environments using an off-the-shelf UAV. This platform could improve the automation of the data gathering process required in road maintenance.

A survey on automatic engagement recognition methods: online and traditional classroom

Student engagement in a learning environment is directly related to students’ perception and involvement of the educational activities in the class, along with their physical and mental health. This paper presents an extensive survey of the various automatic engagement detection approaches and algorithms based on computer vision, physiological and neurological signals analysis-based methods. The computer vision-based techniques depend on the traits captured by image sensors such as facial expressions, gesture and posture analysis, and gaze direction. The physiological and neurological signal based approach depends on the sensor data, like heart rate (HR), electroencephalogram (EEG), blood pressure (BP), and galvanic skin response (GSR). A brief analysis of the available datasets for Engagement Recognition and its features are also summarized. This study highlights a few commercially available wearables which provides the physiological signals that helps in student’s attentivity recognition. Our study reveal that the accuracy of engagement recognition system will increase if we increase the number of modalities used. In this survey, we intend to support the upcoming researchers as well as tutors of smart education set up by providing an overview of existing or proposed approaches of automatic engagement detection techniques in different scenarios.

Cooktop Sensing Based on a YOLO Object Detection Algorithm.

Deep Learning (DL) has provided a significant breakthrough in many areas of research and industry. The development of Convolutional Neural Networks (CNNs) has enabled the improvement of computer vision-based techniques, making the information gathered from cameras more useful. For this reason, recently, studies have been carried out on the use of image-based DL in some areas of people's daily life. In this paper, an object detection-based algorithm is proposed to modify and improve the user experience in relation to the use of cooking appliances. The algorithm can sense common kitchen objects and identify interesting situations for users. Some of these situations are the detection of utensils on lit hobs, recognition of boiling, smoking and oil in kitchenware, and determination of good cookware size adjustment, among others. In addition, the authors have achieved sensor fusion by using a cooker hob with Bluetooth connectivity, so it is possible to automatically interact with it via an external device such as a computer or a mobile phone. Our main contribution focuses on supporting people when they are cooking, controlling heaters, or alerting them with different types of alarms. To the best of our knowledge, this is the first time a YOLO algorithm has been used to control the cooktop by means of visual sensorization. Moreover, this research paper provides a comparison of the detection performance among different YOLO networks. Additionally, a dataset of more than 7500 images has been generated and multiple data augmentation techniques have been compared. The results show that YOLOv5s can successfully detect common kitchen objects with high accuracy and fast speed, and it can be employed for realistic cooking environment applications. Finally, multiple examples of the identification of interesting situations and how we act on the cooktop are presented.

A computer vision-aided analysis of facial similarities in Song dynasty imperial portraits

Similarity between faces in portraiture is incredibly informative for art historical questions involving the sitter's identity, as well for setting a painting in its historical context to understand why someone was depicted a certain way. A set of royal portraits from Song dynasty, China, has been the subject of rich art historical scholarship. Here, I demonstrate the usefulness of computer vision-based quantitative metrics in complementing existing rich subjective evaluations. Working with the portrait set, I show that L2 distances generated by OpenFace support the accepted hypothesis that emperor Lizong is depicted in Listening to the Wind in the Pines. I then use the technique to gain insight into whether the zither player in Listening to the Zither resembles emperor Huizong and why that might be, as well as what degrees of similarity between emperor portraits in the set may mean in terms of metaphorical inclusion or exclusion from the lineage. I then extend discussions on metaphorical inclusion to women in this set by exploring spousal similarity. Fascinating mysteries surrounding posthumous portraiture float amidst confounding factors in the clouds of memory, and this study shows the promise of using computer vision-based techniques as complements to subjective analyses in exploring these mysteries.

IoT-Based Non-Intrusive Automated Driver Drowsiness Monitoring Framework for Logistics and Public Transport Applications to Enhance Road Safety

The exponential growth in road accidents has led to a need for continuous driver monitoring to enhance road safety. Existing techniques rely on vehicle sensor-based and behavior analysis-based approaches, where the behavior analysis-based approaches are generally considered more desirable as they enable reliable detection of a more elaborate set of driver behaviors. They are categorized as intrusive and non-intrusive approaches. Unlike intrusive approaches that generally rely on constant direct human contact with sensors (physiological signals) and are sensitive to artifacts, non-intrusive approaches offer a more effective behavior monitoring using computer vision-based techniques. This paper proposes an end-to-end non-intrusive IoT-based automated framework to monitor driver behaviors, designed specifically for logistic and public transport applications. It consists of an embedded system, edge computing and cloud computing modules, and a mobile phone application, in an attempt to provide a holistic unified solution for drowsiness detection, monitoring, as well as evaluation of drivers. Drowsiness detection is based on detecting sleeping, yawning, and distraction behaviors using an image processing-based technique. To minimize the effects of latency, throughput, and packet losses, edge computing is performed using commercial off-the-shelf embedded boards. Moreover, a cloud-hosted real-time database for remote monitoring on interactive Android mobile application has been set up, where admin can add multiple drivers to get drowsiness notifications along with other useful related information for driver evaluation. An extensive experimental testing has been performed, obtaining encouraging results. An overall accuracy of 96% is achieved along with an enhanced robustness, portability, and usability of the proposed framework.

A Systematic Review on the Detection and Classification of Plant Diseases Using Machine Learning

The occurrence of disease in plants might affect the crop production at a large scale, resulting into decline of the economic growth rate of the country. The disease in plants can be detected and treated at an early stage. Machine learning (ML), deep learning (DL), and computer vision-based techniques could play a pivotal role in detecting and classifying the diseases at an early stage. These approaches have even surpassed the human performance, as well as image processing based traditional approaches in the analysis and classification of plant diseases. Over the years, numerous authors have applied various image processing ML and DL techniques for the diagnosis of different ailments in plants that gives great hope to the farmers and landlords to cure the disease at an early stage. In this study, the authors addressed and evaluated the various currently existing state of art methods and techniques based on machine and deep learning. Besides, the authors have also focused on various limitations and challenges of these approaches that can explore greater possibly of these methods about their usability for disease detection in plants.

Bayesian-optimized deep learning model to segment deterioration patterns underneath bridge decks photographed by unmanned aerial vehicle

In recent years, bridge collapses and fractures have occurred in various countries mostly following a lack of inspection and maintenance. External inspection processes can be very time-consuming and pose labor safety hazards. Terrain obstacles may also prevent the thorough inspection of some structures. The use of artificial intelligence instead of visual inspection by bridge inspectors is state-of-the-art. This study develops a Bayesian-optimized deep learning model for use on an unmanned aerial vehicle (UAV) to identify the deterioration patterns and segment areas of composite decks under bridges by computer vision-based techniques. The proposed module alters traditional labor-intensive methods of visual bridge inspections, reduces labor safety hazards, and increases inspection accuracy. It can be embedded in an artificial intelligence chip, which is then installed in a consumer-grade UAV, making it a dedicated drone for the external inspection of composite bridges.

Editorial: Understanding animals’ phenotype through automatic behavior assessment

Animal behaviour is an elastic trait that changes according to environmental circumstances and 12 intrinsic properties, such as health or genetic background (i.e., breed). The term behaviour elasticity 13 refers to the range of behaviour that an animal is capable to express. The evaluation of behaviour may 14 be used to estimate either the adaptation capacity to a changing environment or the intrinsic properties, 15 such as health status, temperament, etc. However, in order to identify changes in behaviour, it needs to 16 be systematically and objectively assessed. This is because some changes in behaviour are only 17 perceptible if it is monitored continuously, which was not feasible in traditional behaviour assessment 18 using subjective human observations. 19 is undoubtably dynamic, so this can be used to understand how animals cope with their environment. 76Nevertheless, none of the methods can provide a holistic assessment of the behaviour. Integration of 77 the methodologies, a multimodality approach in combining the computer vision-based techniques

Anterior Cruciate Ligament Tear Detection Based on Deep Convolutional Neural Network.

Anterior cruciate ligament (ACL) tear is very common in football players, volleyball players, sprinters, runners, etc. It occurs frequently due to extra stretching and sudden movement and causes extreme pain to the patient. Various computer vision-based techniques have been employed for ACL tear detection, but the performance of most of these systems is challenging because of the complex structure of knee ligaments. This paper presents a three-layered compact parallel deep convolutional neural network (CPDCNN) to enhance the feature distinctiveness of the knee MRI images for anterior cruciate ligament (ACL) tear detection in knee MRI images. The performance of the proposed approach is evaluated for the MRNet knee images dataset using accuracy, recall, precision, and the F1 score. The proposed CPDCNN offers an overall accuracy of 96.60%, a recall rate of 0.9668, a precision of 0.9654, and an F1 score of 0.9582, which shows superiority over the existing state-of-the-art methods for knee tear detection.

Mangroves in Gulf of Khambhat, Gujarat: a case study

The present study is mainly focused on studying the mangroves in the Gulf of Khambhat and exploring the previous research work done over the area for mapping and monitoring of mangrove ecosystem, which will further help to analyze the future scope of mangrove research in this area. To study the status and distribution of mangroves in the Gulf of the Khambhat region, mainly three ground truth locations are selected to collect ground truth data, which are Ghogha (Bhavanagar), Dahej (Bharuch), and Kantiyajal (Bharuch). All three locations are analyzed for the study of the mangroves and their species distribution. Furthermore, the Land Use / Land Cover change in all three areas is also analyzed to find mangrove cover change. The resultant data can be further used for training mangrove monitoring models using remote sensing data. The study also highlights that the current research done over the gulf of Khambhat area is not using the full strength of remote sensing as only multispectral optical imageries are used in various studies along with visual interpretation techniques. The study highly recommends the use of remote sensing with multi-sensor, multitemporal along with state-of-the-art machine learning and computer vision-based techniques for the mapping and monitoring of the Mangrove ecosystem in the Gulf of Khambhat region.

A Novel Vision-Based Framework for Identifying Dynamic Vehicle Loads on Long-Span Bridges: A Case Study of Jiangyin Bridge, China

Since the high randomness feature of traffic flows, it is challenging to be aware of the spatial-temporal characteristics of dynamic vehicle loads on a long-span bridge. Compared with conventional measuring approaches, computer vision-based techniques have competitive advantages in the measurement of factual vehicle loads parameters, including the passing time, velocity, vehicle type, and corresponding locations on the bridge at each timestamp. In this paper, a vision-based framework for the identifications of dynamic vehicle loads on long-span bridges is proposed. To be specific, the framework consists of two major parts. The first part is the extraction of spatial-temporal factors of vehicles, including the detection of vehicles and the estimation of tracklets. Here, Single Shot Multi-Box Detection (SSD) architecture with MobileNets is utilized for detecting vehicles, while the data association technique with Kalman filter is applied on detection results for generating tracklets. Then, in the second part, based on the detection and tracking results, a Multi-layer Monte Carlo (MLMC) model is developed for axle-load intensities inference and estimation. The learned model is utilized to infer the dynamic load of vehicles from the detection results and generated tracklets. The proposed methodology is validated via an experiment of a case study on the Jiangyin bridge, Jiangsu Province, China, and several spatial-temporal distributions of vehicle loads are specifically analyzed and discussed. Experimental results indicate our proposed framework is a feasible and reliable solution to predict and identify dynamic vehicle loads on long-span bridges, without the assistance of the deck-embedded Weigh-in-Motion (WIM) system.

A Systematic Analysis of Machine Learning and Deep Learning Based Approaches for Plant Leaf Disease Classification: A Review

Crops’ production and quality of yields are heavily affected by crop diseases which cause adverse impacts on food security as well as economic losses. In India, agriculture is a prime source of income in most rural areas. Hence, there is an intense need to employ novel and accurate computer vision-based techniques for automatic crop disease detection and their classification so that prophylactic actions can be recommended in a timely manner. In literature, numerous computer vision-based techniques by utilizing divergent combinations of machine learning, deep learning, CNN, and various image-processing techniques along with their associated merits and demerits have already been discussed. In this study, we systematically reviewed recent research studies undertaken by a variety of scholars and researchers of fungal and bacterial plant disease detection and classification and summarized them based on vital parameters like type of crop utilized, deep learning/machine learning architecture used, dataset utilized for experiments, performance matrices, types of disease detected and classified, and highest accuracy achieved by the model. As per the analysis carried out, in the category of machine learning-based approaches, 70% of studies utilized real-field plant leaf images and 30% utilized laboratory condition plant leaf images for disease classification while in the case of deep learning-based approaches, 55% studied employed laboratory-conditioned images from the PlantVillage dataset, 25% utilized real-field images, and 20% utilized open image datasets. The average accuracy attained with deep learning-based approaches is quite higher at 98.8% as compared to machine learning-based approaches at 92.2%. In the case of deep learning-based methods, we also analyzed the performances of pretrained and training from scratch models that have been utilized in various studies for plant leaf disease classification. Pretrained models perform better with 99.64% classification accuracy compared to training from scratch models which achieved 98.64% average accuracy. We also highlighted some major issues encountered in the computer vision-based disease detection and classification approach used in literature and provided recommendations that will help and guide researchers to explore new dimensions in crop disease recognition.