Vision-based System Research Articles

Development of image-based fast defect recognition and localization network (FDRLNet) for steel surfaces

The utilization of vision-based systems for automated inspection and quality control tasks increased in recent years due to the adaptation of Industry 4.0 initiatives in manufacturing. Continuous steel production units extensively employ vision-based solutions to classify and localize surface defects. The inspection system must provide quick and reliable feedback about surface defect type (classification) and location (localization) utilizing images acquired from the camera. This paper presents a novel surface defect detector, Fast Defect Recognition and Localization Network (FDRLNet), for achieving accurate prediction abilities at higher inference speeds. The robustness of the proposed approach is validated by utilizing publicly available Northeastern University (NEU-DET) surface defect dataset. The prediction abilities are corroborated by comparing the mean Average Precision (mAP) and inference speed with other competitive approaches presented in the literature. It has been shown that the proposed approach has robust prediction abilities at higher inference speeds and can be implemented in real-time defect detection tasks.

Comprehensive review on vehicle Detection, classification and counting on highways

Vehicle detection, counting and finally classification has been an important aspect of traffic analysis specially on highways in many developed and developing nations. This has vitalized the monitoring of freeways and reduced the reliance on human traffic monitors specially in developed nations. However, much research carried out in this regard since 1995 have been slow to progress until around 2000. Since then, much more encouraging outcomes have been achieved. This has been mainly due to the advances on vision-based computing and the miniaturizing of hardware since the early 2000. Initial vision-based systems used basic computer vision approaches such as background subtraction and edge detection to detect and count vehicles. Early progress of classification had little success until around 2010. Since 2010, many computer vision approaches using Neural Networks have been gradually increasing the efficiency and the realtime operation of such systems. Since then, many deep learning based neural network approaches have produced remarkable outcomes specially in classification. We have also reported highly accurate and efficient deep learning based YOLOv5 system that has recorded an astounding success. We have used models YOLOv5l, YOLOv5m, YOLOv5n and YOLOv5s in our research to ascertain the best model for the task.However, low light conditions associated with overcast, and dusk have significantly reduced the accuracy in many deep learning-based systems. There are other techniques based on approaches such as headlight detection have appeared to increase the accuracy of counting however, classification at night without adequate lighting still pauses a formidable challenge. The work presented here analyses the vehicle counting problem using computer vision providing many different viewpoints in a view to provide the reader with accurate problem definition.

Development of Obstacle Detection System Based On the Integration of Different Based Sensor for Small-Sized UAV

Due to the physical size and weight limits of small unmanned aerial vehicles (UAVs), developing a reliable obstacle detection a system that can provide an effective and safe avoidance path is extremely difficult. Prior work has tended to use a vision-based sensor as the primary detecting sensor however, this has resulted in a high reliance on texture appearance and a lack of distance sensing capabilities. Furthermore, due to the inability to detect the free region, vision-based sensor detection systems have difficulty developing a trusted safe avoidance path. However, most wide spectrum range sensors are bulky and expensive, making them unsuitable for small UAVs. This projectaims to construct an obstacles detection system with the integration of various based sensors for a small UAV. The potential obstacles are identified by categorizing feature points identified in image frame. The suggested approach was tested in a real-world setting for both of the observed scenarios, which included various obstacles configurations. Two types of scenarios are experimented in this project consists of single frontal obstacles and presence of side obstacles alongside the frontal obstacles. On top of that, the position of the side obstacle is aligned to the frontal obstacle and then will be positioned in the increment of 20cm further from the frontal obstacle in order to analyze the outcome of the proposed algorithm. The proposed detection system had a possibility to be a trustworthy system even after utilizing the depth perception technique, however this does not imply that the proposed system is faultless. The results show that the suggested algorithm system detects and distinguishes between the potential obstacles and free region for a single frontal obstacle perfectly. However, there were improvements that should be implemented with the proposed system's ability of detection for multiple obstacles.

Editorial

Dear Readers,  Welcome to the seventh issue in 2023. I am very pleased to announce the journal’s continued high Scopus CiteScore of 2.7 and Web of Science Impact Factor of 1.0 for 2022, indicating another scientifically successful year. On behalf of the J.UCS team, I would like to thank all the authors for their sound research contributions, the reviewers for their very helpful suggestions for improvements, and the consortium members for their financial support. Your commitment and dedicated work have contributed significantly to the long-lasting success of our journal.  As we want to secure the financial support also for the years to come, we are looking for institutions and libraries to financially support our diamond open access journal as consortium members, who will then benefit from the research community, international visibility, and the opportunity to manage special issues and focused topics within the journal. Please think about the possibility of such financial participation of your institution, we would be very grateful for any kind of support.  In this regular issue, I am very pleased to introduce six accepted papers from seven different countries and 20 involved authors.  Ana P. Allian, Leandro F. Silva, Edson OliveiraJr and Elisa Y. Nakagawa from Brazil present VMTools-RA, a reference architecture that encompasses the knowledge and practice for developing and evolving variability tools. In a collaboration between researchers from the UK and Estonia, Vimal Dwivedi, Mubashar Iqbal, Alex Norta and Raimundas Matulevičius are focusing their research on the evaluation of a legally binding smart-contract language for blockchain applications. Monika, Seema Verma, and Pardeep Kumar from India discuss an intelligent vision-based decision-making system for the exploration of past aviation accidents and incidents, which is based on a visual query-based model capable of analyzing the major factors including flight phases, human factors, weather conditions, and faulty components in particular aircraft models. Luis Eduardo Ordoñez Palacios, Víctor Bucheli Guerrero and Hugo Ordoñez from Colombia present their research on integrating satellite imagery and meteorological data to estimate solar radiation applying and evaluating five machine learning models. In a collaboration between researchers from Palestine and Egypt, Muath Sabha, Thaer Thaher, and Marwa M. Emam apply cooperative swarm intelligence algorithms to adaptive multilevel thresholding segmentation of COVID-19 CT scan images. Geovana Ramos Sousa Silva, Genaína Nunes Rodrigues and Edna Dias Canedo from Brazil introduce their work on a modeling strategy to design and verify chatbot conversational flows via the Uppaal model checking tool.  Enjoy Reading! Cordially, Christian Gütl, Managing Editor 

Intelligent Vision Based Decision Making System for Aviation Accidents and Incidents

Safety has become the primary concern for the air transportation system nowadays primarily due to increasing air traffic throughout the world. Various regulatory bodies have been maintaining enormous amount of aviation accidental data repositories. This past data is highly complex because of its many temporal and geographical components along with multiple variables. To be able to analyze this past data, there is always a need of user friendly and GUI based System. This article has proposed an intelligent vision-based decision-making system for the exploration of past aviation accidents and incidents dataset. The proposed visual query-based model is capable to analyse the major factors like flight phases, human factors, weather conditions and faulty components in particular aircraft models which are responsible for those unsafe events and may claim life of many passengers who are traveling and crew personnels. This model enables the users to express “what” visuals should be created instead of “how” to create them. Various case studies conducted through visual queries have proved that the system will be highly able to improve situational awareness regarding flight conditions to the crew members and air traffic controllers along with aviation authorities so that they are able to take timely decisions and deciding on what kind of training staff members need to reduce the consequences of such accidents and incidents.

Automatic Outdoor Fire detection using Deep learning

Abstract
 One of the unlucky phenomena that contribute to environmental disasters is fire, which also poses a serious threat to human safety and life, particularly when it is not recognized by sensor-based fire detection systems. Therefore, putting inexpensive and efficient sensors in certain locations will greatly speed up the detection of fires. Vision-based fire detection systems take advantage of the three fundamental features of fire: color, movement, and shape (fire shape). Work has been done to build smoke and fire detection systems based on images that use security cameras. In this study, fire photos are used to extract features before inputs are made. To one of the classification techniques that applies SVM-based machine learning. and CNN-based deep learning technique. We use the linear regression model so that the bounding box of the classified object determines the correct coordinates.

A Computer Vision-Based Automatic System for Egg Grading and Defect Detection.

Defective eggs diminish the value of laying hen production, particularly in cage-free systems with a higher incidence of floor eggs. To enhance quality, machine vision and image processing have facilitated the development of automated grading and defect detection systems. Additionally, egg measurement systems utilize weight-sorting for optimal market value. However, few studies have integrated deep learning and machine vision techniques for combined egg classification and weighting. To address this gap, a two-stage model was developed based on real-time multitask detection (RTMDet) and random forest networks to predict egg category and weight. The model uses convolutional neural network (CNN) and regression techniques were used to perform joint egg classification and weighing. RTMDet was used to sort and extract egg features for classification, and a Random Forest algorithm was used to predict egg weight based on the extracted features (major axis and minor axis). The results of the study showed that the best achieved accuracy was 94.8% and best R2 was 96.0%. In addition, the model can be used to automatically exclude non-standard-size eggs and eggs with exterior issues (e.g., calcium deposit, stains, and cracks). This detector is among the first models that perform the joint function of egg-sorting and weighing eggs, and is capable of classifying them into five categories (intact, crack, bloody, floor, and non-standard) and measuring them up to jumbo size. By implementing the findings of this study, the poultry industry can reduce costs and increase productivity, ultimately leading to better-quality products for consumers.

Stereo SLAM in Dynamic Environments Using Semantic Segmentation

As we all know, many dynamic objects appear almost continuously in the real world that are immensely capable of impairing the performance of the majority of vision-based SLAM systems based on the static-world assumption. In order to improve the robustness and accuracy of visual SLAM in high-dynamic environments, a real-time and robust stereo SLAM system for dynamic scenes was proposed. To weaken the influence of dynamic content, the moving-object detection method was put forward in our visual odometry, and then the semantic segmentation network was combined in our stereo SLAM to extract pixel-level contours of dynamic objects. Then, the influences of dynamic objects were significantly weakened and the performance of our system increased markedly in dynamic, complex, and crowed city spaces. Following experiments with both the KITTI Odometry dataset and in a real-life scene, the results showed that our method could dramatically decrease the tracking error or drift, and improve the robustness and stability of our stereo SLAM in high dynamic outdoor scenarios.

Vision-Based In-Flight Collision Avoidance Control Based on Background Subtraction Using Embedded System.

The development of high-performance, low-cost unmanned aerial vehicles paired with rapid progress in vision-based perception systems herald a new era of autonomous flight systems with mission-ready capabilities. One of the key features of an autonomous UAV is a robust mid-air collision avoidance strategy. This paper proposes a vision-based in-flight collision avoidance system based on background subtraction using an embedded computing system for unmanned aerial vehicles (UAVs). The pipeline of proposed in-flight collision avoidance system is as follows: (i) subtract dynamic background subtraction to remove it and to detect moving objects, (ii) denoise using morphology and binarization methods, (iii) cluster the moving objects and remove noise blobs, using Euclidean clustering, (iv) distinguish independent objects and track the movement using the Kalman filter, and (v) avoid collision, using the proposed decision-making techniques. This work focuses on the design and the demonstration of a vision-based fast-moving object detection and tracking system with decision-making capabilities to perform evasive maneuvers to replace a high-vision system such as event camera. The novelty of our method lies in the motion-compensating moving object detection framework, which accomplishes the task with background subtraction via a two-dimensional transformation approximation. Clustering and tracking algorithms process detection data to track independent objects, and stereo-camera-based distance estimation is conducted to estimate the three-dimensional trajectory, which is then used during decision-making procedures. The examination of the system is conducted with a test quadrotor UAV, and appropriate algorithm parameters for various requirements are deduced.

Study on the Interaction Behaviors Identification of Construction Workers Based on ST-GCN and YOLO.

The construction industry is accident-prone, and unsafe behaviors of construction workers have been identified as a leading cause of accidents. One important countermeasure to prevent accidents is monitoring and managing those unsafe behaviors. The most popular way of detecting and identifying workers' unsafe behaviors is the computer vision-based intelligent monitoring system. However, most of the existing research or products focused only on the workers' behaviors (i.e., motions) recognition, limited studies considered the interaction between man-machine, man-material or man-environments. Those interactions are very important for judging whether the workers' behaviors are safe or not, from the standpoint of safety management. This study aims to develop a new method of identifying construction workers' unsafe behaviors, i.e., unsafe interaction between man-machine/material, based on ST-GCN (Spatial Temporal Graph Convolutional Networks) and YOLO (You Only Look Once), which could provide more direct and valuable information for safety management. In this study, two trained YOLO-based models were, respectively, used to detect safety signs in the workplace, and objects that interacted with workers. Then, an ST-GCN model was trained to detect and identify workers' behaviors. Lastly, a decision algorithm was developed considering interactions between man-machine/material, based on YOLO and ST-GCN results. Results show good performance of the developed method, compared to only using ST-GCN, the accuracy was significantly improved from 51.79% to 85.71%, 61.61% to 99.11%, and 58.04% to 100.00%, respectively, in the identification of the following three kinds of behaviors, throwing (throwing hammer, throwing bottle), operating (turning on switch, putting bottle), and crossing (crossing railing and crossing obstacle). The findings of the study have some practical implications for safety management, especially workers' behavior monitoring and management.

Synergistic Integration of Skeletal Kinematic Features for Vision-Based Fall Detection.

According to the World Health Organisation, falling is a major health problem with potentially fatal implications. Each year, thousands of people die as a result of falls, with seniors making up 80% of these fatalities. The automatic detection of falls may reduce the severity of the consequences. Our study focuses on developing a vision-based fall detection system. Our work proposes a new feature descriptor that results in a new fall detection framework. The body geometry of the subject is analyzed and patterns that help to distinguish falls from non-fall activities are identified in our proposed method. An AlphaPose network is employed to identify 17 keypoints on the human skeleton. Thirteen keypoints are used in our study, and we compute two additional keypoints. These 15 keypoints are divided into five segments, each of which consists of a group of three non-collinear points. These five segments represent the left hand, right hand, left leg, right leg and craniocaudal section. A novel feature descriptor is generated by extracting the distances from the segmented parts, angles within the segmented parts and the angle of inclination for every segmented part. As a result, we may extract three features from each segment, giving us 15 features per frame that preserve spatial information. To capture temporal dynamics, the extracted spatial features are arranged in the temporal sequence. As a result, the feature descriptor in the proposed approach preserves the spatio-temporal dynamics. Thus, a feature descriptor of size [m×15] is formed where m is the number of frames. To recognize fall patterns, machine learning approaches such as decision trees, random forests, and gradient boost are applied to the feature descriptor. Our system was evaluated on the UPfall dataset, which is a benchmark dataset. It has shown very good performance compared to the state-of-the-art approaches.

Exploring Learning Curves in Acupuncture Education Using Vision-Based Needle Tracking

Measuring learning curves allows for the inspection of the rate of learning and competency threshold for each individual, training lesson, or training method. In this work, we investigated learning curves in acupuncture needle manipulation training with continuous performance measurement using a vision-based needle training system. We tracked the needle insertion depth of 10 students to investigate their learning curves. The results show that the group-level learning curve was fitted with the Thurstone curve, indicating that students were able to improve their needle insertion skills after repeated practice. Additionally, the analysis of individual learning curves revealed valuable insights into the learning experiences of each participant, highlighting the importance of considering individual differences in learning styles and abilities when designing training programs.

Robust real-time upper limb motion transfer from human to humanoid for manipulation skill teaching

This paper presents a framework for real-time high-fidelity upper limb motion transfer from a human to a humanoid robot, intending to provide robust support for robots to learn manipulation skills. Robot imitation of human motion is modeled as a simultaneous wrist and elbow tracking problem, where the coordination between multiple imitation objectives and physical constraints receives special attention. First, for the vision-based motion capture system, an efficient preprocessing method is introduced to map the raw human skeletal points analogously or reflectively to the robot’s workspace. Then, the generation of velocity-level imitation motion commands is formalized as a constrained hierarchical quadratic programing (HQP) problem, in which multiple objectives for achieving imitation are organized into a hierarchy. The wrist tracking task is set as the highest priority, followed by the adjustment of the elbow and the minimization of the joint velocity. Physical limits of the robot joints, as well as strict kinematic constraints regarding self-motion, are considered as constraints for the optimization problem. Finally, a hierarchical recurrent neural network (RNN) is designed to solve this HQP problem through its unique neural dynamics. The performance of this network is theoretically guaranteed with high computational efficiency. Quantitative results show that the proposed motion imitation control scheme offers excellent integrated tracking accuracy of [Formula: see text] m and is robust to different arm length ratios. Using this method, the humanoid robot can imitate the upper body motions of a human as accurately as possible under physical constraints, without significant delays.

Computer vision-based smart HVAC control system for university classroom in a subtropical climate

To respond to the increasing demand for a comfortable, productive and energy efficient study environment, the application of artificial intelligence technologies in the smart control of Heating, ventilation, and air conditioning (HVAC) systems plays an increasingly important role. This research uses a classroom, equipped with a traditional central HVAC system, in Hong Kong as a case study to demonstrate an innovative approach for a more intelligent and efficient HVAC system. Through a field investigation (i.e. measurement and questionnaire) and Computational Fluid Dynamics (CFD) simulation, it is found that the number and spatial location of students have a significant impact on their thermal comfort. Applying a computer vision model (YOLOv5) detected dynamic occupant information (variations in student numbers and locations) in a classroom, the SimScale (a cloud-native simulation platform) was then used to estimate the current thermal comfort state (predicted mean vote, PMV) and change in PMV (ΔPMV) of students in the classroom. Furthermore, a fuzzy logic control system is implemented to adjust air temperature and air velocity based on the simulation results. Preliminary scenario analysis has proven the feasibility of the proposed smart HVAC system for classrooms, as well as its ability to provide better quality of thermal comfort with more robust control. This study contributes to the smart and low-carbon retrofitting of university buildings with traditional central HVAC systems, while also serving as a benchmark for the energy-efficient transformation of HVAC systems in other types of indoor spaces.

Ambient Monitoring of Gait and Machine Learning Models for Dynamic and Short-Term Falls Risk Assessment in People With Dementia

Falls are a leading cause of morbidity and mortality in older adults with dementia residing in long-term care. Having access to a frequently updated and accurate estimate of the likelihood of a fall over a short time frame for each resident will enable care staff to provide targeted interventions to prevent falls and resulting injuries. To this end, machine learning models to estimate and frequently update the risk of a fall within the next 4 weeks were trained on longitudinal data from 54 older adult participants with dementia. Data from each participant included baseline clinical assessments of gait, mobility, and fall risk at the time of admission, daily medication intake in three medication categories, and frequent assessments of gait performed via a computer vision-based ambient monitoring system. Systematic ablations investigated the effects of various hyperparameters and feature sets and experimentally identified differential contributions from baseline clinical assessments, ambient gait analysis, and daily medication intake. In leave-one-subject-out cross-validation, the best performing model predicts the likelihood of a fall over the next 4 weeks with a sensitivity and specificity of 72.8 and 73.2, respectively, and achieved an area under the receiver operating characteristic curve (AUROC) of 76.2. By contrast, the best model excluding ambient gait features achieved an AUROC of 56.2 with a sensitivity and specificity of 51.9 and 54.0, respectively. Future research will focus on externally validating these findings to prepare for the implementation of this technology to reduce fall and fall-related injuries in long-term care.

Hybrid Visual-Ranging Servoing for Positioning Based on Image and Measurement Features.

In this article, a hybrid visual-ranging servoing method is proposed to realize high-precision positioning tasks with a 6-degree of freedom (DOF) manipulator. This method utilizes the image and measurement features directly in the control loop. Without the need of complex image feature design and attitude estimation, this method realizes the 6-DOF control of a robot. A vital challenge in traditional vision-based systems is avoiding local minima and singularity problems. To tackle this issue, a full-rank interaction matrix hybrid visual servo (FRHVS) design criterion is proposed, which guarantees that the hybrid interaction matrix and its pseudoinverse matrix are both full rank. Moreover, the interaction matrix for these hybrid strategies, which combines image features with other sensors features, is derived in an analytical form. Experiments on a 6-DOF manipulator show that the proposed method is effective and has global asymptotic stability and high precision.

Lightweight image de-snowing: A better trade-off between network capacity and performance

The single image de-snowing task is an essential topic in computer vision, as images captured on snowy days degrade the performance of current vision-based intelligent systems. Existing methods build complex network structures with numerous parameters to pursue continuous performance improvement. Nonetheless, they generally ignore the negative impact of large memory consumption in real applications. This paper aims to address the above problem by making a trade-off between network capacity and performance. We propose two novel networks suitable for different application scenarios. For devices with small memory and requiring fast inference speed, we propose an extremely lightweight recursive network (XLRNet). XLRNet is constructed by a single recursive strategy and two novel lightweight modules. For devices with large memory and pursuing better de-snowing performance, we propose a coupled lightweight dual recursive network (CLDRNet). CLDRNet cascades two XLRNets by a novel dual recursive strategy and a novel dual coupled LSTM module (DC-LSTM). Extensive experiments demonstrate the effectiveness and superiority of our two models on three synthetic datasets and real-world datasets.

A Computer Vision-Based System for Recognition and Classification of Urdu Sign Language Dataset for Differently Abled People Using Artificial Intelligence

Communication between normal people and deaf people is the most difficult part of daily life worldwide. It is difficult for a normal person to understand a word from the deaf one in their daily routine. So, to communicate with deaf people, different countries developed different sign languages to make communication easy. In Pakistan, for deaf people, the government developed Urdu Sign Language to communicate with deaf people. Physical trainers and experts are difficult to provide everywhere in society, so we need such a computer/mobile-based system to convert the deaf sign symbol into voice and written alphabet that the normal person can easily get the intentions of the deaf one. In this paper, we provided an image processing and deep learning-based model for Urdu Sign Language. The proposed model is implemented in Python 3 and uses different image processing and machine techniques to capture the video and transform the symbols into voice and Urdu writing. First, we get a video from the deaf person, and then the model crops the frames into pictures. Then, the individual picture is recognized for the sign symbol such as if the deaf showed a symbol for one, then the model recognizes it and shows the letter which he/she wants to tell. Image processing techniques such as OpenCV are used for image recognition and classification while TensorFlow and linear regression are used for training the model to behave intelligently in the future. The results show that the proposed model increased accuracy from 80% to 97% and 100% accordingly. The accuracy of the previously available work was 80% when we implemented the algorithms, while with the proposed algorithm, when we used linear regression, we achieved the highest accuracy. Similarly, when we used the TensorFlow deep learning algorithm, we achieved 97% accuracy which was less than that of the linear regression model.

Fruit Detection and Counting in Apple Orchards Based on Improved Yolov7 and Multi-Object Tracking Methods.

With the increasing popularity of online fruit sales, accurately predicting fruit yields has become crucial for optimizing logistics and storage strategies. However, existing manual vision-based systems and sensor methods have proven inadequate for solving the complex problem of fruit yield counting, as they struggle with issues such as crop overlap and variable lighting conditions. Recently CNN-based object detection models have emerged as a promising solution in the field of computer vision, but their effectiveness is limited in agricultural scenarios due to challenges such as occlusion and dissimilarity among the same fruits. To address this issue, we propose a novel variant model that combines the self-attentive mechanism of Vision Transform, a non-CNN network architecture, with Yolov7, a state-of-the-art object detection model. Our model utilizes two attention mechanisms, CBAM and CA, and is trained and tested on a dataset of apple images. In order to enable fruit counting across video frames in complex environments, we incorporate two multi-objective tracking methods based on Kalman filtering and motion trajectory prediction, namely SORT, and Cascade-SORT. Our results show that the Yolov7-CA model achieved a 91.3% mAP and 0.85 F1 score, representing a 4% improvement in mAP and 0.02 improvement in F1 score compared to using Yolov7 alone. Furthermore, three multi-object tracking methods demonstrated a significant improvement in MAE for inter-frame counting across all three test videos, with an 0.642 improvement over using yolov7 alone achieved using our multi-object tracking method. These findings suggest that our proposed model has the potential to improve fruit yield assessment methods and could have implications for decision-making in the fruit industry.

A fuzzy logic-based stabilization system for a flying robot, with an embedded energy harvester and a visual decision-making system

“Smart cities” is a popular concept in modern urban development that demands innovative solutions to enhance various aspects of our lives. This paper introduces a novel application that aims to improve public road maintenance by utilizing a flying robot to repaint partially erased sections of the sidewalks’ edges that are typically marked with black and white colors. The first contribution of this paper is the development of a fuzzy-logic-based stabilization system for an octocopter, which can serve as a liquids transporter and be equipped with a robot arm. The second contribution is the design of an embedded energy harvester for the flying robot, which optimizes the management of available power sources. Additionally, this project includes a complementary heuristic study that clarifies fundamental concepts related to a computer vision-based decision-making system.