Vision-based System Research Articles

Real time two-wheeler recognition for Surveillance

In today’s intelligent transportation systems, vision-based traffic tracking systems and automatic vehicle make and model recognition is of significant importance for surveillance. From the literature, vehicle make and model recognition has been implemented for four wheelers easily due to the presence of logo in the frontal and rear parts. But it is not feasible for two wheelers due to the insignificant differential features. Hence in this work, a robust two-wheeler recognition system has been developed with frontal, rear and side images of two wheelers which is implementable in edge devices. The contributions of the paper are (i) Development of a two-wheeler dataset with the frontal, rear and side images of two classes in Indian Scenario. (ii) Development of a two-wheeler recognition framework implementable in Raspberry pi-based devices. (iii) Evaluation of the framework in real time outdoor environment. From the experimental results it is observed that the proposed network classifies two wheelers in real time at 4s with 94% accuracy.

Deep Neural Network-Based Dynamical Object Recognition and Robust Multiobject Tracking Technique for Onboard Unmanned Aerial Vehicle’s Computer Vision-Based Systems

Computer vision-based systems seem highly perspective for semantic analysis of the dynamical objects. However, considering dynamical object recognition and tracking from the unmanned aerial vehicle the task to design a robust model for data association is highly challenging due to additional issues, e.g., image degradation, non-fixed object camera distance and shooting focus, and real-time issues. Thus, we propose an accurate deep neural network-based dynamical object recognition and robust multi-object tracking technique based on bidirectional LSTM with the optimized motion and appearance gates as a multi-object tracking backbone, supported by an advanced single-shot detector network improved with residual prediction model and implemented DenseNet network as well as YOLOv4eff network as feature extraction. The technique has been trained on VisDrone 2022 and UAVDT datasets with the side-shoot dynamical objects at a height of up to 50 meters. The performance analysis on the test stage performed on 7 metrics demonstrate that the proposed technique surpasses by accuracy and robustness ability of other state-of-the-art techniques based on 2 cumulative MOTA and MOTP, as well as MT and IDsw. In particular, we have dramatically decreased the number of IDsw which implies a better capability to handle several occlusions, which is a desirable property in real-time multiple object tracking. We have pointed out the sensitivity of the tracking performance of our technique on the number of utilizing different sequence lengths and have defined an optimum. Finally, the applicability and reliability of the proposed technique for onboard UAV computer-based systems have been discussed.

A vision-based robotic system following the human upper-limb sewing action

Abstract. In human–robot collaborative sewing, the robot follows the sewing action of a worker to complete the corresponding sewing action, which can enhance production efficiency. When the robot follows the sewing action of the worker through interactive information, it still faces the problem of low accuracy. In order to improve the accuracy of the robot following the sewing action, a human upper-limb sewing-action-following system based on visual information is designed in this paper. The system is composed of an improved OpenPose model, Gaussian mixture model (GMM), and Gaussian mixture regression (GMR). In the system, an improved OpenPose model is used to identify the sewing action of the human upper limb, and the label fusion method is used to correct the joint point labels when the upper limb is covered by fabric. Then the GMM is used to encode each motion element and time to obtain the regression work of the Gaussian component. GMR is adopted to predict connections between moving elements and generate sewing motion trajectories. Finally, the experimental verification and simulation are carried out in the experimental platform and simulation environment of the collaborative robot. The experimental results show that the tracking error angle can be controlled within 0.04 rad in the first 2 s of robot movement. Therefore, it can be considered that the sewing-action-following system can realize higher precision and promote the development of human–robot collaboration technology to a certain extent.

Real-time reading system for pointer meter based on YolactEdge

Despite the extensive deployment of digital instruments in modern times, their stability is challenging to maintain in adverse environmental conditions such as extreme temperatures, pressure, or powerful electromagnetic radiation. Analog meters, owing to their mechanical resilience and electromagnetic impedance, persistently find usage across nuclear power plants, petroleum, and chemical industries. However, under these harsh conditions, manual reading of the instruments may prove to be difficult and dangerous while failing to meet the requirements of real-time monitoring. In recent years, several machine vision-based meter reading systems have been proposed, however, achieving high accuracy through camera-based methods under varying angles and lighting conditions poses significant challenges. Cloud deployment may compromise plant privacy, while edge computing faces limitations in real-time meter reading due to limited computing power. To address these issues, we propose a real-time reading system based on the YolactEdge instance segmentation framework for single-point analog meters. Our system is more accurate than previous studies and is implemented and deployed on the Jetson Xavier NX edge computing device. Our performance evaluation shows that our model outperforms other baselines, with low reference values and relative errors of 0.0237% and 0.0300%, respectively, and an average inference speed of 10.26 FPS with INIT 8 linear acceleration on Nvidia Jetson NX.

Real-time computer vision-based gestures recognition system for bangla sign language using multiple linguistic features analysis

Real-time computer vision-based gestures recognition system for bangla sign language using multiple linguistic features analysis

Automatic Inspection System for Segregation of Defective Parts of Heavy Vehicles

Defect Detection is a crucial process in the manufacturing industry. Most of the manufacturing parts tend to get scratches, dents, etc. on their surface during the manufacturing process. Such parts are defective and are not acceptable for further use. So, it is essential to inspect parts before they can be dispatched further. To detect such defects, a team of skilled persons is deployed; which does manual visual inspection of parts to detect those defects. In manual inspection, chances of missing small defects are pretty high. Not only that but it also takes a considerable amount of time. Moreover, the job is tedious and monotonous creating strain on the eyes of members of the inspection team. Some defects are even not visible to the naked eye. To make the process of visual inspection simple and easy, a deep learning Convolutional neural network (CNN) based model is proposed. Reason behind choosing a convolutional neural network is its capability to extract features efficiently as the task accuracy depends upon this capability. A custom dataset has been prepared with utmost care of illumination conditions, resolution of image, etc., so that a clear picture of parts with minute details can be captured. The model has been trained using the dataset thus prepared and containing 960 images and it is observed that it provides a good accuracy of 95%. The same model is deployed using an embedded board with a Jetson Nano processor to prepare a computer vision-based inspection system that can be used to differentiate between perfect and imperfect parts and thus help skilled personnel in the inspection process.

Visual-Based Children and Pet Rescue from Suffocation and Incidence of Hyperthermia Death in Enclosed Vehicles.

Over the past several years, many children have died from suffocation due to being left inside a closed vehicle on a sunny day. Various vehicle manufacturers have proposed a variety of technologies to locate an unattended child in a vehicle, including pressure sensors, passive infrared motion sensors, temperature sensors, and microwave sensors. However, these methods have not yet reliably located forgotten children in the vehicle. Recently, visual-based methods have taken the attention of manufacturers after the emergence of deep learning technology. However, the existing methods focus only on the forgotten child and neglect a forgotten pet. Furthermore, their systems only detect the presence of a child in the car with or without their parents. Therefore, this research introduces a visual-based framework to reduce hyperthermia deaths in enclosed vehicles. This visual-based system detects objects inside a vehicle; if the child or pet are without an adult, a notification is sent to the parents. First, a dataset is constructed for vehicle interiors containing children, pets, and adults. The proposed dataset is collected from different online sources, considering varying illumination, skin color, pet type, clothes, and car brands for guaranteed model robustness. Second, blurring, sharpening, brightness, contrast, noise, perspective transform, and fog effect augmentation algorithms are applied to these images to increase the training data. The augmented images are annotated with three classes: child, pet, and adult. This research concentrates on fine-tuning different state-of-the-art real-time detection models to detect objects inside the vehicle: NanoDet, YOLOv6_1, YOLOv6_3, and YOLO7. The simulation results demonstrate that YOLOv6_1 presents significant values with 96% recall, 95% precision, and 95% F1.

Vision-Based System for Black Rubber Roller Surface Inspection

This paper proposes a machine vision system for the surface inspection of black rubber rollers in manufacturing processes. The system aims to enhance the surface quality of the rollers by detecting and classifying defects. A lighting system is installed to highlight surface defects. Two algorithms are proposed for defect detection: a traditional-based method and a deep learning-based method. The former is fast but limited to surface defect detection, while the latter is slower but capable of detecting and classifying defects. The accuracy of the algorithms is verified through experiments, with the traditional-based method achieving near-perfect accuracy of approximately 98% for defect detection, and the deep learning-based method achieving an accuracy of approximately 95.2% for defect detection and 96% for defect classification. The proposed machine vision system can significantly improve the surface inspection of black rubber rollers, thereby ensuring high-quality production.

Vision-based dirt distribution mapping using deep learning

Cleaning is a fundamental routine task in human life that is now handed over to leading-edge technologies such as robotics and artificial intelligence. Various floor-cleaning robots have been developed with different cleaning functionalities, such as vacuuming and scrubbing. However, failures can occur when a robot tries to clean an incompatible dirt type. These situations will not only reduce the efficiency of the robot but also impose severe damage to the robots. Therefore, developing effective methods to classify the cleaning tasks performed in different regions and assign them to the respective cleaning agent has become a trending research domain. This article proposes a vision-based system that employs YOLOv5 and DeepSORT algorithms to detect and classify dirt to create a dirt distribution map that indicates the regions to be assigned for different cleaning requirements. This map would be useful for a collaborative cleaning framework for deploying each cleaning robot to its respective region to achieve an uninterrupted and energy-efficient operation. The proposed method can be executed with any mobile robot and on any surface and dirt, achieving high accuracy of 81.0%, for dirt indication in the dirt distribution map.

Motion capture based calibration for industrial robots

Path accuracy of industrial robotic systems is of paramount importance for manufacturing applications. Many robot vendors provide guarantees of locational repeatability for robot arms down to 0.01 mm, however, this level of precision is meaningless if the system is not well calibrated. In many cases, where an offline programming environment is used, the physical system needs to match the behavior of the digital twin with minimal error. Calibration is thus required to ensure that the position and motion of the robot matches that of what was programmed. One approach is to use relative coordinate systems to offset the programmed path to account for any consistent deviations across the path. These offset coordinate systems can be created using techniques such as touch-based, laser-based, and/or vision-based systems. The offset coordinate system approach is used by major robot manufacturers and NC machines to arrive at a well-defined path. However, this approach assumes that the workspace is uniformly shifted, which is not always the case for industrial robots. Such systems can suffer from distortion in their coordinate space due to mechanical flexure, gearing backlash, and other factors that distort their workspace. To correct this distortion, we propose the use of a commercially available motion capture system that precisely monitors and reports the location of a Yaskawa industrial robotic arm. The positional and rotational information of the robot end-effector can be captured using this motion capture system and compared with that of the expected TCP (Tool Center Point) location derived from the forward kinematics model. The difference between these coordinates can then be used to determine and correct offsets from paths created using the digital twin. With motion capture driven path corrections, the accuracy of offline programed robots can be greatly improved without the need for exhaustively complex calibration processes.

Visual teach and generalise (VTAG)—Exploiting perceptual aliasing for scalable autonomous robotic navigation in horticultural environments

Nowadays, most agricultural robots rely on precise and expensive localisation, typically based on global navigation satellite systems (GNSS) and real-time kinematic (RTK) receivers. Unfortunately, the precision of GNSS localisation significantly decreases in environments where the signal paths between the receiver and the satellites are obstructed. This precision hampers deployments of these robots in, e.g., polytunnels or forests. An attractive alternative to GNSS is vision-based localisation and navigation. However, perceptual aliasing and landmark deficiency, typical for agricultural environments, cause traditional image processing techniques, such as feature matching, to fail. We propose an approach for an affordable pure vision-based navigation system which is not only robust to perceptual aliasing, but it actually exploits the repetitiveness of agricultural environments. Our system extends the classic concept of visual teach and repeat to visual teach and generalise (VTAG). Our teach and generalise method uses a deep learning-based image registration pipeline to register similar images through meaningful generalised representations obtained from different but similar areas. The proposed system uses only a low-cost uncalibrated monocular camera and the robot’s wheel odometry to produce heading corrections to traverse crop rows in polytunnels safely. We evaluate this method at our test farm and at a commercial farm on three different robotic platforms where an operator teaches only a single crop row. With all platforms, the method successfully navigates the majority of rows with most interventions required at the end of the rows, where the camera no longer has a view of any repeating landmarks such as poles, crop row tables or rows which have visually different features to that of the taught row. For one robot which was taught one row 25 m long our approach autonomously navigated the robot a total distance of over 3.5 km, reaching a teach-generalisation gain of 140.

Mixed Reality for Safe and Reliable Human-Robot Collaboration in Timber Frame Construction

In the field of construction, human-robot collaboration and mixed reality (MR) open new possibilities. However, safety and reliability issues persist. The lack of flexibility and adaptability in current preprogrammed systems hampers real-time human-robot collaboration. A key gap in this area lies in the ability of the robot to interpret and accurately execute operations based on the real-time visual instructions and restrictions provided by the human collaborator and the working environment. This paper focuses on an MR-based human-robot collaboration method through visual feedback from a vision-based collaborative industrial robot system for use in wood stereotomy which we are developing. This method is applied to an alternating workflow in which a skilled carpenter lays out the joinery on the workpiece, and the robot cuts it. Cutting operations are instructed to the robot only through lines and conventional “carpenter’s marks”, which are drawn on the timbers by the carpenter. The robot system’s accuracy in locating and interpreting marks as cutting operations is evaluated by automatically constructing a 3D model of the cut shape from the vision system data. A digital twin of the robot allows the carpenter to previsualize all motions that are required by the robot for task validation and to know when to enter the collaborative workspace. Our experimental results offer some insights into human-robot communication requirements for collaborative robot system applications in timber frame construction.

Development of image-based fast defect recognition and localization network (FDRLNet) for steel surfaces

The utilization of vision-based systems for automated inspection and quality control tasks increased in recent years due to the adaptation of Industry 4.0 initiatives in manufacturing. Continuous steel production units extensively employ vision-based solutions to classify and localize surface defects. The inspection system must provide quick and reliable feedback about surface defect type (classification) and location (localization) utilizing images acquired from the camera. This paper presents a novel surface defect detector, Fast Defect Recognition and Localization Network (FDRLNet), for achieving accurate prediction abilities at higher inference speeds. The robustness of the proposed approach is validated by utilizing publicly available Northeastern University (NEU-DET) surface defect dataset. The prediction abilities are corroborated by comparing the mean Average Precision (mAP) and inference speed with other competitive approaches presented in the literature. It has been shown that the proposed approach has robust prediction abilities at higher inference speeds and can be implemented in real-time defect detection tasks.

Comprehensive review on vehicle Detection, classification and counting on highways

Vehicle detection, counting and finally classification has been an important aspect of traffic analysis specially on highways in many developed and developing nations. This has vitalized the monitoring of freeways and reduced the reliance on human traffic monitors specially in developed nations. However, much research carried out in this regard since 1995 have been slow to progress until around 2000. Since then, much more encouraging outcomes have been achieved. This has been mainly due to the advances on vision-based computing and the miniaturizing of hardware since the early 2000. Initial vision-based systems used basic computer vision approaches such as background subtraction and edge detection to detect and count vehicles. Early progress of classification had little success until around 2010. Since 2010, many computer vision approaches using Neural Networks have been gradually increasing the efficiency and the realtime operation of such systems. Since then, many deep learning based neural network approaches have produced remarkable outcomes specially in classification. We have also reported highly accurate and efficient deep learning based YOLOv5 system that has recorded an astounding success. We have used models YOLOv5l, YOLOv5m, YOLOv5n and YOLOv5s in our research to ascertain the best model for the task.However, low light conditions associated with overcast, and dusk have significantly reduced the accuracy in many deep learning-based systems. There are other techniques based on approaches such as headlight detection have appeared to increase the accuracy of counting however, classification at night without adequate lighting still pauses a formidable challenge. The work presented here analyses the vehicle counting problem using computer vision providing many different viewpoints in a view to provide the reader with accurate problem definition.

Development of Obstacle Detection System Based On the Integration of Different Based Sensor for Small-Sized UAV

Due to the physical size and weight limits of small unmanned aerial vehicles (UAVs), developing a reliable obstacle detection a system that can provide an effective and safe avoidance path is extremely difficult. Prior work has tended to use a vision-based sensor as the primary detecting sensor however, this has resulted in a high reliance on texture appearance and a lack of distance sensing capabilities. Furthermore, due to the inability to detect the free region, vision-based sensor detection systems have difficulty developing a trusted safe avoidance path. However, most wide spectrum range sensors are bulky and expensive, making them unsuitable for small UAVs. This projectaims to construct an obstacles detection system with the integration of various based sensors for a small UAV. The potential obstacles are identified by categorizing feature points identified in image frame. The suggested approach was tested in a real-world setting for both of the observed scenarios, which included various obstacles configurations. Two types of scenarios are experimented in this project consists of single frontal obstacles and presence of side obstacles alongside the frontal obstacles. On top of that, the position of the side obstacle is aligned to the frontal obstacle and then will be positioned in the increment of 20cm further from the frontal obstacle in order to analyze the outcome of the proposed algorithm. The proposed detection system had a possibility to be a trustworthy system even after utilizing the depth perception technique, however this does not imply that the proposed system is faultless. The results show that the suggested algorithm system detects and distinguishes between the potential obstacles and free region for a single frontal obstacle perfectly. However, there were improvements that should be implemented with the proposed system's ability of detection for multiple obstacles.

Editorial

Dear Readers,  Welcome to the seventh issue in 2023. I am very pleased to announce the journal’s continued high Scopus CiteScore of 2.7 and Web of Science Impact Factor of 1.0 for 2022, indicating another scientifically successful year. On behalf of the J.UCS team, I would like to thank all the authors for their sound research contributions, the reviewers for their very helpful suggestions for improvements, and the consortium members for their financial support. Your commitment and dedicated work have contributed significantly to the long-lasting success of our journal.  As we want to secure the financial support also for the years to come, we are looking for institutions and libraries to financially support our diamond open access journal as consortium members, who will then benefit from the research community, international visibility, and the opportunity to manage special issues and focused topics within the journal. Please think about the possibility of such financial participation of your institution, we would be very grateful for any kind of support.  In this regular issue, I am very pleased to introduce six accepted papers from seven different countries and 20 involved authors.  Ana P. Allian, Leandro F. Silva, Edson OliveiraJr and Elisa Y. Nakagawa from Brazil present VMTools-RA, a reference architecture that encompasses the knowledge and practice for developing and evolving variability tools. In a collaboration between researchers from the UK and Estonia, Vimal Dwivedi, Mubashar Iqbal, Alex Norta and Raimundas Matulevičius are focusing their research on the evaluation of a legally binding smart-contract language for blockchain applications. Monika, Seema Verma, and Pardeep Kumar from India discuss an intelligent vision-based decision-making system for the exploration of past aviation accidents and incidents, which is based on a visual query-based model capable of analyzing the major factors including flight phases, human factors, weather conditions, and faulty components in particular aircraft models. Luis Eduardo Ordoñez Palacios, Víctor Bucheli Guerrero and Hugo Ordoñez from Colombia present their research on integrating satellite imagery and meteorological data to estimate solar radiation applying and evaluating five machine learning models. In a collaboration between researchers from Palestine and Egypt, Muath Sabha, Thaer Thaher, and Marwa M. Emam apply cooperative swarm intelligence algorithms to adaptive multilevel thresholding segmentation of COVID-19 CT scan images. Geovana Ramos Sousa Silva, Genaína Nunes Rodrigues and Edna Dias Canedo from Brazil introduce their work on a modeling strategy to design and verify chatbot conversational flows via the Uppaal model checking tool.  Enjoy Reading! Cordially, Christian Gütl, Managing Editor 

Intelligent Vision Based Decision Making System for Aviation Accidents and Incidents

Safety has become the primary concern for the air transportation system nowadays primarily due to increasing air traffic throughout the world. Various regulatory bodies have been maintaining enormous amount of aviation accidental data repositories. This past data is highly complex because of its many temporal and geographical components along with multiple variables. To be able to analyze this past data, there is always a need of user friendly and GUI based System. This article has proposed an intelligent vision-based decision-making system for the exploration of past aviation accidents and incidents dataset. The proposed visual query-based model is capable to analyse the major factors like flight phases, human factors, weather conditions and faulty components in particular aircraft models which are responsible for those unsafe events and may claim life of many passengers who are traveling and crew personnels. This model enables the users to express “what” visuals should be created instead of “how” to create them. Various case studies conducted through visual queries have proved that the system will be highly able to improve situational awareness regarding flight conditions to the crew members and air traffic controllers along with aviation authorities so that they are able to take timely decisions and deciding on what kind of training staff members need to reduce the consequences of such accidents and incidents.

Automatic Outdoor Fire detection using Deep learning

Abstract
 One of the unlucky phenomena that contribute to environmental disasters is fire, which also poses a serious threat to human safety and life, particularly when it is not recognized by sensor-based fire detection systems. Therefore, putting inexpensive and efficient sensors in certain locations will greatly speed up the detection of fires. Vision-based fire detection systems take advantage of the three fundamental features of fire: color, movement, and shape (fire shape). Work has been done to build smoke and fire detection systems based on images that use security cameras. In this study, fire photos are used to extract features before inputs are made. To one of the classification techniques that applies SVM-based machine learning. and CNN-based deep learning technique. We use the linear regression model so that the bounding box of the classified object determines the correct coordinates.

A Computer Vision-Based Automatic System for Egg Grading and Defect Detection.

Defective eggs diminish the value of laying hen production, particularly in cage-free systems with a higher incidence of floor eggs. To enhance quality, machine vision and image processing have facilitated the development of automated grading and defect detection systems. Additionally, egg measurement systems utilize weight-sorting for optimal market value. However, few studies have integrated deep learning and machine vision techniques for combined egg classification and weighting. To address this gap, a two-stage model was developed based on real-time multitask detection (RTMDet) and random forest networks to predict egg category and weight. The model uses convolutional neural network (CNN) and regression techniques were used to perform joint egg classification and weighing. RTMDet was used to sort and extract egg features for classification, and a Random Forest algorithm was used to predict egg weight based on the extracted features (major axis and minor axis). The results of the study showed that the best achieved accuracy was 94.8% and best R2 was 96.0%. In addition, the model can be used to automatically exclude non-standard-size eggs and eggs with exterior issues (e.g., calcium deposit, stains, and cracks). This detector is among the first models that perform the joint function of egg-sorting and weighing eggs, and is capable of classifying them into five categories (intact, crack, bloody, floor, and non-standard) and measuring them up to jumbo size. By implementing the findings of this study, the poultry industry can reduce costs and increase productivity, ultimately leading to better-quality products for consumers.

Stereo SLAM in Dynamic Environments Using Semantic Segmentation

As we all know, many dynamic objects appear almost continuously in the real world that are immensely capable of impairing the performance of the majority of vision-based SLAM systems based on the static-world assumption. In order to improve the robustness and accuracy of visual SLAM in high-dynamic environments, a real-time and robust stereo SLAM system for dynamic scenes was proposed. To weaken the influence of dynamic content, the moving-object detection method was put forward in our visual odometry, and then the semantic segmentation network was combined in our stereo SLAM to extract pixel-level contours of dynamic objects. Then, the influences of dynamic objects were significantly weakened and the performance of our system increased markedly in dynamic, complex, and crowed city spaces. Following experiments with both the KITTI Odometry dataset and in a real-life scene, the results showed that our method could dramatically decrease the tracking error or drift, and improve the robustness and stability of our stereo SLAM in high dynamic outdoor scenarios.