Integrated Machine Vision Research Articles

Maize quality detection based on MConv-SwinT high-precision model.

The traditional method of corn quality detection relies heavily on the subjective judgment of inspectors and suffers from a high error rate. To address these issues, this study employs the Swin Transformer as an enhanced base model, integrating machine vision and deep learning techniques for corn quality assessment. Initially, images of high-quality, moldy, and broken corn were collected. After preprocessing, a total of 20,152 valid images were obtained for the experimental samples. The network then extracts both shallow and deep features from these maize images, which are subsequently fused. Concurrently, the extracted features undergo further processing through a specially designed convolutional block. The fused features, combined with those processed by the convolutional module, are fed into an attention layer. This attention layer assigns weights to the features, facilitating accurate final classification. Experimental results demonstrate that the MC-Swin Transformer model proposed in this paper significantly outperforms traditional convolutional neural network models in key metrics such as accuracy, precision, recall, and F1 score, achieving a recognition accuracy rate of 99.89%. Thus, the network effectively and efficiently classifies different corn qualities. This study not only offers a novel perspective and technical approach to corn quality detection but also holds significant implications for the advancement of smart agriculture.

Online stress monitoring during laser-directed energy deposition based on dynamic contour method

ABSTRACT Deformation and cracking caused by internal stress have been a long-standing challenge in the field of metal additive manufacturing. This paper presents a novel method for real-time stress assessment of laser-directed energy deposition (LDED) based on the shrinkage phenomenon of deposition layer – the Dynamic Contour Method (DCM). It integrates machine vision, three-dimensional reconstruction based on actual morphology, and numerical simulation to calculate rapidly stress development during the LDED process. Meanwhile, a mapping relationship between the surface shrinkage of the deposition layer and stress is established, providing a theoretical basis for the DCM. Regarding the validation of this method, the DCM simulations are compared with the experimentally calibrated thermo-mechanical coupling simulations. The results show a high degree of consistency, demonstrating the feasibility and accuracy of the DCM. This method provides a new digital twin framework for additive manufacturing.

Integrating machine vision in the development of automatic mounting system for massive electroplating components

Abstract The traditional industrial electroplating parts manufacturing process typically relies on manual labour to suspend them in the electroplating tank, requiring a significant amount of manual work. However, due to a shortage of manpower and frequent staff turnover, this has posed challenges to production efficiency. Therefore, this research proposes an automated mounting system by integrating machine vision technology and the development of robotic arms to replace the conventional manual suspension method. The automated mounting system initially separates individual ring-shaped electroplating components using a disk vibratory feeder. Subsequently, it utilizes a unidirectional reciprocating mechanism and infrared detection to position these electroplating components onto a backlit panel. Then, the IDS uEye XS2 autofocus camera captures images, and in conjunction with machine vision technology, rapidly determines the position, characteristics, and orientation of these ring components. Finally, through TCP communication protocol, commands for gripping points are transmitted to the robotic arm, which uses electromagnetic valves on the arm to pick up the components and suspend them onto the hanger using an optimized path. The final test results demonstrate that the automated installation system can complete 14 installation operations per minute, effectively reducing the time and cost associated with manual labour, significantly enhancing the production output of the assembly line. Therefore, this study combines machine vision technology to propose a solution for the automated installation system of large-scale electroplating components.

Tool wear monitoring based on scSE-ResNet-50-TSCNN model integrating machine vision and force signals

In the realm of mechanical machining, tool wear is an unavoidable phenomenon. Monitoring the condition of tool wear is crucial for enhancing machining quality and advancing automation in the manufacturing process. This paper investigates an innovative approach to tool wear monitoring that integrates machine vision with force signal analysis. It relies on a deep residual two-stream convolutional model optimized with the scSE (concurrent spatial and channel squeeze and excitation) attention mechanism (scSE-ResNet-50-TSCNN). The force signals are converted into the corresponding wavelet scale images following wavelet threshold denoising and continuous wavelet transform. Concurrently, the images undergo processing using contrast limited adaptive histogram equalization and the structural similarity index method, allowing for the selection of the most suitable image inputs. The processed data are subsequently input into the developed scSE-ResNet-50-TSCNN model for precise identification of the tool wear state. To validate the model, the paper employed X850 carbon fibre reinforced polymer and Ti–6Al–4V titanium alloy as laminated experimental materials, conducting a series of tool wear tests while collecting pertinent machining data. The experimental results underscore the model’s effectiveness, achieving an impressive recognition accuracy of 93.86%. When compared with alternative models, the proposed approach surpasses them in performance on the identical dataset, showcasing its efficient monitoring capabilities in contrast to single-stream networks or unoptimized networks. Consequently, it excels in monitoring tool wear status and promots crucial technical support for enhancing machining quality control and advancing the field of intelligent manufacturing.

A machine vision-assisted Argonaute-mediated fluorescence biosensor for the detection of viable Salmonella in food without convoluted DNA extraction and amplification procedures

The precise identification viable pathogens hold paramount significance in the prevention of foodborne diseases outbreaks. In this study, we integrated machine vision and learning with single microsphere to develop a phage and Clostridium butyricum Argonaute (CbAgo)-mediated fluorescence biosensor for detecting viable Salmonella typhimurium (S. typhimurium) without convoluted DNA extraction and amplification procedures. Phage and lysis buffer was utilized to capture and lyse viable S. typhimurium, respectively. Subsequently, CbAgo can cleave the bacterial DNA to obtain target DNA that guides a newly targeted cleavage of fluorescent probes. After that, the resulting fluorescent signal accumulates on the streptavidin-modified single microsphere. The overall detection process is then analyzed and interpreted by machine vision and learning algorithms, achieving highly sensitive detection of S. typhimurium with a limit of detection at 40.5 CFU/mL and a linear range of 50-107 CFU/mL. Furthermore, the proposed biosensor demonstrates standard recovery rates and coefficients of variation at 93.22%-106.02% and 1.47%-12.75%, respectively. This biosensor exhibits exceptional sensitivity and selectivity, presenting a promising method for the rapid and effective detection of foodborne pathogens. ENVIRONMENTAL IMPLICATION: Bacterial pathogens exist widely in the environment and seriously threaten the safety of human life. In this study, we developed a phage and Clostridium butyricum Argonaute-mediated fluorescence biosensor for the detection of viable Salmonella typhimurium in environmental water and food samples. Compared with other Salmonella detection methods, this method does not need complex DNA extraction and amplification steps, which reduces the use of chemical reagents and experimental consumables in classic DNA extraction kit methods.

Transmission Tower Re-Identification Algorithm Based on Machine Vision

Transmission tower re-identification refers to the recognition of the location and identity of transmission towers, facilitating the rapid localization of transmission towers during power system inspection. Although there are established methods for the defect detection of transmission towers and accessories (such as crossarms and insulators), there is a lack of automated methods for transmission tower identity matching. This paper proposes an identity-matching method for transmission towers that integrates machine vision and deep learning. Initially, the method requires the creation of a template library. Firstly, the YOLOv8 object detection algorithm is employed to extract the transmission tower images, which are then mapped into a d-dimensional feature vector through a matching network. During the training process of the matching network, a strategy for the online generation of triplet samples is introduced. Secondly, a template library is built upon these d-dimensional feature vectors, which forms the basis of transmission tower re-identification. Subsequently, our method re-identifies the input images. Firstly, we propose that the YOLOv5n-conv head detects and crops the transmission towers in images. Secondly, images without transmission towers are skipped; for those with transmission towers, The matching network maps transmission tower instances into feature vectors. Ultimately, transmission tower re-identification is realized by comparing feature vectors with those in the template library using Euclidean distance. Concurrently, it can be combined with GPS information to narrow down the comparison range. Experiments show that the YOLOv5n-conv head model achieved a mean Average Precision at an Intersection Over Union threshold of 0.5 (mAP@0.5) score of 0.974 in transmission tower detection, reducing the detection speed by 2.4 ms compared to the original YOLOv5n. Integrating the online triplet sample generation into the matching network training with Inception-ResNet-v1 (d = 128) as the backbone enhanced the network’s rank-1 performance by 3.86%.

Design of a 3D emotion mapping model for visual feature analysis using improved Gaussian mixture models.

Given the integration of color emotion space information from multiple feature sources in multimodal recognition systems, effectively fusing this information presents a significant challenge. This article proposes a three-dimensional (3D) color-emotion space visual feature extraction model for multimodal data integration based on an improved Gaussian mixture model to address these issues. Unlike traditional methods, which often struggle with redundant information and high model complexity, our approach optimizes feature fusion by employing entropy and visual feature sequences. By integrating machine vision with six activation functions and utilizing multiple aesthetic features, the proposed method exhibits strong performance in a high emotion mapping accuracy (EMA) of 92.4%, emotion recognition precision (ERP) of 88.35%, and an emotion recognition F1 score (ERFS) of 96.22%. These improvements over traditional approaches highlight the model's effectiveness in reducing complexity while enhancing emotional recognition accuracy, positioning it as a more efficient solution for visual emotion analysis in multimedia applications. The findings indicate that the model significantly enhances emotional recognition accuracy.

Automated System for Continuous Green Shoot Thinning in Vineyards

Highlights An automated green shoot thinning system was developed by integrating machine vision and mechanical systems. Mask R-CNN used for segmenting cordons at different shoot growth stages. Point cloud registration used to accurately stitch cordon trajectories for continuous thinning operation. Field and lab tests demonstrated the integrated system can efficiently automate green shoot thinning operation. Abstract. Mechanized green shoot thinning in vineyards has helped reduce labor and associated costs. However, high variability in the thinning efficiency has been observed with the current machines in non-selective thinning due to challenges faced by the operators to precisely position thinning end-effectors along the cordon trajectories. In this study, an automated green shoot thinning system is proposed for cordon-trained grapevines to enhance the precision and efficiency of mechanical thinning operation. The automated system consisted of (1) a machine vision system for estimating cordon trajectories and (2) a mechanical system to position thinning rollers along the estimated trajectories during the real-time thinning operation. The vision pipeline used a Mask R-CNN model to determine cordon profiles at different shoot growth stages and a depth image to estimate the 3D cordon trajectory. In addition, point-cloud registration estimated transformation matrices for accurately stitching adjacent cordon trajectories and generating a long map of trajectory for the thinning rollers to follow. An image processing-based calibration method was used to obtain transformation parameters that transformed the estimated trajectories from the camera coordinate space to the thinning-roller coordinate space required for trajectory following. Lab tests, conducted using simulated cordons, showed the integrated system can precisely position the thinning rollers with root mean square errors (RMSE) of 2.0 cm and 1.7 cm for vertical and lateral positions, respectively. To quantify the thinning performance of the integrated system in the field, shoot thinning efficiency (the percentage of target shoots impacted) was determined. From the field evaluation, shoot thinning efficiency of 76% and 85% were achieved for two rows of grapevines, with an overall shoot thinning efficiency of 80%. The favorable test results demonstrated that the proposed integrated system is capable of effectively automating green shoot thinning operations, which has the potential to improve shoot thinning performance in vineyards. Keywords: Automated green shoot thinning, Cordon trajectory estimation, Mask R-CNN, Vineyard canopy management.

Importance of Machine Vision Framework with Nondestructive Approach for Fruit Classification and Grading: A Review

Machine vision technology has gained significant importance in the agricultural industry, particularly in the non-destructive classification and grading of fruits. This paper presents a comprehensive review of the existing literature, highlighting the crucial role of machine vision in automating the fruit quality assessment process. The study encompasses various aspects, including image acquisition techniques, feature extraction methods, and classification algorithms. The analysis reveals the substantial progress made in the field, such as developing sophisticated hardware and software solutions, which have improved accuracy and efficiency in fruit grading. Furthermore, it discusses the challenges and limitations, such as dealing with variability in fruit appearance, handling different fruit types, and real-time processing. The identification of future research needs emphasizes the potential for enhancing machine vision frameworks through the integration of advanced technologies like deep learning and artificial intelligence.Additionally, it underscores the importance of addressing the specific needs of different fruit varieties and exploring the applicability of machine vision in real-world scenarios, such as fruit packaging and logistics. This review underscores the critical role of machine vision in non-destructive fruit classification and grading, with numerous opportunities for further research and innovation. As the agricultural industry continues to evolve, integrating machine vision technologies will be instrumental in improving fruit quality assessment, reducing food waste, and enhancing the overall efficiency of fruit processing and distribution.

Transmission-Based Vertebrae Strength Probe Development: Far Field Probe Property Extraction and Integrated Machine Vision Distance Validation Experiments

We are developing a transmission-based probe for point-of-care assessment of vertebrae strength needed for fabricating the instrumentation used in supporting the spinal column during spinal fusion surgery. The device is based on a transmission probe whereby thin coaxial probes are inserted into the small canals through the pedicles and into the vertebrae, and a broad band signal is transmitted from one probe to the other across the bone tissue. Simultaneously, a machine vision scheme has been developed to measure the separation distance between the probe tips while they are inserted into the vertebrae. The latter technique includes a small camera mounted to the handle of one probe and associated fiducials printed on the other. Machine vision techniques make it possible to track the location of the fiducial-based probe tip and compare it to the fixed coordinate location of the camera-based probe tip. The combination of the two methods allows for straightforward calculation of tissue characteristics by exploiting the antenna far field approximation. Validation tests of the two concepts are presented as a precursor to clinical prototype development.

Autonomous Pick and Place for Mechanical Assembly WRS 2020 Assembly Challenge

The World Robot Summit (WRS) 2020 Assembly Challenge aims to realize the future of industrial robotics by building agile and lean production systems that can (1) respond to ever-changing manufacturing requirements for high-mix low-volume production, (2) be configured easily without paying a high cost for system integration, (3) be reconfigured in an agile and lean manner, and (4) be implemented as agile one-off manufacturing for the industry. One of the technical challenges of assembly is the detection, identification, and classification of a small object in an unstructured environment to grasp, manipulate, and insert it into the mating part to assemble the mechanical system. We presented an end-to-end system solution based on flexibility, modularity, and robustness with integrated machine vision and robust planning and execution. To achieve this, we utilized the behavior tree paradigm and a hierarchy of recovery strategies to ensure effective operation. We selected the behavior tree approach because of its design efficiency in creating complex systems that are modular, hierarchical, and reactive, and offer graphical representations with semantic meaning. These properties are essential for any part assembly task. Our system was selected for the WRS 2020 Assembly Challenge finals based on its successful performance in qualifying rounds.

Architecture for determining the cleanliness in shared vehicles using an integrated machine vision and indoor air quality-monitoring system

In an attempt to mitigate emissions and road traffic, a significant interest has been recently noted in expanding the use of shared vehicles to replace private modes of transport. However, one outstanding issue has been the hesitancy of passengers to use shared vehicles due to the substandard levels of interior cleanliness, as a result of leftover items from previous users. The current research focuses on developing a novel prediction model using computer vision capable of detecting various types of trash and valuables from a vehicle interior in a timely manner to enhance ambience and passenger comfort. The interior state is captured by a stationary wide-angled camera unit located above the seating area. The acquired images are preprocessed to remove unwanted areas and subjected to a convolutional neural network (CNN) capable of predicting the type and location of leftover items. The algorithm was validated using data collected from two research vehicles under varying conditions of light and shadow levels. The experiments yielded an accuracy of 89% over distinct classes of leftover items and an accuracy of 91% among the general classes of trash and valuables. The average execution time was 65 s from image acquisition in the vehicle to displaying the results in a remote server. A custom dataset of 1379 raw images was also made publicly available for future development work. Additionally, an indoor air quality (IAQ) unit capable of detecting specific air pollutants inside the vehicle was implemented. Based on the pilots conducted for air quality monitoring within the vehicle cabin, an IAQ index was derived which corresponded to a 6-level scale in which each level was associated with the explicit state of interior odour. Future work will focus on integrating the two systems (item detection and air quality monitoring) explicitly to produce a discrete level of cleanliness. The current dataset will also be expanded by collecting data from real shared vehicles in operation.

Design of Fatigue Driving Behavior Detection Based on Circle Hough Transform.

Chronic fatigue symptoms of jobs are risk factors that may cause errors and lead to occupational accidents. For instance, occupational injuries and traffic accidents stem from overlooking long-term fatigue. According to statistics for fatigue driving, it was found that fatigue driving is one of the main causes of traffic accidents. The resulting decrease in the quality of traffic, as well as impaired traffic flow efficiency and functioning, contributes markedly to the societal costs of fatigue. This article proposes a noninvasive physical method for fatigue detection using a machine vision image algorithm. The main technology was implemented using a software framework based on optimized skin color segmentation and edge detection, as well as eye contour extraction. By integrating machine vision and an optimized Hove transform algorithm, our method mainly identifies fatigue based on the detected target's face, head gestures, mouth aspect ratio (MAR), and eye condition, and then triggers an alarm through an intelligent auxiliary device. Our evaluation results of facial image data analysis showed that with an ideal eye threshold of 0.3, PERCLOS-80 standard, MAR, and head gesture-nod frequency, the method can be used to detect fatigue data accurately and systematically, thereby fulfilling the purpose of alerting a group of high-risk drivers and preventing them from engaging in high-risk activities in an involuntary state.

Comparative assessment of common pre-trained CNNs for vision-based surface defect detection of machined components

Comparative assessment of common pre-trained CNNs for vision-based surface defect detection of machined components

Cultural and Creative Product Design and Image Recognition Based on Deep Learning.

In today's technological world, advanced intelligence technologies such as deep learning (DL) techniques are widely applied in various fields. In this study, people are going to research cultural and creative product design and image recognition based on deep learning. Cultural creative products are referred to as products that are designed by taking inspiration from the cultural aspects. The use of cultural and creative products has increased among the people, thus creating a fair market. Artificial intelligence deep learning (DL) is employed for the design of culturally creative objects. Deep learning is referred to as a machine learning technique that is used to teach machines to imitate human behaviour so that computers can learn from examples. The proposed system utilizes image recognition technique which is referred as the ability of computer systems to identify objects from an image. The image recognition technique integrates machine vision technology, which uses cameras and artificial intelligent software for recognising images. This technology is widely used for various functions, such as self-driven cars, image content searches, and machine vision robots. In our proposed system, image recognition based on deep learning is used in the design of cultural and creative products through the utilisation of randomized algorithms. The system is found to deliver more accurate solutions when compared with the existing LDA, HMM, and optimization algorithms.

Integrated Machine Vision System for Evaluating Hole Expansion Ratio of Advanced High-Strength Steels.

In this paper, we propose a new method to estimate the hole expansion ratio (HER) using an integrated analysis system. To precisely measure the HER, three kinds of analysis methods (computer vision, punch load, and acoustic emission) were utilized to detect edge cracks during a hole expansion test. Cracks can be recognized by employing both computer vision and a punch load analysis system to determine the moment of crack initiation. However, the acoustic emission analysis system has difficulty detecting the instant of crack appearance since the magnitude of the audio signal is drowned out by noise from the press, which interrupts the differentiation of crack configuration. To enhance the accuracy for determining the HER, an integrated analysis system that combines computer vision with punch load analysis, and improves on the shortcomings of each analysis system, is newly suggested.

Sample-to-Answer Robotic ELISA.

Enzyme-linked immunosorbent assays (ELISA), as one of the most used immunoassays, have been conducted ubiquitously in hospitals, research laboratories, etc. However, the conventional ELISA procedure is usually laborious, occupies bulky instruments, consumes lengthy operation time, and relies considerably on the skills of technicians, and such limitations call for innovations to develop a fully automated ELISA platform. In this paper, we have presented a system incorporating a robotic-microfluidic interface (RoMI) and a modular hybrid microfluidic chip that embeds a highly sensitive nanofibrous membrane, referred to as the Robotic ELISA, to achieve human-free sample-to-answer ELISA tests in a fully programmable and automated manner. It carries out multiple bioanalytical procedures to replace the manual steps involved in classic ELISA operations, including the pneumatically driven high-precision pipetting, efficient mixing and enrichment enabled by back-and-forth flows, washing, and integrated machine vision for colorimetric readout. The Robotic ELISA platform has achieved a low limit of detection of 0.1 ng/mL in the detection of a low sample volume (15 μL) of chloramphenicol within 20 min without human intervention, which is significantly faster than that of the conventional ELISA procedure. Benefiting from its modular design and automated operations, the Robotic ELISA platform has great potential to be deployed for a broad range of detections in various resource-limited settings or high-risk environments, where human involvement needs to be minimized while the testing timeliness, consistency, and sensitivity are all desired.

Toward Future Automatic Warehouses: An Autonomous Depalletizing System Based on Mobile Manipulation and 3D Perception

This paper presents a mobile manipulation platform designed for autonomous depalletizing tasks. The proposed solution integrates machine vision, control and mechanical components to increase flexibility and ease of deployment in industrial environments such as warehouses. A collaborative robot mounted on a mobile base is proposed, equipped with a simple manipulation tool and a 3D in-hand vision system that detects parcel boxes on a pallet, and that pulls them one by one on the mobile base for transportation. The robot setup allows to avoid the cumbersome implementation of pick-and-place operations, since it does not require lifting the boxes. The 3D vision system is used to provide an initial estimation of the pose of the boxes on the top layer of the pallet, and to accurately detect the separation between the boxes for manipulation. Force measurement provided by the robot together with admittance control are exploited to verify the correct execution of the manipulation task. The proposed system was implemented and tested in a simplified laboratory scenario and the results of experimental trials are reported.

Bridging the Gap between Automated Manufacturing of Fuel Cell Components and Robotic Assembly of Fuel Cell Stacks

Recently demonstrated robotic assembling technologies for fuel cell stacks used fuel cell components manually pre-arranged in stacks (presenters). Identifying the original orientation of fuel cell components and loading them in presenters for a subsequent automated assembly process is a difficult, repetitive work cycle which if done manually, deceives the advantages offered by either the automated fabrication technologies for fuel cell components or by the robotic assembly processes. We present for the first time a robotic technology which enables the integration of automated fabrication processes for fuel cell components with a robotic assembly process of fuel cell stacks into a fully automated fuel cell manufacturing line. This task uses a Yaskawa Motoman SDA5F dual arm robot with integrated machine vision system. The process is used to identify and grasp randomly placed, slightly asymmetric fuel cell components, to reorient them all in the same position and stack them in presenters in preparation for a subsequent robotic assembly process. The process was demonstrated as part of a larger endeavor of bringing to readiness advanced manufacturing technologies for alternative energy systems, and responds the high priority needs identified by the U.S. Department of Energy for fuel cells manufacturing research and development.

Development and field evaluation of a machine vision based in-season weed detection system for wild blueberry

Abstract The wild blueberry crop requires the heavy application of agrochemicals for the proper crop yield and weed control. An integrated machine vision based weed detection system was developed to target goldenrod weed spot-specifically. Color co-occurrence matrices and statistical classifiers were used for the goldenrod detection. The linear and quadratic classifiers were developed using different reduced sub-sets of textural features. The classifiers were evaluated using accuracy, specificity, sensitivity, and false negative ratio at Laboratory scale. Performance of the developed weed detection system was also evaluated in two wild blueberry fields. The results indicated that quadratic classifier with DM-HSISD model showed the best performance at laboratory scale with the classification accuracies of 94.98% and 93.80% for training and testing datasets, respectively. Another quadratic classifier (with DM-HSI model) containing all 39 textural features was found to be the second best with the accuracies of 94.29% and 91.46% on training and test datasets, respectively. The linear classifiers didn’t perform well compare to their respective quadratic counter-parts. Field performance of the developed system equipped with the quadratic DM-HSISD classifier indicated no significant difference between variable rate (VR) and uniform application (UA) in terms of mean percentage area coverage (PAC) for the targeted goldenrod spots in both fields. However, a significant difference was observed between mean PAC of VR and UA applications for the non-targeted wild blueberry spots. The potential and actual chemical savings were in ranges between 46.71% and 74.83% and 30.12% and 60.58% depending on the weed and sprayed area, respectively. These results demonstrated that the developed weed detection system has potential for targeted application of agrochemicals to control goldenrod in wild blueberry fields.