Real-time Machine Vision System Research Articles

Correction: Real-time Machine Vision System for the Visually Impaired

Correction: Real-time Machine Vision System for the Visually Impaired

Real-time Machine Vision System for the Visually Impaired

Real-time Machine Vision System for the Visually Impaired

Unsupervised surface defect detection of aluminum sheets with combined bright-field and dark-field illumination

In the field of quality control for aluminum sheets, the presence of surface defects can greatly impact the overall product quality. Supervised machine learning methods, which necessitate a substantial quantity of expertly annotated defect samples, present limitations when faced with a limited number and lack of diversity of defect samples. To address these challenges, an unsupervised defect detection method, which utilizes a combination of bright-field and dark-field illumination, is proposed. Combined bright-field and dark-field illumination can provide more information about the surface and enhance the visibility of defects by illuminating the surface from different angles and at different levels of reflectivity. Based on the combined illumination condition, an unsupervised patch-aware feature matching network, which is inspired by the anomaly detection paradigm and only requires defect-free samples for training, is proposed in this paper. The network can extract high-resolution features from bright-field and their corresponding dark-field images simultaneously. Additionally, a well-designed scoring function that considers both intra-field and inter-field relationships is introduced to obtain more accurate anomaly scores for the score map. Moreover, artificially simulated abnormal samples are incorporated into the training phase, which assists the network in explicitly learning potential differences between normal and abnormal samples. The proposed method was thoroughly evaluated on a dataset of surface defects in aluminum sheets. The experimental results demonstrate the superior performance of the proposed method in defect identification and segmentation, achieving a high level of accuracy with small model size and short inference time, outperforming other neural network-based methods. The method has been implemented in a real-time machine vision system, resulting in a significant improvement in detection efficiency and product quality.

A machine vision system to predict individual cow feed intake of different feeds in a cowshed

Data on individual feed intake of dairy cows, an important variable for farm management, are currently unavailable in commercial dairies. A real-time machine vision system including models that are able to adapt to multiple types of feed was developed to predict individual feed intake of dairy cows. Using a Red-Green-Blue-Depth (RGBD) camera, images of feed piles of two different feed types (lactating cows' feed and heifers' feed) were acquired in a research dairy farm, for a range of feed weights under varied configurations and illuminations. Several models were developed to predict individual feed intake: two Transfer Learning (TL) models based on Convolutional Neural Networks (CNNs), one CNN model trained on both feed types, and one Multilayer Perceptron and Convolutional Neural Network model trained on both feed types, along with categorical data. We also implemented a statistical method to compare these four models using a Linear Mixed Model and a Generalised Linear Mixed Model, showing that all models are significantly different. The TL models performed best and were trained on both feeds with TL methods. These models achieved Mean Absolute Errors (MAEs) of 0.12 and 0.13 kg per meal with RMSE of 0.18 and 0.17 kg per meal for the two different feeds, when tested on varied data collected manually in a cowshed. Testing the model with actual cows’ meals data automatically collected by the system in the cowshed resulted in a MAE of 0.14 kg per meal and RMSE of 0.19 kg per meal. These results suggest the potential of measuring individual feed intake of dairy cows in a cowshed using RGBD cameras and Deep Learning models that can be applied and tuned to different types of feed.

Development of a real-time machine vision system for functional textile fabric defect detection using a deep YOLOv4 model

We introduce a real-time machine vision system we developed with the aim of detecting defects in functional textile fabrics with good precision at relatively fast detection speeds to assist in textile industry quality control. The system consists of image acquisition hardware and image processing software. The software we developed uses data preprocessing techniques to break down raw images to smaller suitable sizes. Filtering is employed to denoise and enhance some features. To generalize and multiply the data to create robustness, we use data augmentation, which is followed by labeling where the defects in the images are labeled and tagged. Lastly, we utilize YOLOv4 for localization where the system is trained with weights of a pretrained model. Our software is deployed with the hardware that we designed to implement the detection system. The designed system shows strong performance in defect detection with precision of [Formula: see text], and recall and [Formula: see text] scores of [Formula: see text] and [Formula: see text], respectively. The detection speed is relatively fast at [Formula: see text] fps with a prediction speed of [Formula: see text] ms. Our system can automatically locate functional textile fabric defects with high confidence in real time.

Defective egg detection based on deep features and Bidirectional Long-Short-Term-Memory

Eggs are one of the most important nutritional sources worldwide. In addition, defects that may occur in the eggshells endanger food safety and cause adverse effects on production companies. Automatic separation of defective eggs from quality eggs is an important issue due to economic and health reasons. On this motivation, a real-time machine vision system based on deep learning has been developed for the detection of cracked, bloody and dirty eggs. In this study, a continuous rotating system was designed to visualize all surfaces of the egg. Thus, the adverse conditions such as dirt, blood, and cracks that may occur on any surface of the egg have been successfully monitored. In the proposed system for the detection of robust eggs, deep features are extracted using a pre-trained residual network model and then the obtained features are fed into the Bidirectional Long-Short-Term-Memory (BiLSTM). The efficiency of the proposed model was calculated using dirty, bloody, cracked and robust egg images with the developed machine vision system. The experimental works showed that the proposed model achieved a 99.17% accuracy score. The obtained result was also compared with state-of-the-art methods, and the proposed model was observed to exhibit the highest accuracy among the compared methods.

Real-Time Detection of Strawberry Powdery Mildew Disease Using a Mobile Machine Vision System

Strawberry cropping system relies heavily on proper disease management to maintain high crop yield. Powdery mildew, caused by Sphaerotheca macularis (Wall. Ex Fries) is one of the major leaf diseases in strawberry which can cause significant yield losses up to 70%. Field scouts manually walk beside strawberry fields and visually observe the plants to monitor for powdery mildew disease infection each week during summer months which is a laborious and time-consuming endeavor. The objective of this research was to increase the efficiency of field scouting by automatically detecting powdery mildew disease in strawberry fields by using a real-time machine vision system. A global positioning system, two cameras, a custom image processing program, and a ruggedized laptop computer were utilized for development of the disease detection system. The custom image processing program was developed using color co-occurrence matrix-based texture analysis along with artificial neural network technique to process and classify continuously acquired image data simultaneously. Three commercial strawberry field sites in central Nova Scotia were used to evaluate the performance of the developed system. A total of 36 strawberry rows (~1.06 ha) were tested within three fields and powdery mildew detected points were measured manually followed by automatic detection system. The manually detected points were compared with automatically detected points to ensure the accuracy of the developed system. Results of regression and scatter plots revealed that the system was able to detect disease having mean absolute error values of 4.00, 3.42, and 2.83 per row and root mean square error values of 4.12, 3.71, and 3.00 per row in field site-I, field site-II, and field site-III, respectively. The slight deviation in performance was likely caused by high wind speeds (>8 km h−1), leaf overlapping, leaf angle, and presence of spider mite disease during field testing.

Utilising low cost RGB-D cameras to track the real time progress of a manual assembly sequence

Purpose The purpose of this paper is to explore the role that computer vision can play within new industrial paradigms such as Industry 4.0 and in particular to support production line improvements to achieve flexible manufacturing. As Industry 4.0 requires “big data”, it is accepted that computer vision could be one of the tools for its capture and efficient analysis. RGB-D data gathered from real-time machine vision systems such as Kinect ® can be processed using computer vision techniques. Design/methodology/approach This research exploits RGB-D cameras such as Kinect® to investigate the feasibility of using computer vision techniques to track the progress of a manual assembly task on a production line. Several techniques to track the progress of a manual assembly task are presented. The use of CAD model files to track the manufacturing tasks is also outlined. Findings This research has found that RGB-D cameras can be suitable for object recognition within an industrial environment if a number of constraints are considered or different devices/techniques combined. Furthermore, through the use of a HMM inspired state-based workflow, the algorithm presented in this paper is computationally tractable. Originality/value Processing of data from robust and cheap real-time machine vision systems could bring increased understanding of production line features. In addition, new techniques that enable the progress tracking of manual assembly sequences may be defined through the further analysis of such visual data. The approaches explored within this paper make a contribution to the utilisation of visual information “big data” sets for more efficient and automated production.

ASP-SIFT: Using Analog Signal Processing Architecture to Accelerate Keypoint Detection of SIFT Algorithm

The scale-invariant feature transform (SIFT) algorithm is still one of the most reliable image feature extraction methods. Despite its excellent robustness on various image transformations, SIFT’s intensive computational burden has been severely preventing it from being used in real-time and energy-efficient embedded machine vision systems. To reduce processing time and energy cost while executing SIFT, an analog signal processing architecture, analog signal processing (ASP)-SIFT, is proposed in this article. In ASP-SIFT, the Gaussian pyramid construction, difference-of-Gaussian (DoG) pyramid construction and keypoint locating, which are the primary steps of the keypoint detection part of the SIFT algorithm, are done directly with analog circuit networks. Thus, by completing keypoint detection in the analog domain, the total processing time is approximately equal to the settling time of the circuit network. Besides, by adopting a current-mode circuit network operating in the subthreshold region, the power dissipation would be very low. Simulation results show that the total processing speed for a typical video graphics array (VGA)-format ( $640\times480$ ) image is up to 2.3 kframes per second, which is at least $3.26\times $ faster than the state-of-the-art digital hardware accelerators, while the system power is 94.5 mW and the energy consumption is only $40~\mu \text{J}$ per frame.

Fast and Resource-Efficient Hardware Implementation of Modified Line Segment Detector

Lines are significant features enclosing high-level information in an image. The line segment Detector (LSD) Algorithm with low error rate is a widely used method to extract lines in images effectively and accurately. However, the algorithm on PC performs too costly both in time and resources for the real-time video processing. This paper provides a fast and resource-efficient hardware implementation solution for a modified LSD algorithm on Field Programmable Gate Arrays (FPGA) for real-time line detection. The task-level pipeline structures are exploited fully in a stream process mapped to the hardware architecture free of frame buffer. Our proposed hardware implementation processes in a stream-in–stream-out manner with little consumption of the on-chip block RAM to store intermediate values. We first employ hardware Gaussian filter and adjust Canny edge detection to obtain an edge map at single-pixel width. Then, a novel structure of region growing model based on dynamic rooted tree is used to detect line segment regions accurately with a latency of only a few rows of pixels. The low cost in time, on-chip resources, and power consumption makes our proposed algorithm suitable for portable real-time streaming video processing applications using line segment features, such as Lane departure warning systems. It can also be applied in real-time machine vision systems that use line segments information for further recognition or stereo correspondence and many others. The proposed algorithm is synthesized and tested on XC7Z020 FPGA with high reliability, accuracy speed, and low cost in both resources and energy.

New efficient lower bound for the Hybrid Flow Shop Scheduling Problem with Multiprocessor Tasks

In this paper, we address the hybrid flow shop scheduling problem with multiprocessor tasks. The objective is to minimize the maximum completion time. This problem is encountered in manufacturing, parallel and distributed computing, and real-time machine vision systems. This problem is strongly NP-hard, and consequently, several heuristics and meta heuristics were proposed in the literature in order to provide a near optimal solution. Assessing the performance of these heuristics requires efficient lower bounds. Surprisingly, few lower bounds with moderate performance were proposed. Because of this reason, we propose in this paper a new efficient destructive lower bound. This lower bound is based on the concept of revisited energetic reasoning, which is basically a feasible test with window time adjustments. The efficiency of the proposed lower bound is assessed throughout an extensive computational experiments conducted on a benchmark of 2,100 instances with up to ten centers. The numerical results provide evidence that the proposed lower bound consistently improves the best existing ones.

Real-Time Automatic Inspection System for the Classification of PCB Flux Defects

Automatic inspection based on a real-time machine vision system may serve as substitute for the manual human visual inspection of flux defects in Printed Circuit Boards (PCBs), which often cause damage on the board in the form of corrosions that harm the assembly. The concept of automatic inspection contributes to the improvement of the manufacturing quality of PCBs and facilitates their approval or rejection. The Automatic Inspection System for Printed Circuit Boards (AIS-PCB) is developed with the capability to identify the defects and the quality of PCBs. It is based on a real time system machine vision. The developed AIS-PCB is capable of detecting, indexing and classifying by measuring the flux defects in PCBs during the re-flow of the real-time process. The AIS-PCB is The total automation control system is the core of the AIS-PCB. This system consist of vision inspection station, mechanical loader and unloader, final decision station and the pneumatic system handler. To detect and classify the quality of PCBs, segmentation in conjunction with Radon transform approaches are used for feature indexing and line detection based on the gradient field of PCB images. The Feed-Forward Back-Propagation (FFBP) model is used to classify the product quality of the PCBs via a learning concept. A number of trainings using the FFBP are performed to learn and match the targets. The images of each PCB classes are used as inputs to the classification module. The obtained results from the classification and rule decision are used to establish the receiver operating characteristic curve. The classifier, which is based on the proposed approach and is tested on the PCBs from a factory’s production line, achieves a sorting Coefficient Of Efficiency (COE >95%). The developed AIS-PCB system shows promising results in successfully segmenting and classifying flux defects in PCBs through computerized visual information and facilitates their automatic inspection, thereby aiding humans in conducting rapid inspections.

Real-Time Machine Vision System for an Automated Quality Monitoring in Mass Production of Multiaxial Non-Crimp Fabrics

Fiber-reinforced plastics (FRPs) are used in an increasing number of applications due to their advanced light-weight properties. Beside classical deployments in high-value industries like aerospace and medical engineering, FRP materials are pushed towards mass production by the automotive industry. To mass-produce FRP products, textile structures are commonly used as semifinished products, such as multiaxial non-crimp fabrics (NCFs). However, poor repeatability and missing textile defect detection in the automated manufacturing of FRP components are major cost factors and challenge economically the series production. Reduction of these cost factors is not yet possible due to the lack of closed-loop control systems. There is currently no real-time quality monitoring system capable of ensuring quality in NCF production. The purpose of this study is to develop tools and concepts for real-time quality control of non-crimp fabrics. Therefore, a real-time machine-vision system has been developed with the purpose of detecting relevant quality features in a textile sample in deterministic time conditions. The embedded system ensures the execution of all process steps, i.e. image acquisition, processing, and evaluation, under real-time conditions. The main focus of this work is laid on the real-time algorithms for an accurate and robust detection of the fiber orientation under industrial conditions. The developed real-time system has been tested on a textile sample and an assessment of the measurement uncertainty has been performed. Results show that the proposed system can successfully assess common textile quality features.

A Real-Time Apple Grading System Using Multicolor Space

This study was focused on the multicolor space which provides a better specification of the color and size of the apple in an image. In the study, a real-time machine vision system classifying apples into four categories with respect to color and size was designed. In the analysis, different color spaces were used. As a result, 97% identification success for the red fields of the apple was obtained depending on the values of the parameter “a” of CIE L*a*b*color space. Similarly, 94% identification success for the yellow fields was obtained depending on the values of the parameter y of CIE XYZ color space. With the designed system, three kinds of apples (Golden, Starking, and Jonagold) were investigated by classifying them into four groups with respect to two parameters, color and size. Finally, 99% success rate was achieved in the analyses conducted for 595 apples.

Real-time hyperspectral processing for automatic nonferrous material sorting

The application of hyperspectral sensors in the develop- ment of machine vision solutions has become increasingly popular as the spectral characteristics of the imaged materials are better mod- eled in the hyperspectral domain than in the standard trichromatic red, green, blue data. While there is no doubt that the availability of detailed spectral information is opportune as it opens the possibility to construct robust image descriptors, it also raises a substantial chal- lenge when this high-dimensional data is used in the development of real-time machine vision systems. To alleviate the computational demand, often decorrelation techniques are commonly applied prior to feature extraction. While this approach has reduced to some extent the size of the spectral descriptor, data decorrelation alone proved insufficient in attaining real-time classification. This fact is particularly apparent when pixel-wise image descriptors are not sufficiently robust to model the spectral characteristics of the imaged materials, a case when the spatial information (or textural properties) also has to be included in the classification process. The integration of spectral and spatial information entails a substantial computational cost, and as a result the prospects of real-time opera- tion for the developed machine vision system are compromised. To answer this requirement, in this paper we have reengineered the approach behind the integration of the spectral and spatial information in the material classification process to allow the real-time sorting of the nonferrous fractions that are contained in the waste of electric and electronic equipment scrap. © 2012 SPIE and IS&T. (DOI: 10.1117/ 1.JEI.21.1.013018)

Development of a real-time laser-based machine vision system to monitor and control welding processes

In this study, a laser-based machine vision system is developed and implemented to monitor and control welding processes. The system consists of three main modules: a laser-based vision sensor module, an image processing module, and a multi-axis motion control module. The laser-based vision sensor is designed and fabricated based on the principle of laser triangulation. By developing and implementing a new image processing algorithm on the platform of LabVIEW, the image processing module is capable of processing the images captured by the vision sensor, identifying the different types of weld joints, and detecting the feature points. Based on the detected feature points, the position information and geometrical features of the weld joint such as its depth, width, plates mismatch, and cross-sectional area can be obtained and monitored in real time. Meanwhile, by feeding these data into the multi-axis motion control module, a non-contact seam tracking is achieved by adaptively adjusting the position of the welding torch with respect to the depth and width variations of the weld joint. A 3D profile of the weld joint is also obtained in real time for the purposes of in-process weld joint monitoring and post-weld quality inspection. The results indicate that the developed laser-based machine vision system can be well suited for the measurement of weld joint geometrical features, seam tracking, and 3D profiling.

Cost-Effective Video Filtering Solution for Real-Time Vision Systems

This paper presents an efficient video filtering scheme and its implementation in a field-programmable logic device (FPLD). Since the proposed nonlinear, spatiotemporal filtering scheme is based on order statistics, its efficient implementation benefits from a bit-serial realization. The utilization of both the spatial and temporal correlation characteristics of the processed video significantly increases the computational demands on this solution, and thus, implementation becomes a significant challenge. Simulation studies reported in this paper indicate that the proposed pipelined bit-serial FPLD filtering solution can achieve speeds of up to 97.6 Mpixels/s and consumes 1700 to 2700 logic cells for the speed-optimized and area-optimized versions, respectively. Thus, the filter area represents only 6.6 to 10.5% of the Altera STRATIX EP1S25 device available on the Altera Stratix DSP evaluation board, which has been used to implement a prototype of the entire real-time vision system. As such, the proposed adaptive video filtering scheme is both practical and attractive for real-time machine vision and surveillance systems as well as conventional video and multimedia applications.

A Genetic Algorithm for Hybrid Flow-shop Scheduling with Multiprocessor Tasks

The hybrid flow-shop scheduling problem with multiprocessor tasks finds its applications in real-time machine-vision systems among others. Motivated by this application and the computational complexity of the problem, we propose a genetic algorithm in this paper. We first describe the implementation details, which include a new crossover operator. We then perform a preliminary test to set the best values of the control parameters, namely the population size, crossover rate and mutation rate. Next, given these values, we carry out an extensive computational experiment to evaluate the performance of four versions of the proposed genetic algorithm in terms of the percentage deviation of the solution from the lower bound value. The results of the experiments demonstrate that the genetic algorithm performs the best when the new crossover operator is used along with the insertion mutation. This genetic algorithm also outperforms the tabu search algorithm proposed in the literature for the same problem.

PRECISION WEED CONTROL SYSTEM FOR COTTON

A real–time robotic weed control system was developed and tested in commercial cotton fields. The precision weed control system was capable of distinguishing grass–like weeds from cotton plants and applying a chemical spray only to targeted weeds while traveling at a continuous speed of 0.45 m/s. The robot consisted of a real–time machine vision system, a controlled illumination chamber, and a precision chemical applicator. In commercial cotton fields, the system correctly sprayed 88.8% of the weeds while correctly identifying and not spraying 78.7% of the cotton plants while traveling at 0.45 m/s.

Monitoring mineral wool production using real-time machine vision

This paper describes the development and use of a real-time machine vision system in the industrial production of mineral wool. Because of specific operating requirements and the very hostile environment the system employs unique hardware and software features. The main requirement is for real-time detection and control of the trajectory of molten rock as it falls onto a system of spinning wheels. The jet exhibits significant variation of its position with time, as a consequence of complex thermodynamic and hydrodynamic processes, resulting in a wide variation of quality and quantity of final product. Algorithms were developed to detect variations in the jet trajectory with time and this information is used to generate suitable control data.