Machine Vision Applications Research Articles

Hyper Progressive Single Shot Detector; HPSSD

Studies into intelligent production and the implementation of smart factories have been promising in recent years, first generation artificial intelligence is out and all major companies are trying to integrate some form of machine learning into their systems. In the field of Machine vision, promising architectures have been built to strengthen the literature, this paper proses the development of a Neural Network called HPSSD (Hyper Progressive Single Shot Detector) to serve as the computational backbone for Machine vision applications specifically for Inspection. HPSSD is a Single Shot Detector that was built on the basis of a RCBAM, an MDCM, and a PAM module. The improvements shown after the development of HPSSD shows that there is no extent that machine Learning cannot take us through the Lens of Machine Vision. The tests demonstrates encouraging outcomes and enhancing the inspection accuracy in intelligent workshops.

FuF-Det: An Early Forest Fire Detection Method under Fog

In recent years, frequent forest fires have seriously threatened the earth’s ecosystem and people’s lives and safety. With the development of machine vision and unmanned aerial vehicle (UAVs) technology, UAV monitoring combined with machine vision has become an important development trend in forest fire monitoring. In the early stages, fire shows the characteristics of a small fire target and obvious smoke. However, the presence of fog interference in the forest will reduce the accuracy of fire point location and smoke identification. Therefore, an anchor-free target detection algorithm called FuF-Det based on an encoder–decoder structure is proposed to accurately detect early fire points obscured by fog. The residual efficient channel attention block (RECAB) is designed as a decoder unit to improve the problem of the loss of fire point characteristics under fog caused by upsampling. Moreover, the attention-based adaptive fusion residual module (AAFRM) is used to self-enhance the encoder features, so that the features retain more fire point location information. Finally, coordinate attention (CA) is introduced to the detection head to make the image features correspond to the position information, and improve the accuracy of the algorithm to locate the fire point. The experimental results show that compared with eight mainstream target detection algorithms, FuF-Det has higher average precision and recall as an early forest fire detection method in fog and provides a new solution for the application of machine vision to early forest fire detection.

High-performance Reconfigurable DNN Accelerator on a Bandwidth-limited Embedded System

Deep convolutional neural networks (DNNs) have been widely used in many applications, particularly in machine vision. It is challenging to accelerate DNNs on embedded systems because real-world machine vision applications should reserve a lot of external memory bandwidth for other tasks, such as video capture and display, while leaving little bandwidth for accelerating DNNs. In order to solve this issue, in this study, we propose a high-throughput accelerator, called reconfigurable tiny neural network accelerator (ReTiNNA), for the bandwidth-limited system and present a real-time object detection system for the high-resolution video image. We first present a dedicated computation engine that takes different data mapping methods for various filter types to improve data reuse and reduce hardware resources. We then propose an adaptive layer-wise tiling strategy that tiles the feature maps into strips to reduce the control complexity of data transmission dramatically and to improve the efficiency of data transmission. Finally, a design space exploration (DSE) approach is presented to explore design space more accurately in the case of insufficient bandwidth to improve the performance of the low-bandwidth accelerator. With a low bandwidth of 2.23 GB/s and a low hardware consumption of 90.261K LUTs and 448 DSPs, ReTiNNA can still achieve a high performance of 155.86 GOPS on VGG16 and 68.20 GOPS on ResNet50, which is better than other state-of-the-art designs implemented on FPGA devices. Furthermore, the real-time object detection system can achieve a high object detection speed of 19 fps for high-resolution video.

Terminals and Connector Inspection Innovation using Machine Vision

Customer demands for product quality that are increasingly complex and require more accurate inspection are not sufficient if done manually due to high costs and different operator precision. To solve this problem, automatic vision inspection was created to check the product quality of terminal-type electronic components. Machine Vision is robust equipment for examining electrical components, agroindustry, surface mounting technology industry, plastic industry, and glass industry. In this research, the design and application of machine vision are used to inspect terminals and connectors, supporting automotive and electrical equipment components. Machine vision checks three quality functions, namely structural quality, surface quality, and dimensional quality. Optimization of image capture using backlighting and darkfield lighting. Implementing machine vision is carried out by verifying image retrieval's stability using a gage study with the results that there is no measurement bias.

EPCS: Endpoint-based part-aware curve skeleton extraction for low-quality point clouds

The curve skeleton is an important shape descriptor which has been utilized in various applications in computer graphics, machine vision, and artificial intelligence. In this study, the endpoint-based part-aware curve skeleton (EPCS) extraction method for low-quality point clouds is proposed. The novel random center shift (RCS) method is first proposed for detecting the endpoints on point clouds. The endpoints are used as the initial seed points for dividing each part into layers, and then the skeletal points are obtained by computing the center points of the oriented bounding box (OBB) of the layers. Subsequently, the skeletal points are connected, thus forming the branches. Furthermore, the multi-vector momentum-driven (MVMD) method is also proposed for locating the junction points which connect the branches. Due to the shape differences between different parts on point clouds, the global topology of the skeleton is finally optimized by removing the redundant junction points, re-connecting some branches using the proposed MVMD method, and applying an interpolation method based on the splitting operator. Consequently, a complete and smooth curve skeleton is achieved. The proposed EPCS method is compared with several state-of-the-art methods, and the experimental results verify its robustness and effectiveness. Furthermore, the skeleton extraction and model segmentation results on challenging point clouds of broken Terracotta also highlight the utility of the proposed method.

Single-shot wide-field full-stokes polarization imaging

Polarization is an intrinsic property of light. With the advancements in nanophotonics, it is now possible to achieve arbitrary polarization transformations in space. Incorporating special functions in different spatial dimensions expands the design freedom of metasurfaces and other optical devices. To understand and utilize the metasurface, we design a metasurface optical element to achieve single-shot, wide-field polarization imaging. Our demonstration exhibits a compact, full Stokes wide-field polarization imaging device without requiring conventional optical elements, such as moving parts or wide-field lens sets. Wide-field polarization imaging devices are potential candidates for machine vision and biological tissue imaging applications.

Monocular Vision-aided Depth Measurement from RGB Images for Autonomous UAV Navigation

Monocular vision-based 3D scene understanding has been an integral part of many machine vision applications. Always, the objective is to measure the depth using a single RGB camera, which is at par with the depth cameras. In this regard, monocular vision-guided autonomous navigation of robots is rapidly gaining popularity among the research community. We propose an effective monocular vision-assisted method to measure the depth of an Unmanned Aerial Vehicle (UAV) from an impending frontal obstacle. This is followed by collision-free navigation in unknown GPS-denied environments. Our approach deals upon the fundamental principle of perspective vision that the size of an object relative to its field of view (FoV) increases as the center of projection moves closer towards the object. Our contribution involves modeling the depth followed by its realization through scale-invariant SURF features. Noisy depth measurements arising due to external wind, or the turbulence in the UAV, are rectified by employing a constant velocity-based Kalman filter model. Necessary control commands are then designed based on the rectified depth value to avoid the obstacle before collision. Rigorous experiments with SURF scale-invariant features reveal an overall accuracy of 88.6% with varying obstacles, in both indoor and outdoor environments.

Novel Bayesian Inference-Based Approach for the Uncertainty Characterization of Zhang's Camera Calibration Method.

Camera calibration is necessary for many machine vision applications. The calibration methods are based on linear or non-linear optimization techniques that aim to find the best estimate of the camera parameters. One of the most commonly used methods in computer vision for the calibration of intrinsic camera parameters and lens distortion (interior orientation) is Zhang's method. Additionally, the uncertainty of the camera parameters is normally estimated by assuming that their variability can be explained by the images of the different poses of a checkerboard. However, the degree of reliability for both the best parameter values and their associated uncertainties has not yet been verified. Inaccurate estimates of intrinsic and extrinsic parameters during camera calibration may introduce additional biases in post-processing. This is why we propose a novel Bayesian inference-based approach that has allowed us to evaluate the degree of certainty of Zhang's camera calibration procedure. For this purpose, the a prioriprobability was assumed to be the one estimated by Zhang, and the intrinsic parameters were recalibrated by Bayesian inversion. The uncertainty of the intrinsic parameters was found to differ from the ones estimated with Zhang's method. However, the major source of inaccuracy is caused by the procedure for calculating the extrinsic parameters. The procedure used in the novel Bayesian inference-based approach significantly improves the reliability of the predictions of the image points, as it optimizes the extrinsic parameters.

An Overview of the Application of Machine Vision in Recognition and Localization of Fruit and Vegetable Harvesting Robots

Intelligent agriculture imposes higher requirements on the recognition and localization of fruit and vegetable picking robots. Due to its unique visual information and relatively low hardware cost, machine vision is widely applied in the recognition and localization of fruit and vegetable picking robots. This article provides an overview of the application of machine vision in the recognition and localization of fruit and vegetable picking robots. Firstly, the advantages, disadvantages, and the roles of different visual sensors and machine vision algorithms in the recognition and localization of fruit and vegetable picking robots are introduced, including monocular cameras, stereo cameras, structured light cameras, multispectral cameras, image segmentation algorithms, object detection algorithms, and 3D reconstruction algorithms. Then, the current status and challenges faced by machine vision in the recognition and localization of fruit and vegetable picking robots are summarized. These challenges include the stability of fast recognition under complex background interference, stability of recognition under different lighting environments for the same crop, the reliance of recognition and localization on prior information in the presence of fruit overlap and occlusions caused by leaves and branches, and the uncertainty of picking caused by complex working environments. In current research on algorithms dealing with complex background interference and various occlusion disturbances, good results have been achieved. Different lighting environments have a significant impact on the recognition and positioning of fruits and vegetables, with a minimum accuracy of 59.2%. Finally, this article outlines future research directions to address these challenges.

Application of machine vision and convolutional neural networks in discriminating tobacco leaf maturity on mobile devices

As harvesting at the right time is crucial to ensuring the best quality and maximizing the yield of tobacco leaves, great attention has been paid to the research of “harvest maturity”. Fresh tobacco leaves can be manually categorized into four maturity stages, including immature, pseudomature, mature, and hypermature, according to the leaf tissue structure and color. To improve the discriminative accuracy, convenience, and automation of maturity levels, a tobacco leaf maturity discriminative method based on machine vision and deep learning was developed and integrated into a system based on the mobile terminal. In this method, the chromatic features and texture features can be automatically extracted from the acquired digital image of tobacco leaf by convolutional neural networks (CNNs). Experimental results of an independent test set consisting of 480 tobacco leaf samples demonstrated that the proposed system was able to classify the maturity stage of tobacco leaf efficiently and accurately. The system's ability to run on mobile devices, like smartphones, makes it easier to accurately collect mature tobacco leaves and reduces subjectivity compared to the current visual identification approach. Moreover, this system can provide a picking date recommendation to tobacco growers.

IRIS: Integrated Retinal Functionality in Image Sensors.

Neuromorphic image sensors draw inspiration from the biological retina to implement visual computations in electronic hardware. Gain control in phototransduction and temporal differentiation at the first retinal synapse inspired the first generation of neuromorphic sensors, but processing in downstream retinal circuits, much of which has been discovered in the past decade, has not been implemented in image sensor technology. We present a technology-circuit co-design solution that implements two motion computations-object motion sensitivity and looming detection-at the retina's output that could have wide applications for vision-based decision-making in dynamic environments. Our simulations on Globalfoundries 22 nm technology node show that the proposed retina-inspired circuits can be fabricated on image sensing platforms in existing semiconductor foundries by taking advantage of the recent advances in semiconductor chip stacking technology. Integrated Retinal Functionality in Image Sensors (IRIS) technology could drive advances in machine vision applications that demand energy-efficient and low-bandwidth real-time decision-making.

Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging.

We report a microlens array camera with variable apertures (MACVA) for high dynamic range (HDR) imaging by using microlens arrays with various sizes of apertures. The MACVA comprises variable apertures, microlens arrays, gap spacers, and a CMOS image sensor. The microlenses with variable apertures capture low dynamic range (LDR) images with different f-stops under single-shot exposure. The reconstructed HDR images clearly exhibit expanded dynamic ranges surpassing LDR images as well as high resolution without motion artifacts, comparable to the maximum MTF50 value observed among the LDR images. This compact camera provides, what we believe to be, a new perspective for various machine vision or mobile devices applications.

Power Supply Technologies for Drones and Machine Vision Applications: A Comparative Analysis and Future Trends

The field of Unmanned Aerial Vehicles (UAVs), or drones, is encountering quick development in the areas of air transportation and computerization. Progress in innovation has prompted more noteworthy capacities and highlights in UAVs, which are currently broadly involved by the military and flying industry for an assortment of high-end generally safe errands. Highly advanced UAVs that can be controlled remotely via a controller, mobile phone, or ground station cockpit have been developed through the integration of automation technology and machine vision, which includes thermal imaging, cameras, sensors, and other sensors. The three primary characteristics of UAVs will be investigated in this study, namely power-source technology, deep-learning neural networks for computer vision, and some of the applications that are used the most. The goal is to thoroughly examine these characteristics and offer suggestions for addressing some of the difficulties of optimizing UAV performance and also exploring potential future trends.

Machine Learning Based Performance Analysis of Video Object Detection and Classification Using Modified Yolov3 and Mobilenet Algorithm

Detecting foreground objects in video is crucial in various machine vision applications and computerized video surveillance technologies. Object tracking and detection are essential in object identification, surveillance, and navigation approaches. Object detection is the technique of differentiating between background and foreground features in a photograph. Recent improvements in vision systems, including distributed smart cameras, have inspired researchers to develop enhanced machine vision applications for embedded systems. The efficiency of featured object detection algorithms declines as dynamic video data increases as contrasted to conventional object detection methods. Moving subjects that are blurred, fast-moving objects, backdrop occlusion, or dynamic background shifts within the foreground area of a video frame can all cause problems. These challenges result in insufficient prominence detection. This work develops a deep-learning model to overcome this issue. For object detection, a novel method utilizing YOLOv3 and MobileNet was built. First, rather than picking predefined feature maps in the conventional YOLOv3 architecture, the technique for determining feature maps in the MobileNet is optimized based on examining the receptive fields. This work focuses on three primary processes: object detection, recognition, and classification, to classify moving objects before shared features. Compared to existing algorithms, experimental findings on public datasets and our dataset reveal that the suggested approach achieves 99% correct classification accuracy for urban settings with moving objects. Experiments reveal that the suggested model beats existing cutting-edge models by speed and computation.

Environmentally Friendly Approach to the Reduction of Microplastics during Domestic Washing: Prospects for Machine Vision in Microplastics Reduction.

The increase in the global population is directly responsible for the acceleration in the production as well as the consumption of textile products. The use of textiles and garment materials is one of the primary reasons for the microfibers generation and it is anticipated to grow increasingly. Textile microfibers have been found in marine sediments and organisms, posing a real threat to the environment as it is invisible pollution caused by the textile industry. To protect against the damaging effects that microplastics can have, the formulation of mitigation strategies is urgently required. Therefore, the primary focus of this review manuscript is on finding an environmentally friendly long-term solution to the problem of microfiber emissions caused by the domestic washing process, as well as gaining an understanding of the various properties of textiles and how they influence this problem. In addition, it discussed the effect that mechanical and chemical finishes have on microfiber emissions and identified research gaps in order to direct future research objectives in the area of chemical finishing processes. In addition to that, it included a variety of preventative and minimizing strategies for reduction. Last but not least, an emphasis was placed on the potential and foreseeable applications of machine vision (i.e., quantification, data storage, and data sharing) to reduce the amount of microfibers emitted by residential washing machines.

Compression of Multiscale Features of FPN with Channel-Wise Reduction for VCM

With the development of deep learning technology and the abundance of sensors, machine vision applications that utilize vast amounts of image/video data are rapidly increasing in the autonomous vehicle, video surveillance and smart city fields. However, achieving a more compact image/video representation and lower latency solutions is challenging for such machine-based applications. Therefore, it is essential to develop a more efficient video coding standard for machine vision applications. Currently, the Moving Picture Experts Group (MPEG) is developing a new standard called video coding for machines (VCM) with two tracks, each mainly dealing with compression of the input image/video (Track 2) and compression of the features extracted from it (Track 1). In this paper, an enhanced multiscale feature compression (E-MSFC) method is proposed to efficiently compress multiscale features generated by a feature pyramid network (FPN), which is the backbone network of machine vision networks specified in the VCM evaluation framework. The proposed E-MSFC reduces the feature channels to be included in a single feature map and compresses the feature map using versatile video coding (VVC), the latest video standard, rather than the single stream feature compression (SSFC) module in the existing MSFC. In addition, the performance of the E-MSFC is further enhanced by adding a bottom-up structure to the multiscale feature fusion (MSFF) module, which performs the channel-wise reduction in the E-MSFC. Experimental results reveal that the proposed E-MSFC significantly outperforms the VCM image anchor with a BD-rate gain of up to 85.94%, which includes an additional gain of 0.96% achieved by the MSFF with the bottom-up structure.

An Efficient Vehicle Detection and Shadow Removal Using Gaussian Mixture Models with Blob Analysis for Machine Vision Application

An Efficient Vehicle Detection and Shadow Removal Using Gaussian Mixture Models with Blob Analysis for Machine Vision Application

Real-Time Defect Detection for Metal Components: A Fusion of Enhanced Canny–Devernay and YOLOv6 Algorithms

Due to the presence of numerous surface defects, the inadequate contrast between defective and non-defective regions, and the resemblance between noise and subtle defects, edge detection poses a significant challenge in dimensional error detection, leading to increased dimensional measurement inaccuracies. These issues serve as major bottlenecks in the domain of automatic detection of high-precision metal parts. To address these challenges, this research proposes a combined approach involving the utilization of the YOLOv6 deep learning network in conjunction with metal lock body parts for the rapid and accurate detection of surface flaws in metal workpieces. Additionally, an enhanced Canny–Devernay sub-pixel edge detection algorithm is employed to determine the size of the lock core bead hole. The methodology is as follows: The data set for surface defect detection is acquired using the labeling software lableImg and subsequently utilized for training the YOLOv6 model to obtain the model weights. For size measurement, the region of interest (ROI) corresponding to the lock cylinder bead hole is first extracted. Subsequently, Gaussian filtering is applied to the ROI, followed by a sub-pixel edge detection using the improved Canny–Devernay algorithm. Finally, the edges are fitted using the least squares method to determine the radius of the fitted circle. The measured value is obtained through size conversion. Experimental detection involves employing the YOLOv6 method to identify surface defects in the lock body workpiece, resulting in an achieved mean Average Precision (mAP) value of 0.911. Furthermore, the size of the lock core bead hole is measured using an upgraded technique based on the Canny–Devernay sub-pixel edge detection, yielding an average inaccuracy of less than 0.03 mm. The findings of this research showcase the successful development of a practical method for applying machine vision in the realm of the automatic detection of metal parts. This achievement is accomplished through the exploration of identification methods and size-measuring techniques for common defects found in metal parts. Consequently, the study establishes a valuable framework for effectively utilizing machine vision in the field of metal parts inspection and defect detection.

Fast prediction in marmoset reach-to-grasp movements for dynamic prey

Primates have evolved sophisticated, visually guided reaching behaviors for interacting with dynamic objects, such as insects, during foraging.1,2,3,4,5 Reaching control in dynamic natural conditions requires active prediction of the target's future position to compensate for visuo-motor processing delays and to enhance online movement adjustments.6,7,8,9,10,11,12 Past reaching research in non-human primates mainly focused on seated subjects engaged in repeated ballistic arm movements to either stationary targets or targets that instantaneously change position during the movement.13,14,15,16,17 However, those approaches impose task constraints that limit the natural dynamics of reaching. A recent field study in marmoset monkeys highlights predictive aspects of visually guided reaching during insect prey capture among wild marmoset monkeys.5 To examine the complementary dynamics of similar natural behavior within a laboratory context, we developed an ecologically motivated, unrestrained reach-to-grasp task involving live crickets. We used multiple high-speed video cameras to capture the movements of common marmosets (Callithrix jacchus) and crickets stereoscopically and applied machine vision algorithms for marker-free object and hand tracking. Contrary to estimates under traditional constrained reaching paradigms, we find that reaching for dynamic targets can operate at incredibly short visuo-motor delays around 80ms, rivaling the speeds that are typical of the oculomotor systems during closed-loop visual pursuit.18 Multivariate linear regression modeling of the kinematic relationships betweenthe hand and cricket velocity revealed that predictions of the expected future location can compensate for visuo-motor delays during fast reaching. These results suggest a critical role of visual prediction facilitatingonline movement adjustments for dynamic prey.

An active foveated gaze prediction algorithm based on a Bayesian ideal observer

Predicting human eye movements is a crucial task for understanding human behavior and has numerous applications in machine vision. Most current models for predicting eye movements are data-driven and require large datasets of recorded eye movements, which can be expensive and time-consuming to collect. In this paper, we present a novel theory-based model for predicting eye movements in a foveated visual system that maximizes information gain at each fixation. Our model uses a region-proposal network and eccentricity-based max pooling to account for the loss of detail in peripheral vision. We apply our model to predict human fixations in a visual search task for objects in real-world scenes. Unlike data-driven models, our model does not require training on large eye movement datasets and can generalize to any set of natural images and targets. We evaluate the generalization capability of our model by demonstrating its results on two publicly available visual search datasets, Ehinger and COCO-search18, without any further training on those datasets. Our model outperforms or performs comparably to data-driven models that are directly trained on human eye movement datasets.