Machine Vision Applications Research Articles

Application of Traffic Cone Target Detection Algorithm Based on Improved YOLOv5

To improve the automation level of highway maintenance operations, the lightweight YOLOv5-Lite-s neural network was deployed in embedded devices to assist an automatic traffic cone retractor in completing recognition and positioning operations. The system used the lightweight shuffle Net network as a backbone for feature extraction, replaced convolutional layers with focus modules to reduce computational complexity, and reduced the use of the C3 layer to increase network speed, thereby meeting the speed and accuracy requirements of traffic cone placement and retraction operations while maintaining acceptable model inference accuracy. The experimental results show that the network could maintain recognition accuracy and speed values of around 89% and 9 fps under different working conditions such as varying distances, lighting conditions, and occlusions, meeting the technical requirements for deploying and retrieving cones at a speed of 30 cones per minute when the operating vehicle’s speed was 20 km/h. The automatic traffic cone placement and retraction system operated accurately and stably, achieving the application of machine vision in traffic cone retraction operations.

Cloud IaaS Optimization Using Machine Vision at the IoT Edge and the Grid Sensing Algorithm.

Security grids consisting of High-Definition (HD) Internet of Things (IoT) cameras are gaining popularity for organizational perimeter surveillance and security monitoring. Transmitting HD video data to cloud infrastructure requires high bandwidth and more storage space than text, audio, and image data. It becomes more challenging for large-scale organizations with massive security grids to minimize cloud network bandwidth and storage costs. This paper presents an application of Machine Vision at the IoT Edge (Mez) technology in association with a novel Grid Sensing (GRS) algorithm to optimize cloud Infrastructure as a Service (IaaS) resource allocation, leading to cost minimization. Experimental results demonstrated a 31.29% reduction in bandwidth and a 22.43% reduction in storage requirements. The Mez technology offers a network latency feedback module with knobs for transforming video frames to adjust to the latency sensitivity. The association of the GRS algorithm introduces its compatibility in the IoT camera-driven security grid by automatically ranking the existing bandwidth requirements by different IoT nodes. As a result, the proposed system minimizes the entire grid's throughput, contributing to significant cloud resource optimization.

Broadband and parallel multiple-order optical spatial differentiation enabled by Bessel vortex modulated metalens

Optical analog image processing technology is expected to provide an effective solution for high-throughput and real-time data processing with low power consumption. In various operations, optical spatial differential operations are essential in edge extraction, data compression, and feature classification. Unfortunately, existing methods can only perform low-order or selectively perform a particular high-order differential operation. Here, we propose and experimentally demonstrate a Bessel vortex modulated metalens composed of a single complex amplitude metasurface, which can perform multiple-order radial differential operations over a wide band by presetting the order of the corresponding Bessel vortex. This architecture further enables angle multiplexing to create multiple information channels that synchronously perform multi-order spatial differential operations, indicating the superiority of the proposed devices in parallel processing. Our approach may find various applications in artificial intelligence, machine vision, autonomous driving, and advanced biomedical imaging.

Vision-guided robot calibration using photogrammetric methods

We propose novel photogrammetry-based robot calibration methods for industrial robots that are guided by cameras or 3D sensors. Compared to state-of-the-art methods, our methods are capable of calibrating the robot kinematics, the hand–eye transformations, and, for camera-guided robots, the interior orientation of the camera simultaneously. Our approach uses a minimal parameterization of the robot kinematics and hand–eye transformations. Furthermore, it uses a camera model that is capable of handling a large range of complex lens distortions that can occur in cameras that are typically used in machine vision applications. To determine the model parameters, geometrically meaningful photogrammetric error measures are used. They are independent of the parameterization of the model and typically result in a higher accuracy. We apply a stochastic model for all parameters (observations and unknowns), which allows us to assess the precision and significance of the calibrated model parameters. To evaluate our methods, we propose novel procedures that are relevant in real-world applications and do not require ground truth values. Experiments on synthetic and real data show that our approach improves the absolute positioning accuracy of industrial robots significantly. By applying our approach to two different uncalibrated UR3e robots, one guided by a camera and one by a 3D sensor, we were able to reduce the RMS evaluation error by approximately 85% for each robot.

Enhancing industrial IoT with edge computing and computer vision: An analog gauge visual digitization approach

The advent of Industry 4.0 has led to the widespread adoption of Industrial Internet of Things (IIoT) and Computer Vision technologies in manufacturing. However, handling the massive data generated by IIoT poses challenges. To address this, we propose a software and hardware architecture that combines edge computing with IIoT to enable computer vision applications on manufacturing floors. This research contributes to integrating edge computing and IIoT with Computer Vision technologies in manufacturing facilities, offering a cost-effective and scalable solution for machine vision applications. This approach benefits Small and Medium Manufacturers (SMMs) by utilizing low-cost hardware and open-source technologies, enabling them to harness Industry 4.0 advantages. Our approach involves IoT devices using Raspberry Pi microcomputers and USB webcams as clients and a Python-based edge node server software as the central component. Our test case study results show that a centralized edge-computing solution can offer several useability benefits and a significant performance increase over using a non-edge-enabled IIoT approach. The proposed architecture is tested, validated, and verified through a case study focused on the digitization of the readings of analog gauges. Our experimentation results can offer valuable insights and guidance for the broader adoption of edge-enabled IIoT solutions in manufacturing contexts.

Application of machine vision in food computing: A review

With global intelligence advancing and the awareness of sustainable development growing, artificial intelligence technology is increasingly being applied to the food industry. This paper, grounded in practical application cases, reviews the current research status and prospects of machine vision–based image recognition technology in food computing. It explores the general workflow of image recognition, applications based on traditional machine learning and deep learning methods. The paper covers areas including food safety detection, dietary nutrition analysis, process monitoring, and enterprise management model optimization. The aim is to provide a solid theoretical foundation and technical guidance for the integration and cross-fertilization of the food industry with artificial intelligence technology.

Nonclassical Spin‐Multiplexing Metasurfaces Enabled Multifunctional Meta‐Scope

AbstractDielectric metasurfaces have emerged as attractive devices for advanced imaging systems because of their high efficiency, ability of wavefront manipulation, and lightweight. The classical spin‐multiplexing metasurfaces can only provide two orthogonal circular polarization channels and require high phase contrast which limits their applications. Here, metasurfaces with arbitrary three independent channels are demonstrated by proposing a nonclassical spin‐multiplexing approach exploring the low refractive index meta‐atoms. A zoom microscope with on‐axis tri‐foci and a synchronous achiral‐chiral microscope with in‐plane tri‐foci based on silicon nitride metasurfaces are experimentally demonstrated. Based on the on‐axis tri‐foci metasurface, singlet zoom imaging with three magnifications and a broadband response (blue to red) based on a single metasurface is first demonstrated. A compact microscope (meta‐scope) consisting of two metasurfaces with three magnifications of 9.5, 10, and 29X with diffraction‐limited resolutions is further constructed, respectively. Utilizing the in‐plane tri‐foci metasurface, a singlet microscope with three achiral‐chiral channels is demonstrated. It offers a magnification of 53X and a diffraction‐limited resolution, enabling simultaneous imaging of an object's achiral and chiral properties. Our multifunctional metasurfaces and meta‐scope approaches could boost the applications in biological imaging and machine vision.

An efficient nano-design of image processor circuits for morphology operations based on quantum dots

Quantum-dot cellular automata (QCA) are one of the most promising alternatives to traditional VLSI technology despite significant current obstacles. The QCA has the advantages of very low power dissipation, faster switching speed, and extremely low circuit area, which can be used in designing nano-scale image processing circuits. Morphological operations and processing of digital image processing is a significant topic for researchers because it is widely used for analyzing, enhancing, and modifying images to extract meaningful information or improve their visual quality. Image processing is also used for image retrieval and enhancement, image compression, object recognition, machine vision, and medical applications. QCA technology, as a new and leading technology with great potential, can play a fundamental role in morphological operations, processing digital images, image editing, medical imaging, facial recognition, and autonomous vehicles. In recent years, researchers in this field have presented many circuits, but they have many flaws in terms of speed, accuracy, and area consumption, and the need to create more efficient circuits is felt more than ever. Therefore, in this article, a new design for morphological operations and processing digital images is presented using QCA technology. This paper presents a new efficient QCA-based implementation of image processing based on the direct interactions between the QCA cells. This circuit uses two majority gates of five new inputs to produce the output and produces the desired output. In addition, a comparison and analysis of the area and clocking complexity, design cost, and energy dissipation through simulation using QCADesigner and QCADesigner-E are done. The results show that the presented circuit produces the expected and correct output results in 0.75 clock phases, and the obtained results show the high speed and low consumption space of the presented circuit. In addition, the presented circuit performs better than the previous best circuits in terms of quantum cost and delay, and according to the mentioned advantages, it can be used to improve and expand other circuits in image processing.

Quantum-tunneling deep neural network for optical illusion recognition

The discovery of the quantum tunneling (QT) effect—the transmission of particles through a high potential barrier—was one of the most impressive achievements of quantum mechanics made in the 1920s. Responding to the contemporary challenges, I introduce a deep neural network (DNN) architecture that processes information using the effect of QT. I demonstrate the ability of QT-DNN to recognize optical illusions like a human. Tasking QT-DNN to simulate human perception of the Necker cube and Rubin’s vase, I provide arguments in favor of the superiority of QT-based activation functions over the activation functions optimized for modern applications in machine vision, also showing that, at the fundamental level, QT-DNN is closely related to biology-inspired DNNs and models based on the principles of quantum information processing.

Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array

The bioinspired camera, comprising a single lens and a curved image sensor—a photodiode array on a curved surface—, was born of flexible electronics. Its economical build lends itself well to space-constrained machine vision applications. The curved sensor, much akin to the retina, helps image focusing, but the curvature also creates a problem of image distortion, which can undermine machine vision tasks such as object recognition. Here we report an anti-distortion single-lens camera, where 4096 silicon photodiodes arrayed on a curved surface in a nonuniform pattern assimilated to the distorting optics are the key to anti-distortion engineering. That is, the photo-pixel distribution pattern itself is warped in the same manner as images are warped, which correctively reverses distortion. Acquired images feature no appreciable distortion across a 120° horizontal view, as confirmed by their neural-network recognition accuracies. This distortion correction via photo-pixel array reconfiguration is a form of in-sensor computing.

Implementation of Integrated Development Environment for Machine Vision-Based IEC 61131-3

IEC 61131-3 is an international standard for developing standardized software for automation and control systems. Machine vision systems are a prominent technology in the field of computer vision and are widely used in various industries, such as manufacturing, robotics, healthcare, and automotive, and are often combined with AI technologies. In industrial automation systems, software developed for defect detection or product classification typically involves separate systems for automation and machine vision programs, leading to increased system complexity and unnecessary resource wastage. To address these limitations, this study proposes an IEC 61131-3-based integrated development environment for programmable machine vision. We selected 11 APIs commonly used in machine vision systems, evaluated their functions in an IEC 61131-3 compliant development environment, and measured the performance of representative machine vision applications. This approach demonstrates the feasibility of developing PLC and machine vision programs within a single-controller system. We investigated the impact of controller performance on function execution.

Deep underwater image compression for enhanced machine vision applications

Underwater image compression is fundamental in underwater visual applications. The storage resources of autonomous underwater vehicles (AUVs) and underwater cameras are limited. By employing effective image compression methods, it is possible to optimize the resource utilization of these devices, thereby extending the operational time underwater. Current image compression methods neglect the unique characteristics of the underwater environment, thus failing to support downstream underwater visual tasks efficiently. We propose a novel underwater image compression framework that integrates frequency priors and feature decomposition fusion in response to these challenges. Our framework incorporates a task-driven feature decomposition fusion module (FDFM). This module enables the network to understand and preserve machine-friendly information during the compression process, prioritizing task relevance over human visual perception. Additionally, we propose a frequency-guided underwater image correction module (UICM) to address noise issues and accurately identify redundant information, enhancing the overall compression process. Our framework effectively preserves machine-friendly features at a low bit rate. Extensive experiments across various downstream visual tasks, including object detection, semantic segmentation, and saliency detection, consistently demonstrated significant improvements achieved by our approach.

Meta‐Optics Based Parallel Convolutional Processing for Neural Network Accelerator

AbstractConvolutional neural networks (CNNs) have shown great performance in computer vision tasks, from image classification to pattern recognition. However, superior performance arises at the expense of high computational costs, which restricts their employment in real‐time decision‐making applications. Computationally intensive convolutions can be offloaded to optical metasurfaces, enabling sub‐picosecond latency and nearly zero energy consumption, but the currently reported approaches require additional bulk optics and can only process polarized light, which limits their practical usages in integrated lightweight systems. To solve these challenges, a novel design of the metasurface‐based optical convolutional accelerator is experimentally demonstrated, offering an ultra‐compact volume of 0.016 , a low cross‐talk of ‐20 dB, polarization insensitivity, and is capable of implementing multiple convolution operations and extracting simultaneously various features from light‐encoded images. The ultra‐compact metasurface‐based optical accelerator can be compactly integrated with a digital imaging system to constitute an optical‐electronic hybrid CNN, which experimentally achieves a consistent accuracy of 96 % in arbitrarily polarized MNIST digits classification. The proposed ultra‐compact metasurface‐based optical convolutional accelerator paves the way for power‐efficient edge‐computing platforms for a range of machine vision applications.

Is human perception reliable? Toward illumination robust food freshness prediction from food appearance — Taking lettuce freshness evaluation as an example

Machine vision-based food quality evaluation systems have achieved great attention in industry and academia in recent years because of their high-throughput and non-invasive properties. In practice, the environmental illumination conditions variations would cause visual evaluation bias for both human and machine perceptions. Compared to other existing studies which take sample images and human visual grading under fixed illumination conditions, this study first investigated the environmental illumination effects on the performance of visual-based food grading for both humans and machines. Taking lettuce samples as an example, an image dataset that considered the environmental illumination variations was first established. This dataset encompasses human visual grading scores obtained from sensory panels as well. In contrast to current studies that utilize a single grader or L*a*b* color features to assess sample freshness, our study reveals a substantial impact of environmental illumination on both human perception (p<0.0001) and L*a*b* color features (p<0.0001). In order to enhance the performance of machine vision-based freshness prediction, multitask learning protocols were incorporated into the proposed new network architectures. This allowed the simultaneous prediction of both sample freshness and illumination conditions. In comparison to commonly used generic convolutional neural network models and vision transformer models, the newly proposed model exhibited superior freshness prediction performance. It minimized the prediction error by 20.36%, outperforming the generic ResNet model. This research represents the first quantitative study addressing human and machine perceptions under varied illumination conditions for food quality evaluation. The findings are anticipated to play a pivotal role in expediting the integration of machine vision applications into food engineering practices.

Solution-Processed Ultralow Voltage Organic Transistors With Sharp Switching for Adaptive Visual Perception.

Neuromorphic visual systems can emulate biological retinal systems to perceive visual information under different levels of illumination, making them have considerable potential for future intelligent vehicles and vision automation. However, the complex circuits and high operating voltages of conventional artificial vision systems present great challenges for device integration and power consumption. Here, bioinspired synaptic transistors based on organic single crystal phototransistors are reported, which exhibit excitation and inhibition synaptic plasticity with time-varying. By manipulating the charge dynamics of the trapping centers of organic crystal-electret vertical stacks, organic transistors can operate below 1V with record high on/off ratios close to 108 and sharp switching with a subthreshold swing of 59.8mV dec-1. Moreover, the approach offers visual adaptation with highly localized modulation and over 98.2% recognition accuracy under different illumination levels. These bioinspired visual adaptation transistors offer great potential for simplifying the circuitry of artificial vision systems and will contribute to the development of machine vision applications.

Hybrid synaptic structure for spiking neural network realization

Neural networks and neuromorphic computing represent fundamental paradigms as alternative approaches to Von-Neumann-based implementations, advancing in the applications of deep learning and machine vision. Nonetheless, conventional semiconductor circuits encounter challenges in achieving ultra-fast processing speed and low power consumption due to their dissipative properties. Conversely, single flux quantum circuits exhibit inherent spiking behavior, showcasing their characteristics as a promising candidate for spiking neural networks (SNNs). In this work, we present a compact hybrid synapse circuit to mimic the biological interconnect functionality, enabling the weighting operations for excitatory and inhibitory impulses. Additionally, the proposed structure facilitates input accumulation, which is performed before the activation function. In the experiments, our synaptic structure interfaces with a soma circuit fabricated using a commercial Nb process, underscoring its compatibility and supporting its potential for integration into efficient neural network architectures. The weight value on the synapse is configurable by utilizing cryo-CMOS circuits, providing adaptability to the inference networks. We’ve successfully designed, fabricated, and partially tested the JJ-Synapse within our cryocooler system, enabling high-speed inference implementation for SNNs.

HDR vision sensor with neuro-memristive skin detection for edge computing

Human skin classification is an essential task for several machine vision applications such as human-machine interfaces, people/object tracking, and classification. In this paper, we describe a hybrid CMOS/memristor vision sensor architecture embedding skin detection over a wide dynamic range. In-sensor RGB to rg-chromaticity color-space conversion is executed on-the-fly through a pixel-level automatic exposure time control. Each pixel of the array delivers two pre-filtered analog signals, the r and g values, suitable for being efficiently classified as skin or non-skin through an analog memristive neural network (NN), without the need for any further signal processing. Moreover, we study the NN performance and theorize how it should be added in the hardware. The skin classifier is organized in an array of column-level memristor-based NN to exploit the nano-scale device characteristics and non-volatile analog memory capabilities, making the proposed sensor architecture highly flexible, customizable for various use-case scenarios, and low-power. The output is a skin bitmap that is robust against variations of the illuminant color and intensity.

Shadow Detection and Elimination for Robot and Machine Vision Applications

Shadow removal is crucial for robot and machine vision as the accuracy of object detection is greatly influenced by the uncertainty and ambiguity of the visual scene. In this paper, we introduce a new algorithm for shadow detection and removal based on different shapes, orientations, and spatial extents of Gaussian equations. Here, the contrast information of the visual scene is utilized for shadow detection and removal through five consecutive processing stages. In the first stage, contrast filtering is performed to obtain the contrast information of the image. The second stage involves a normalization process that suppresses noise and generates a balanced intensity at a specific position compared to the neighboring intensities. In the third stage, the boundary of the target object is extracted, and in the fourth and fifth stages, respectively, the region of interest (ROI) is highlighted and reconstructed. Our model was tested and evaluated using realistic scenarios which include outdoor and indoor scenes. The results reflect the ability of our approach to detect and remove shadows and reconstruct a shadow free image with a small error of approximately 6%.

Application of Machine Vision Based on the Second Law of Thermodynamics in High-Temperature Industrial Inspection

Application of Machine Vision Based on the Second Law of Thermodynamics in High-Temperature Industrial Inspection

Single-metalens-assisted polarization imaging and edge detection for target recognition

Simultaneous capture of various light information, including polarization and edge information of the objects, has consistently been a fundamental concern within the field of target recognition. However, these tasks are typically accompanied by bulky optical components and active illumination methods, which significantly restricts their use in compact and lightweight applications. Here, we demonstrate a metalens-assisted imaging system that can simultaneously achieve polarization imaging and optoelectronic edge detection in a single shot with low consumption. The dielectric metalens is designed to achieve polarization imaging by dispersing the input polarized light into two orthogonal components, resulting in optoelectronic isotropic edge detection of two-dimensional images after digital post-processing. Compared with the algorithmic methods using a convolution kernel, the proposed system has a much lower computational complexity. The work presented in this study demonstrates the potential applications in machine vision and paves the way for the development of compact target recognition and real-time image processing systems.