Machine Vision Research Articles

Hybrid synaptic structure for spiking neural network realization

Neural networks and neuromorphic computing represent fundamental paradigms as alternative approaches to Von-Neumann-based implementations, advancing in the applications of deep learning and machine vision. Nonetheless, conventional semiconductor circuits encounter challenges in achieving ultra-fast processing speed and low power consumption due to their dissipative properties. Conversely, single flux quantum circuits exhibit inherent spiking behavior, showcasing their characteristics as a promising candidate for spiking neural networks (SNNs). In this work, we present a compact hybrid synapse circuit to mimic the biological interconnect functionality, enabling the weighting operations for excitatory and inhibitory impulses. Additionally, the proposed structure facilitates input accumulation, which is performed before the activation function. In the experiments, our synaptic structure interfaces with a soma circuit fabricated using a commercial Nb process, underscoring its compatibility and supporting its potential for integration into efficient neural network architectures. The weight value on the synapse is configurable by utilizing cryo-CMOS circuits, providing adaptability to the inference networks. We’ve successfully designed, fabricated, and partially tested the JJ-Synapse within our cryocooler system, enabling high-speed inference implementation for SNNs.

Vehicle Detection Algorithms for Autonomous Driving: A Review

Autonomous driving, as a pivotal technology in modern transportation, is progressively transforming the modalities of human mobility. In this domain, vehicle detection is a significant research direction that involves the intersection of multiple disciplines, including sensor technology and computer vision. In recent years, many excellent vehicle detection methods have been reported, but few studies have focused on summarizing and analyzing these algorithms. This work provides a comprehensive review of existing vehicle detection algorithms and discusses their practical applications in the field of autonomous driving. First, we provide a brief description of the tasks, evaluation metrics, and datasets for vehicle detection. Second, more than 200 classical and latest vehicle detection algorithms are summarized in detail, including those based on machine vision, LiDAR, millimeter-wave radar, and sensor fusion. Finally, this article discusses the strengths and limitations of different algorithms and sensors, and proposes future trends.

Tool wear monitoring based on scSE-ResNet-50-TSCNN model integrating machine vision and force signals

In the realm of mechanical machining, tool wear is an unavoidable phenomenon. Monitoring the condition of tool wear is crucial for enhancing machining quality and advancing automation in the manufacturing process. This paper investigates an innovative approach to tool wear monitoring that integrates machine vision with force signal analysis. It relies on a deep residual two-stream convolutional model optimized with the scSE (concurrent spatial and channel squeeze and excitation) attention mechanism (scSE-ResNet-50-TSCNN). The force signals are converted into the corresponding wavelet scale images following wavelet threshold denoising and continuous wavelet transform. Concurrently, the images undergo processing using contrast limited adaptive histogram equalization and the structural similarity index method, allowing for the selection of the most suitable image inputs. The processed data are subsequently input into the developed scSE-ResNet-50-TSCNN model for precise identification of the tool wear state. To validate the model, the paper employed X850 carbon fibre reinforced polymer and Ti–6Al–4V titanium alloy as laminated experimental materials, conducting a series of tool wear tests while collecting pertinent machining data. The experimental results underscore the model’s effectiveness, achieving an impressive recognition accuracy of 93.86%. When compared with alternative models, the proposed approach surpasses them in performance on the identical dataset, showcasing its efficient monitoring capabilities in contrast to single-stream networks or unoptimized networks. Consequently, it excels in monitoring tool wear status and promots crucial technical support for enhancing machining quality control and advancing the field of intelligent manufacturing.

A Novel Pretrained General-purpose Vision Language Model for the Vietnamese Language

Lying in the cross-section of computer vision and natural language processing, vision language models are capable of processing images and text at once. These models are helpful in various tasks: text generation from image and vice versa, image-text retrieval, or visual navigation. Besides building a model trained on a dataset for a task, people also study general-purpose models to utilize many datasets for multitasks. Their two primary applications are image captioning and visual question answering. For English, large datasets and foundation models are already abundant. However, for Vietnamese, they are still limited. To expand the language range, this work proposes a pretrained general-purpose image-text model named VisualRoBERTa. A dataset of 600k images with captions (translated MS COCO 2017 from English to Vietnamese) is introduced to pretrain VisualRoBERTa. The model’s architecture is built using Convolutional Neural Network and Transformer blocks. Fine-tuning VisualRoBERTa shows promising results on the ViVQA dataset with 34.49% accuracy, 0.4173 BLEU 4, and 0.4390 RougeL (in visual question answering task), and best outcomes on the sViIC dataset with 0.6685 BLEU 4, 0.6320 RougeL (in image captioning task).

Research on load monitoring technology of mine hoist based on machine vision

A hoist load monitoring method based on machine vision technology is proposed in this paper to address the frequent overloading accidents of mine hoists and the low safety and reliability of existing contact load monitoring technologies. The depth image of the skip undergoes time domain and spatial bilateral filtering algorithms for noise reduction, followed by conditional filtering and downsampling algorithms to remove redundant point cloud data. Point cloud recognition, extraction, segmentation, and alignment algorithms are then applied to quickly generate a skip point cloud model. A surface reconstruction optimization process combining greedy projection triangulation algorithm and void repair algorithm is proposed to obtain a smooth and complete sealing of the skip. The closed surface model volume is calculated using VTK volume function. Based on single-rope winding hoist, a load visual monitoring system is constructed for relevant experimental research. Results show that this method can accurately measure the loaded coal volume with a relative error range of 0.05%–4.13%, meeting practical application requirements while providing an effective way for non-contact accurate measurement of hoist loads in mines.

Nighttime Harvesting of OrBot (Orchard RoBot)

The Robotics Vision Lab of Northwest Nazarene University has developed the Orchard Robot (OrBot), which was designed for harvesting fruits. OrBot is composed of a machine vision system to locate fruits on the tree, a robotic manipulator to approach the target fruit, and a gripper to remove the target fruit. Field trials conducted at commercial orchards for apples and peaches during the harvesting season of 2021 yielded a harvesting success rate of about 85% and had an average harvesting cycle time of 12 s. Building upon this success, the goal of this study is to evaluate the performance of OrBot during nighttime harvesting. The idea is to have OrBot harvest at night, and then human pickers continue the harvesting operation during the day. This human and robot collaboration will leverage the labor shortage issue with a relatively slower robot working at night. The specific objectives are to determine the artificial lighting parameters suitable for nighttime harvesting and to evaluate the harvesting viability of OrBot during the night. LED lighting was selected as the source for artificial illumination with a color temperature of 5600 K and 10% intensity. This combination resulted in images with the lowest noise. OrBot was tested in a commercial orchard using twenty Pink Lady apple trees. Results showed an increased success rate during the night, with OrBot gaining 94% compared to 88% during the daytime operations.

Study on positioning and detection of crayfish body parts based on machine vision

Study on positioning and detection of crayfish body parts based on machine vision

Displacement sensing based on microscopic vision with high resolution and large measuring range

AbstractMicroimage strain sensing (MISS) is a novel piston‐type sensor based on microscopic vision. In this study, optical disc slice is used as information carriers to improve MISS. There are multiple pits on the surface of an optical disc. By using machine vision algorithms, the pits can be converted into digital information, making them scales for recording displacements. By this means, we proposed a sensing method that combines high resolution, wide range, and strong robustness. The study measured displacement under different conditions. To address inevitable factors such as pixel drift, and manufacturing errors, corresponding compensation methods were provided. The results show that the measurements closely match those of the linear variable differential transformer, with a resolution of up to 20 nm and a range approaching the sensor size. Despite the sensor's dependence on machine vision, it demonstrates strong resistance to environmental factors such as brightness and angle. Combining compensation methods for pixel drift, and manufacturing errors, this sensor can be well‐applied in various working conditions.

Detection method for contact stress distribution of tapered roller bearings

The axle box of high-speed train adopts double row tapered roller bearings as transmission parts, and the reliability of bearings directly affects the safety of train operation. Tapered roller bearings can withstand axial and radial loads, their service life being closely related to the distribution of contact stress. The test object is the axle box bearing of high-speed train. According to bearing’s structural characteristics, based on the digital speckle correlation method (DSCM) and machine vision, the contact stress distribution detection devices for rollers and complete sets of bearings are developed respectively and an effective detection method is proposed. The contact stress data of the bearing which has reached the service life and the new bearing under the condition of no lubrication and grease lubrication are collected and normalized. Through the geometric relationship, based on the measured data of the selected detection points, the normal and tangential contact stress distribution of the roller and raceway under different contact conditions is obtained by data fitting. The test can be used as an evaluation basis for the effectiveness of bearing modification and provide reference for bearing design.

Correction: Real-time Machine Vision System for the Visually Impaired

Correction: Real-time Machine Vision System for the Visually Impaired

Camera calibration optimization algorithm that uses a step function

Camera calibration is very important when planning machine vision tasks. Calibration may involve 3D reconstruction, size measurement, or careful target positioning. Calibration accuracy directly affects the accuracy of machine vision. The parameters in many image distortion models are usually applied to all image pixels. However, this may be associated with rather high pixel reprojection errors at image edges, compromising camera calibration accuracy. In this paper, we present a new camera calibration optimization algorithm that features a step function that splits images into center and edge regions. First, based on the increasing pixel reprojection errors according to the pixel distance away from the image center, we gave a flexible method to divide an image into two regions, center and boundary. Then, the algorithm automatically determines the step position, and the calibration model is rebuilt. The new model can calibrate the distortions at the center and boundary regions separately. Optimized by the method, the number of distortion parameters in the old model is doubled, and different parameters represent different distortions within two regions. In this way, our method can optimize traditional calibration models, which define a global model to describe the distortion of the whole image and get a higher calibration accuracy. Experimentally, the method significantly improved pixel reprojection accuracy, particularly at image edges. Simulations revealed that our method was more flexible than traditional methods.

Differentiable design of a double-freeform lens with multi-level radial basis functions for extended source irradiance tailoring

Freeform optics are key for generating prescribed illumination patterns from given sources, which are crucial for solid-state lighting and machine vision illumination. There is an increasing demand for compact freeform optics, which presents a substantial challenge for current design methods since the source dimensions must be considered. Most current extended-source design methods, although requiring profound knowledge of optics and mathematics, focus on the modest goal of obtaining uniform irradiance distributions. We address a more challenging design problem of generating an irradiance distribution of arbitrary shape through a double-freeform lens that can fully encompass the extended source. We propose a differentiable design method whose uniqueness lies in the representation of the double-freeform surfaces using multi-level spherical radial basis functions, which has a natural link to a multi-scale optimization technique. In addition, we employ a sequential unconstrained minimization technology complemented with Lagrange multipliers that add key feasibility constraints on lens shape and size. The proposed method is flexible, general, and efficient in designing highly compact freeform lenses for generating both simple and complex irradiance distributions, as demonstrated through the design examples. This could enable a universal solution to the extended-source design problem.

Multi-model feature aggregation for classification of laser welding images with vision transformer

The laser welding technique is quite common in various manufacturing lines, including those for lithium-ion power batteries, due to its remarkable productivity, effectiveness, and flexibility. However, it has been observed that the consistency of the welding quality is not always optimal. This study investigates the adoption of laser welding in the manufacturing of lithium-ion batteries, such as their anode, cathode and safety vent, where consistent weld quality is crucial for battery performance and safety. Therefore, evaluating the laser-welding product is indispensable for industrial production and the public domain. Several techniques have been utilized for laser welding evaluation, both destructive and non-destructive. However, these methods are ineffective and too cumbersome to be adopted in mass manufacturing. In contrast, a machine vision strategy has recently been adopted to distinguish between successful and unsuccessful laser welding items. This opened new perspectives for evaluating the quality of the laser welding product using digital image techniques. However, these methods cannot deliver outstanding performance across multiple laser-welding products and achieve optimal classification accuracy. Deep learning has evolved remarkably in recent years and gained popularity in detecting welding defects. This paper presents the observation of a Hybrid Vision Transformer (HViT) to classify the laser welding images corresponding to their feature patch images based on multi-model feature aggregation. The VGG-16 and MobileNet, which were pre-trained on ImageNet, were utilized as the core models to extract the rich features of the laser welding image. To integrate these features, the squeeze excitation module (SE) was utilized, and a multi-layer perception (MLP) approach with a label smoothing optimizer was used for classification. To determine the effectiveness of the proposed strategy, a comparative analysis was conducted with a multitude of alternative machine learning and deep learning approaches. The results indicate that the proposed strategy surpasses all other methods in terms of classification accuracy and evaluation metrics on the test dataset.

A Reconfigurable Bipolar Image Sensor for High-Efficiency Dynamic Vision Recognition.

Dynamic vision perception and processing (DVPP) is in high demand by booming edge artificial intelligence. However, existing imaging systems suffer from low efficiency or low compatibility with advanced machine vision techniques. Here, we propose a reconfigurable bipolar image sensor (RBIS) for in-sensor DVPP based on a two-dimensional WSe2/GeSe heterostructure device. Owing to the gate-tunable and reversible built-in electric field, its photoresponse shows bipolarity as being positive or negative. High-efficiency DVPP incorporating front-end RBIS and back-end CNN is then demonstrated. It shows a high recognition accuracy of over 94.9% on the derived DVS128 data set and requires much fewer neural network parameters than that without RBIS. Moreover, we demonstrate an optimized device with a vertically stacked structure and a stable nonvolatile bipolarity, which enables more efficient DVPP hardware. Our work demonstrates the potential of fabricating DVPP devices with a simple structure, high efficiency, and outputs compatible with advanced algorithms.

Movement and binding control strategy based on a new type of rebar-binding robot

Purpose The paper focuses on the issue of manual rebar-binding tasks in the construction industry, which are marked by high labor intensity, high costs and inefficient operations. The rebar-binding robots that are currently available are not fully mature. Most of them can only bind one or two nodes in one position, which leads to significant time wastage in movement. Based on a new type of rebar-binding robot, this paper aims to propose a new movement and binding control that reduces manpower and enhances efficiency. Design/methodology/approach The robot is combined with photoelectric sensors, travel switches and other sensors. It is supposed to move accurately and run in a limited area on the rebar mesh through logical judgment, speed control and position control. Machine vision is used by the robot to locate the rebar nodes and then adjusts the binding-gun position to ensure that multiple rebar nodes are bound sequentially. Findings By moving on the rebar mesh with accuracy, the robot meets the positioning accuracy requirements of the binding module, with experimental testing accuracy within 5 mm. Furthermore, its ability to bind four rebar nodes in one place results in a high efficiency and a binding effect that meets building standards. Originality/value The innovative design of the robot can adapt itself to the rebar mesh, move accurately to the target position and bind four nodes at that position, which reduces the number of movements on the mesh. Repetitive and heavy rebar-binding tasks can be efficiently completed by the robot, which saves human resources, reduces worker labor intensity and reduces construction overhead. It provides a more feasible and practical solution for using robots to bind rebar nodes.

Automation inspection in metal fabrication: enhancing sheet metal forming processes with automation, machine vision, and image correlation for quality assurance

Automation inspection in metal fabrication: enhancing sheet metal forming processes with automation, machine vision, and image correlation for quality assurance

Light‐adaptive Mimicking Retina with In Situ Image Memorization via Resistive Switching Photomemristor Arrays

AbstractArtificial visual memory systems have been of particular interest since the development of machine vision and bionic robots. Ordinarily, the conventional system architecture requires the complex integration of two functional modules, a photo‐sensor and a memory device, which greatly limits the operating efficiency and increases the extra energy consumption. Nonetheless, other simply configured optoelectronics memory devices generally face challenges of adaption in complex light environments. Here, a resistive switching (RS) perovskite‐based photomemristor is presented that mimics the retina function. The dual function of light perception and in situ storage are both achieved. In the dark condition, it exhibits impressive memory performance with a high ON/OFF ratio of 104, a long retention time of over 104 s, and a low operating voltage of 0.38 V. With illumination, it shows self‐powered, broadband photo‐detecting characteristics with responsivity of 70 mA W−1 and detectivity of 7.5 × 1010 Jones. More importantly, benefiting from the material dual‐phase configuration, the highly steady photo‐adjusted RS windows are achieved. Its light‐adaptive memory application in dynamic environments is further demonstrated using the mimicking retina for a machine eye. This work can provide a strategy for enhanced RS photomemristor and its application in changing and varied scenarios.

Navigation line extraction algorithm for corn spraying robot based on YOLOv8s‐CornNet

AbstractThe continuous and close combination of artificial intelligence technology and agriculture promotes the rapid development of smart agriculture, among which the agricultural robot navigation line recognition algorithm based on deep learning has achieved great success in detection accuracy and detection speed. However, there are still many problems, such as the large size of the algorithm is difficult to deploy in hardware equipment, and the accuracy and speed of crop row detection in real farmland environment are low. To solve the above problems, this paper proposed a navigation line extraction algorithm for corn spraying robot based on YOLOv8s‐CornNet. First, the Convolution (Conv) module and C2f module of YOLOv8s network are replaced with Depthwise Convolution (DWConv) module and PP‐LCNet module respectively to reduce the parameters (Params) and giga floating‐point operations per second of the network, so as to achieve the purpose of network lightweight. Second, to reduce the precision loss caused by network lightweight, the spatial pyramid pooling fast module in the backbone network is changed to atrous spatial pyramid pooling faster module to improve the accuracy of network feature extraction. Meanwhile, normalization‐based attention module is introduced into the network to improve the network's attention to corn plants. Then the corn plant was located by using the midpoint of the corn plant detection box. Finally, the least square method is used to extract the corn crop row line, and the middle line of the corn crop row line is the navigation line of the corn spraying robot. From the experimental results, it can be seen that the navigation line extraction algorithm proposed in this paper ensures both the real‐time and accuracy of the navigation line extraction of the corn spraying robot, which contributes to the development of the visual navigation technology of agricultural robots.

HDR vision sensor with neuro-memristive skin detection for edge computing

Human skin classification is an essential task for several machine vision applications such as human-machine interfaces, people/object tracking, and classification. In this paper, we describe a hybrid CMOS/memristor vision sensor architecture embedding skin detection over a wide dynamic range. In-sensor RGB to rg-chromaticity color-space conversion is executed on-the-fly through a pixel-level automatic exposure time control. Each pixel of the array delivers two pre-filtered analog signals, the r and g values, suitable for being efficiently classified as skin or non-skin through an analog memristive neural network (NN), without the need for any further signal processing. Moreover, we study the NN performance and theorize how it should be added in the hardware. The skin classifier is organized in an array of column-level memristor-based NN to exploit the nano-scale device characteristics and non-volatile analog memory capabilities, making the proposed sensor architecture highly flexible, customizable for various use-case scenarios, and low-power. The output is a skin bitmap that is robust against variations of the illuminant color and intensity.

Accurate and rapid quantification of tea saponins using a thin-layer chromatography (TLC) method based on hemolysis and machine vision

Tea saponins are increasingly used in the food industries due to their biological activities. Current methods for the quantification of tea saponins are either inaccurate or time consuming and costly. In this study, a quantitative thin-layer chromatographic (TLC) method using machine vision and based on the hemolytic activity (HA) of tea saponins was developed. The optimized method consisted of ethanol extraction, sample loading, separation, color development, and smartphone-machine vision quantification. The new HA-TLC method was compared with UV spectrophotometric and liquid chromatographic methods, and achieved higher precision (RSD% = 0.75%) and accuracy (96.4%-106.2% recovery), with a limit of detection of 0.285 μg and limit of quantification of 0.95 μg. The method enables simultaneous qualitative and quantitative determination of saponins from different parts of tea plant extracts with high specificity, yet rapid and inexpensive, with high throughput and low solvent consumption, providing new options for saponin analysis in the food industry.