Intelligent Vision System Research Articles

Ultraweak light-modulated heterostructure with bidirectional photoresponse for static and dynamic image perception

The human visual system’s adaptability to varying brightness levels has inspired the development of optoelectronic neuromorphic devices. However, achieving bidirectional photoresponse, essential for mimicking these functions, often requires high operation voltages or high light intensities. Here, we propose a bidirectional ZnO/CsPbBr3 heterostructure based neuromorphic image sensor array (10 × 10 pixels) capable of ultraweak light stimulation. The device demonstrates positive and negative photoconductivity through the ionization and deionization of oxygen vacancies in the ZnO channel, extendable to other ZnO/perovskites and IGZO/perovskites heterostructures. Operating at a reduced bias voltage of 2.0 V, the array achieves synaptic weight updates under green (525 nm) and UV (365 nm) light with light intensities ranging from as low as 45 nW/cm² to 15.69 mW/cm², mimicking basic synaptic functions and visual adaptation. It performs multiple image pre-processing tasks, including background denoising and encoding spatiotemporal motion, achieving 92% accuracy in pattern recognition and 100% accuracy in motion clustering. This straightforward strategy highlights a potential for intelligent visual systems capable of real-time image processing under low voltage and dark conditions.

An audiovisual cognitive optimization strategy guided by salient object ranking for intelligent visual prothesis systems

Objective.Visual prostheses are effective tools for restoring vision, yet real-world complexities pose ongoing challenges. The progress in AI has led to the emergence of the concept of intelligent visual prosthetics with auditory support, leveraging deep learning to create practical artificial vision perception beyond merely restoring natural sight for the blind.Approach.This study introduces an object-based attention mechanism that simulates human gaze points when observing the external world to descriptions of physical regions. By transforming this mechanism into a ranking problem of salient entity regions, we introduce prior visual attention cues to build a new salient object ranking (SaOR) dataset, and propose a SaOR network aimed at providing depth perception for prosthetic vision. Furthermore, we propose a SaOR-guided image description method to align with human observation patterns, toward providing additional visual information by auditory feedback. Finally, the integration of the two aforementioned algorithms constitutes an audiovisual cognitive optimization strategy for prosthetic vision.Main results.Through conducting psychophysical experiments based on scene description tasks under simulated prosthetic vision, we verify that the SaOR method improves the subjects' performance in terms of object identification and understanding the correlation among objects. Additionally, the cognitive optimization strategy incorporating image description further enhances their prosthetic visual cognition.Significance.This offers valuable technical insights for designing next-generation intelligent visual prostheses and establishes a theoretical groundwork for developing their visual information processing strategies. Code will be made publicly available.

Modular scaffolds with intelligent visual guidance system for in situ bone tissue repair

Abstract In clinical practice, the irregular shapes of traumas pose a significant challenge in rapidly manufacturing personalized scaffolds. To address these challenges, inspired by LEGO® bricks, this study proposed a novel concept of modular scaffolds and developed an innovative system based on machine vision for their rapid and intelligent assembly tailored to defect shapes. Trapezoidal interfaces effectively connect standardized bone units based on magnesium-doped silicate calcium, ensuring high stability of the modular scaffolds (compressive strength up to 135 MPa and bending strength up to 17 MPa). Through self-developed defect recognition and reconstruction algorithms, defect recognition and personalized assembly schemes for bone scaffolds can be achieved autonomously. Modular scaffolds seamlessly integrate with surrounding bone tissue, promoting new bone growth, with no apparent differences compared to fully 3D printed integral scaffolds in the skull and femur repair experiments. In summary, the adoption of modular scaffolds not only integrates personalization and standardization but also satisfies the optimal treatment window.

Intelligent Vision System with Pruning and Web Interface for Real-Time Defect Detection on African Plum Surfaces

Agriculture stands as the cornerstone of Africa’s economy, supporting over 60% of the continent’s labor force. Despite its significance, the quality assessment of agricultural products remains a challenging task, particularly at a large scale, consuming valuable time and resources. The African plum is an agricultural fruit that is widely consumed across West and Central Africa but remains underrepresented in AI research. In this paper, we collected a dataset of 2892 African plum samples from fields in Cameroon representing the first dataset of its kind for training AI models. The dataset contains images of plums annotated with quality grades. We then trained and evaluated various state-of-the-art object detection and image classification models, including YOLOv5, YOLOv8, YOLOv9, Fast R-CNN, Mask R-CNN, VGG-16, DenseNet-121, MobileNet, and ResNet, on this African plum dataset. Our experimentation resulted in mean average precision scores ranging from 88.2% to 89.9% and accuracies between 86% and 91% for the object detection models and the classification models, respectively. We then performed model pruning to reduce model sizes while preserving performance, achieving up to 93.6% mean average precision and 99.09% accuracy after pruning YOLOv5, YOLOv8 and ResNet by 10–30%. We deployed the high-performing YOLOv8 system in a web application, offering an accessible AI-based quality assessment tool tailored for African plums. To the best of our knowledge, this represents the first such solution for assessing this underrepresented fruit, empowering farmers with efficient tools. Our approach integrates agriculture and AI to fill a key gap.

Pose Estimation of a Cobot Implemented on a Small AI-Powered Computing System and a Stereo Camera for Precision Evaluation.

The precision of robotic manipulators in the industrial or medical field is very important, especially when it comes to repetitive or exhaustive tasks. Geometric deformations are the most common in this field. For this reason, new robotic vision techniques have been proposed, including 3D methods that made it possible to determine the geometric distances between the parts of a robotic manipulator. The aim of this work is to measure the angular position of a robotic arm with six degrees of freedom. For this purpose, a stereo camera and a convolutional neural network algorithm are used to reduce the degradation of precision caused by geometric errors. This method is not intended to replace encoders, but to enhance accuracy by compensating for degradation through an intelligent visual measurement system. The camera is tested and the accuracy is about one millimeter. The implementation of this method leads to better results than traditional and simple neural network methods.

Rotation-invariant image recognition using interconnected floating-gate phototransistor

Rotational invariance is fundamental for robust image recognition systems, ensuring accurate analysis irrespective of image orientation. However, existing systems predominantly reliant on software often encounter challenges such as increased computational demands and compromises between processing speed and accuracy. In this study, we propose leveraging the interconnected floating-gate (FG) structure as an effective hardware-level solution to achieve rotational invariance in image recognition. Our design features a reconfigurable two-dimensional material FG phototransistor array, where each processing unit integrates four sensory devices sharing a common FG. This configuration facilitates uniform distribution of stored charges across the interconnected FG layer, which is typically made of metal, enabling consistent application of a single weight matrix to images across varied rotational conditions. The photoactive material, tungsten diselenide (WSe2), possesses a distinctive bipolar property that facilitates both hole and electron tunneling into the FG layer. This property directly contributes to the efficiency of state transition within the setup and improves its overall adaptability. In this manner, our design achieves stable and predictable outputs in recognizing identical digital numbers regardless of their rotation, while also demonstrating variable performance essential for accurately distinguishing between different digital numbers. This dual capability guarantees both the adaptability and precision required for rotation-invariant image recognition, suggesting that our work may open up a promising venue for exploring advanced hardware designs, such as optimized interconnected FG architectures, tailored for enhancing recognition accuracy and efficiency in the field of intelligent visual systems.

Computer Vision System Based on the Analysis of Gait Features for Fall Risk Assessment in Elderly People

Up to 30% of people over the age of 60 are at high risk of falling, which can cause injury, aggravation of pre-existing conditions, or even death, with up to 684,000 fatal falls reported annually. This is due to the difficult task of establishing a preventive system for the care of the elderly, both in the hospital environment and at home. Therefore, this work proposes the development of an intelligent vision system that uses a novel methodology to infer fall risk from the analysis of kinetic and spatiotemporal gait parameters. In general, each patient is assessed using the Tinetti scale. Then, the computer vision system estimates the biomechanics of walking and obtains gait features, such as stride length, cadence, period, and range of motion. Subsequently, this information serves as input to an artificial neural network that diagnoses the risk of falling. Ninety-six participants took part in the study. The system’s performance was 99.1% accuracy, 94.4% precision, 96.9% recall, 99.4% specificity, and 95.5% F1-Score. Thus, the proposed system can evaluate the fall risk assessment, which could benefit clinics, hospitals, and even homes by allowing them to assess in real time whether a person is at high risk of falling to provide timely assistance.

ПРОГРАММНО-АЛГОРИТМИЧЕСКОЕ ОБЕСПЕЧЕНИЕ ИНТЕЛЛЕКТУАЛЬНОЙ СИСТЕМЫ ТЕХНИЧЕСКОГО ЗРЕНИЯ ДЛЯ ОПРЕДЕЛЕНИЯ ПАРАМЕТРОВ КОРНЕПЛОДОВ САХАРНОЙ СВЕКЛЫ ПРИ СОРТИРОВКЕ ПЕРЕД УКЛАДКОЙ НА ХРАНЕНИЕ

The application of methods of long-term storage of sugar beet for a period of more than 2 months is justified. It has been found that small root crops are more susceptible to temperature fluctuations and relative humidity of the environment, which leads to a change in the partial pressure of water vapor between the surface of sugar beet and the surrounding air. During storage in a sugar beet mound, the relative humidity reaches from 90 to 95% due to the removal of moisture from the surface of root crops, which is a source of beet mass losses. The criteria for the size of sugar beet root crops have been determined. The necessity of creating a method for sorting trucks at the receiving point of sugar factories is shown. It was revealed that when determining the requirements for sugar beet entering storage, it is necessary to take into account, in addition to physical and chemical parameters, the value of the specific surface area of the root crop. Therefore, it is recommended to send large root crops weighing more than 570 g to kagats. The purpose of the article is formulated, which is to develop software and algorithmic support for determining the parameters of sugar beet root crops when sorting trucks. The process of distribution of trucks at the receiving point of sugar factories by man at the present time is described. A block diagram of the algorithm of the intelligent vision system program is presented and described. The features of the created software are described, such as working with sensors, processing software or hardware errors. According to the results of the study, an algorithm for the operation of the program and software of an intelligent vision system for determining the parameters of sugar beet root crops have been developed [1].

Low-Power Perovskite Neuromorphic Synapse with Enhanced Photon Efficiency for Directional Motion Perception.

The advancement of artificial intelligent vision systems heavily relies on the development of fast and accurate optical imaging detection, identification, and tracking. Framed by restricted response speeds and low computational efficiency, traditional optoelectronic information devices are facing challenges in real-time optical imaging tasks and their ability to efficiently process complex visual data. To address the limitations of current optoelectronic information devices, this study introduces a novel photomemristor utilizing halide perovskite thin films. The fabrication process involves adjusting the iodide proportion to enhance the quality of the halide perovskite films and minimize the dark current. The photomemristor exhibits a high external quantum efficiency of over 85%, which leads to a low energy consumption of 0.6 nJ. The spike timing-dependent plasticity characteristics of the device are leveraged to construct a spiking neural network and achieve a 99.1% accuracy rate of directional perception for moving objects. The notable results offer a promising hardware solution for efficient optoneuromorphic and edge computing applications.

Deep Learning-Based Multiple Droplet Contamination Detector for Vision Systems Using a You Only Look Once Algorithm

This paper presents a practical contamination detection system for camera lenses using image analysis with deep learning. The proposed system can detect contamination in camera digital images through contamination learning utilizing deep learning, and it aims to prevent performance degradation of intelligent vision systems due to lens contamination in cameras. This system is based on the object detection algorithm YOLO (v5n, v5s, v5m, v5l, and v5x), which is trained with 4000 images captured under different lighting and background conditions. The trained models showed that the average precision improves as the algorithm size increases, especially for YOLOv5x, which showed excellent efficiency in detecting droplet contamination within 23 ms. They also achieved an average precision (mAP@0.5) of 87.46%, recall (mAP@0.5:0.95) of 51.90%, precision of 90.28%, recall of 81.47%, and F1 score of 85.64%. As a proof of concept, we demonstrated the identification and removal of contamination on camera lenses by integrating a contamination detection system and a transparent heater-based cleaning system. The proposed system is anticipated to be applied to autonomous driving systems, public safety surveillance cameras, environmental monitoring drones, etc., to increase operational safety and reliability.

Intrarow Uncut Weed Detection Using You-Only-Look-Once Instance Segmentation for Orchard Plantations.

Mechanical weed management is a drudging task that requires manpower and has risks when conducted within rows of orchards. However, intrarow weeding must still be conducted by manual labor due to the restricted movements of riding mowers within the rows of orchards due to their confined structures with nets and poles. However, autonomous robotic weeders still face challenges identifying uncut weeds due to the obstruction of Global Navigation Satellite System (GNSS) signals caused by poles and tree canopies. A properly designed intelligent vision system would have the potential to achieve the desired outcome by utilizing an autonomous weeder to perform operations in uncut sections. Therefore, the objective of this study is to develop a vision module using a custom-trained dataset on YOLO instance segmentation algorithms to support autonomous robotic weeders in recognizing uncut weeds and obstacles (i.e., fruit tree trunks, fixed poles) within rows. The training dataset was acquired from a pear orchard located at the Tsukuba Plant Innovation Research Center (T-PIRC) at the University of Tsukuba, Japan. In total, 5000 images were preprocessed and labeled for training and testing using YOLO models. Four versions of edge-device-dedicated YOLO instance segmentation were utilized in this research-YOLOv5n-seg, YOLOv5s-seg, YOLOv8n-seg, and YOLOv8s-seg-for real-time application with an autonomous weeder. A comparison study was conducted to evaluate all YOLO models in terms of detection accuracy, model complexity, and inference speed. The smaller YOLOv5-based and YOLOv8-based models were found to be more efficient than the larger models, and YOLOv8n-seg was selected as the vision module for the autonomous weeder. In the evaluation process, YOLOv8n-seg had better segmentation accuracy than YOLOv5n-seg, while the latter had the fastest inference time. The performance of YOLOv8n-seg was also acceptable when it was deployed on a resource-constrained device that is appropriate for robotic weeders. The results indicated that the proposed deep learning-based detection accuracy and inference speed can be used for object recognition via edge devices for robotic operation during intrarow weeding operations in orchards.

Ultra-thin vertical tft photosensor and photosynapse based on au-doped- graphene under transition metal selenide reaction

The development of artificial photosensitive synapses with high sensitivity and biomimetic properties that combine innovative concepts and neuromorphic architectures is crucial to achieving highly integrated and flexible intelligent visual systems. Recently, graphene heterostructure-based photosensitive synaptic transistors have been extensively studied for this purpose. However, compared to traditional transistors, vertical structure thin film transistors (VTFTs) with ultra-short channels and advantages, such as high integration, have yet to be investigated in photosensitive synapses. Here, we report an ultra-thin VTFT featuring a graphene/WxSex–1 van der Waals heterostructure that combines photonic and neuromorphic elements. We demonstrate a VTFT in which the channel layer is formed by covalently bonded WxSex–1 nanomaterials produced by introducing Se atoms on the surface of a tungsten metal thin film deposited via radio-frequency sputtering. This structure successfully simulated the main synaptic function, exhibited photosensitive synaptic responses to ultraviolet (λ = 365 nm) light, and demonstrated highly reliable electrical performance. Furthermore, the incorporation of gold nanoparticles changed the photosensitive synaptic response properties of the graphene/WxSex–1 heterostructure from excitatory to inhibitory, showing a responsivity of about ∼14 A W–1, which was attributed to the heterojunction interface resonant effects and efficient charge transfer induced by localized surface plasmons. This further enabled optical artificial synaptic applications while operating with low voltage spikes and low light intensity. This work provides a novel strategy for integrating and developing biological and nano-electronic systems.

Deep learning-based visual navigation control method for autonomous trajectory of UAVs

Abstract In this paper, a UAV intelligent visual navigation system is designed based on deep learning. To convert the pixel gray values, a Gaussian smoothing function is employed, which ensures that the main features of the visual image are preserved. A convolutional neural network is employed to mark the target with a frame using image pixels and obtain the coordinate position of the center point. Finally, the initial particles generated near the beacon are analyzed by particle filtering with color histograms, which are used to predict the position of the UAV at each autonomous trajectory point location. The control method proposed in this paper can keep the UAV attitude angle control error within 15%, and the minimum velocity error is 0.07%, as shown in the results. A deep learning-based visual navigation control system can guarantee that the UAV can accurately recognize the target in every autonomous trajectory.

Residual Tuning: Toward Novel Category Discovery Without Labels.

Discovering novel visual categories from a set of unlabeled images is a crucial and essential capability for intelligent vision systems since it enables them to automatically learn new concepts with no need for human-annotated supervision anymore. To tackle this problem, existing approaches first pretrain a neural network with a set of labeled images and then fine-tune the network to cluster unlabeled images into a few categorical groups. However, their unified feature representation hits a tradeoff bottleneck between feature preservation on labeled data and feature adaptation on unlabeled data. To circumvent this bottleneck, we propose a residual-tuning approach, which estimates a new residual feature from the pretrained network and adds it with a previous basic feature to compute the clustering objective together. Our disentangled representation approach facilitates adjusting visual representations for unlabeled images and overcoming forgetting old knowledge acquired from labeled images, with no need of replaying the labeled images again. In addition, residual-tuning is an efficient solution, adding few parameters and consuming modest training time. Our results on three common benchmarks show consistent and considerable gains over other state-of-the-art methods, and further reduce the performance gap to the fully supervised learning setup. Moreover, we explore two extended scenarios, including using fewer labeled classes and continually discovering more unlabeled sets, where the results further signify the advantages and effectiveness of our residual-tuning approach against previous approaches. Our code is available at https://github.com/liuyudut/ResTune.

Studying the Relationship between the Traffic Flow Structure, the Traffic Capacity of Intersections, and Vehicle-Related Emissions

This paper proposes a new approach to assessing the impact of changes in the traffic flow on pollutant emissions and the traffic capacity of signal-controlled intersections. We present an intelligent vision system tailored to monitor the traffic behavior at signal-controlled intersections in urban areas. Traffic cameras are used to collect real-time vehicle traffic data. Our system provides valuable insight into the relationship between traffic flows, emissions, and intersection capacity. This study shows how changes in the traffic composition reduce the traffic capacity of intersections and increase emissions, especially those involving fine dust particles. Using the combination of fuzzy logic methods and Gaussian spline distribution functions, we demonstrate the variability of these relationships and highlight the need to further study compromises between mobility and air quality. Ultimately, our results offer promising opportunities for the development of intelligent traffic management systems aimed at balancing the demands of urban mobility while minimizing environmental impact. This study demonstrates the importance of taking into account the correlation between the change in the composition of traffic queues due to a random change in the traffic flow and its impact on emissions and the traffic capacity of intersections. This study found that the presence of various groups of vehicles and their position in the queue can reduce the traffic capacity by up to 70% and increase the growth of harmful emissions by 14 fold.

Amorphous gallium oxide homojunction-based optoelectronic synapse for multi-functional signal processing

In the era of accelerated development in artificial intelligence as well as explosive growth of information and data throughput, underlying hardware devices that can integrate perception and memory while simultaneously offering the benefits of low power consumption and high transmission rates are particularly valuable. Neuromorphic devices inspired by the human brain are considered to be one of the most promising successors to the efficient in-sensory process. In this paper, a homojunction-based multi-functional optoelectronic synapse (MFOS) is proposed and testified. It enables a series of basic electrical synaptic plasticity, including paired-pulse facilitation/depression (PPF/PPD) and long-term promotion/depression (LTP/LTD). In addition, the synaptic behaviors induced by electrical signals could be instead achieved through optical signals, where its sensitivity to optical frequency allows the MFOS to simulate high-pass filtering applications in situ and the perception capability integrated into memory endows it with the information acquisition and processing functions as a visual system. Meanwhile, the MFOS exhibits its performances of associative learning and logic gates following the illumination with two different wavelengths. As a result, the proposed MFOS offers a solution for the realization of intelligent visual system and bionic electronic eye, and will provide more diverse application scenarios for future neuromorphic computing.

Zero-referenced low-light image enhancement with adaptive filter network

The current strive toward efficient intelligent visual systems suffers from challenges in the task of low-light image enhancement. To improve image perception, the low-light scenes in different illumination conditions must be properly focused. However, typical CNN-based methods use the same set of parameters for all images, which limits the capability for handling complex scenes. Meanwhile, the existing deep models integrate the low-level and high-level features by simply adding or concatenating operations, lacking unique designs for the low-light image enhancement task. To address the above challenges, we propose a zero-referenced adaptive filter network (ZAFN) for low-light image enhancement. Specifically, the adaptive filters are generated by the integration of high-level contents from multiple partial scenes. The iterative enlightening process is then conducted using the low-level features that are dynamically modulated with the adaptive filters. To alleviate the requirement of paired training data and enable zero-referenced learning, we propose a color enhancement loss, a global consistency loss, and a self-regularized denoising loss for high-quality results. Our ZAFN model, which has a light model size and low computational cost, outperforms other state-of-the-art zero-referenced methods on four popular datasets.

Welding Robot Design with Machine Learning Based Intelligent Vision System

The use of welding technologies in the manufacturing sector plays a very important role and increases its popularity thanks to developing technologies. Welding technologies are used in almost every field where production takes place, and the speed and efficiency of welding technologies have increased in these sectors in recent years. The fact that artificial intelligence techniques are at the forefront and the efficient use of these techniques together with sensors has led to development in welding technology. Thus, welding robots emerged with the support of robots with artificial intelligence techniques, and adaptive systems that can adapt to different types of workpieces working autonomously in the manufacturing sector are shaping the sector. Despite these developments, non-autonomous systems are still used today by teaching the welding points to the robots by the operators. Along with the concept robotic system to be designed and implemented within the scope of this study, it is planned to determine the welding trajectory autonomously with artificial intelligence techniques, and to perform the welding process by following the welding trajectory by the robot.

Porous Metal-organic-framework/ReS2 Heterojunction Phototransistor for Polarization-sensitive Visual Adaptation Emulation.

Visual adaptation allows organisms to accurately perceive the external world even in dramatically changing environments, from dim starlight to bright sunlight. In particular, polarization-sensitive visual adaptation can effectively process the polarized visual information that is ubiquitous in nature. However, such an intriguing characteristic still remains to be a great challenge in the semiconductor device. Herein, a novel porous metal-organic-framework phototransistor with anisotropic-ReS2 -based heterojunction is demonstrated for the polarization-sensitive visual adaptation emulation. The device exhibits the intriguing polarized sensitivity and adaptive ability due to its strong anisotropic and trapping-detrapping characteristics, respectively. A series of polarization-sensitive neuromorphic behaviors like polarization-perceptual excitatory postsynaptic current, multimode adjustable dichroic ratio, and reconfigurable sensory adaption, are experimentally demonstrated through this porous heterojunction phototransistor. More importantly, with the polarization-electricity cooperation strategy, the advanced polarization-sensitive visual adaptation with strong bottom-gate control and environment dependence is successfully realized for the first time. These results represent a significant step toward the new generation of intelligent visual perception systems in autonomous navigation and human-machine interaction, etc. This article is protected by copyright. All rights reserved.

ALFPN: Adaptive Learning Feature Pyramid Network for Small Object Detection

Object detection has become a crucial technology in intelligent vision systems, enabling automatic detection of target objects. While most detectors perform well on open datasets, they often struggle with small-scale objects. This is due to the traditional top-down feature fusion methods that weaken the semantic and location information of small objects, leading to poor classification performance. To address this issue, we propose a novel feature pyramid network, the adaptive learnable feature pyramid network (ALFPN). Our approach features an adaptive feature inspection that incorporates learnable fusion coefficients in the fusion of different levels of feature layers, aiding the network in learning features with less noise. In addition, we construct a context-aligned supervisor that adjusts the feature maps fused at different levels to avoid scaling-related offset effects. Our experiments demonstrate that our method achieves state-of-the-art results and is highly robust for the small object detection on the TT-100K, PASCAL VOC, and COCO datasets. These findings indicate that a model’s ability to extract discriminant features is positively correlated with its performance in detecting small objects.