Vision Sensor Research Articles

Cellulose-based halochromic sensor for real-time surveillance of spoilage of packed fish

The development of a biopolymer-based color-changing visual sensor for monitoring the spoilage of fishery products is described in this study. At first, a cellulose-based composite film containing mixed-indicator dyes was fabricated. The film was characterized using UV–visible and FT-IR spectroscopy, scanning electron microscopy, thermogravimetric analysis, lab, and RGB measurements. The halochromic properties of the film were evaluated through ammonia test and colorimetry. The developed film sensor is tested for monitoring the spoilage of packed seer fish fillets at room temperature for up to 48 h and at 4 °C for up to 10 days. The sensor displayed distinct color changes correlating with the spoilage of the packed fish. Analysis of total volatile basic nitrogen and pH levels of the samples revealed a correlation between the halochromism of the sensor and the spoilage of the fish. The color-changing pattern of the sensor was consistent with the results of microbial analysis. The present study reveals a promising material combination for fabricating simple visual sensors from biomaterials, which can be used to monitor the real-time deterioration of packaged fishery products.

Computer vision-based substructure isolation method for localized damage identification

In recent years, vision-based structural damage identification techniques have garnered significant attention due to their simplicity and cost-effectiveness. Nevertheless, dynamic response-based vision methods face challenges related to the limitations of the field of view (FOV), i.e., the extent of the FOV is often sacrificed in order to ensure sufficient monitoring accuracy, which reduces the amount of data available for structural health monitoring (SHM). This study presents a solution to this problem by employing the substructure isolation method (SIM), which focuses on local vibrations of the structure and enables local damage identification by isolating the area of interest. Additionally, a method combining kernelized correlation filter (KCF) with sub-pixel template matching is used to extract vibration information recorded by visual sensor. This strategy balances both the efficiency and accuracy of displacement extraction. In numerical simulations, the performance of the SIM based on acceleration response and displacement response is compared, demonstrating the potential compatibility of visual sensors with the SIM. Finally, the effectiveness of the proposed vision-based local damage identification method is validated through vibration testing of a shear frame model.

Identifying the Laser Stripes via Ray Model for Multiline Structured Light Stereo Vision Sensors

Identifying the Laser Stripes via Ray Model for Multiline Structured Light Stereo Vision Sensors

Analysis of modeling nasal narrow space based on visual slam

Abstract. Now, surgeries are becoming more stable, safe, efficient and low-cost. During the surgical treatment of nasal diseases, surgical robotic robotics can help operate accurately and reduce the discomfort after surgery. However, due to the internal space of the nasal cavity being relatively narrow, it is difficult for the nasal surgical robot to contain multiple vision sensors and the monocular camera could not get information about the depth of the 3D objects in the scene, so the existing surgical robots cannot accomplish the three-dimensional modeling about the internal space of nasal cavity well. In practice, doctors still have to analyze pictures from the robotic, which may decrease the efficiency of the surgery and increase the risk to patients. This article designed a SLAM algorithm framework based on a depth estimation network, it can simulate the internal structure of the nasal cavity more accurately through pictures, which come from monocular endoscopic on surgical robotic. The insights gained in this study verify that the method of image segmentation can also make the depth representation of the nasal internal space more accurate and this method may help robots realize their self-position in the narrow area of the nasal cavity, which lays the foundations for the development of fully autonomous surgical robots.

Design of Single-Camera Mirrored Binocular Vision Sensor With Single-Plane Mirror

Design of Single-Camera Mirrored Binocular Vision Sensor With Single-Plane Mirror

Spiking Neural Networks for Real-Time Pedestrian Street-Crossing Detection Using Dynamic Vision Sensors in Simulated Adverse Weather Conditions

This study explores the integration of Spiking Neural Networks (SNNs) with Dynamic Vision Sensors (DVSs) to enhance pedestrian street-crossing detection in adverse weather conditions—a critical challenge for autonomous vehicle systems. Utilizing the high temporal resolution and low latency of DVSs, which excel in dynamic, low-light, and high-contrast environments, this research evaluates the effectiveness of SNNs compared to traditional Convolutional Neural Networks (CNNs). The experimental setup involved a custom dataset from the CARLA simulator, designed to mimic real-world variability, including rain, fog, and varying lighting conditions. Additionally, the JAAD dataset was adopted to allow for evaluations using real-world data. The SNN models were optimized using Temporally Effective Batch Normalization (TEBN) and benchmarked against well-established deep learning models, concerning their accuracy, computational efficiency, and energy efficiency in complex weather conditions. This study also conducted a comprehensive analysis of energy consumption, highlighting the significant reduction in energy usage achieved by SNNs when processing DVS data. The results indicate that SNNs, when integrated with DVSs, not only reduce computational overhead but also dramatically lower energy consumption, making them a highly efficient choice for real-time applications in autonomous vehicles (AVs).

The influence of neighbor selection on self-organized UAV swarm based on finite perception vision.

Recently, vision-based unmanned aerial vehicle (UAV) swarming has emerged as a promising alternative that can overcome the adaptability and scalability limitations of distributed and communication-based UAV swarm systems. While most vision-based control algorithms are predicated on the detection of neighboring objects, they often overlook key perceptual factors such as visual occlusion and the impact of visual sensor limitations on swarm performance. To address the interaction problem of neighbor selection at the core of self-organizing UAV swarm control, a perceptually realistic finite perception visual (FPV) neighbor selection model is proposed, which is based on the lateral visual characteristics of birds, incorporates adjustable lateral visual field widths and orientations, and is able to ignore occluded agents. Based on the FPV model, a neighbor selection method based on the Acute Angle Test (AAT) is proposed,
which overcomes the limitation that the traditional neighbor selection mechanism can only interact with the nearest neighboring agents. A large number of Monte Carlo simulation comparison experiments show that the proposed FPV+AAT neighborselection mechanism can reduce the redundant communication burden between large scale self-organized UAV swarms, and outperforms the traditional neighbor selection method in terms of order, safety, union, connectivity, and noise resistance.

High-performance surface defect detection of aluminum substrate based on event camera

Abstract Traditional industrial surface defect detection often employs CCD/CMOS cameras, but they are difficult to detect the minute defects on aluminum substrates in highly dynamic industrial scenes due to their nature. Event camera is a novel high-resolution vision sensor which measures per-pixel brightness changes in an asynchronous manner and outputs as event information flow (EIF). Small and weak defects on aluminum substrate can be captured by event camera effectively, but the EIF contain a large amount of noise, making it difficult to perform accurate and high-precision defect detection. To address this problem, we propose a frame aggregation method to realize good event information flow processing (EIFP), and then use an improved circle detection method to locate the aluminum substrate in each frame, removing abundant event information outside the aluminum substrate. Subsequently, we enhance the event signals under different frames based on optical flow tracking using multiple features, and construct a semi-supervised detector based on pseudo-labels to achieve high-precision defect localization. Finally, considering the small inter-class differences in defects on the surface of aluminum substrates, we construct a defect class corrector based on ensemble learning to enhance the ability to determine defect classes, achieving high-precision automatic quality inspection of aluminum substrate surfaces. The performance of our method is compared with other advanced methods based on event camera data of aluminum substrates in real industrial scenarios. The experimental results show that our method has improved the detection accuracy by about 10% and the classification accuracy by about 25% compared to the original state-of-the-art methods.

A New Deep Learning Methodology for Alarm Supervision in Marine Power Stations

Marine engineering officers operate and maintain the ship’s machinery during normal navigation. Most accidents on board are related to human factors which, at the same time, are associated with the workload of the crew members and the working environment. The number of alarms is so high that, most of the time, instead of helping to prevent accidents, it causes more stress for crew members, which can result in accidents. Convolutional Neural Networks (CNNs) are being employed in the recognition of images, which depends on the quality of the images, the image recognition algorithm, and the very complex configuration of the neural network. This research study aims to develop a user-friendly image recognition tool that may act as a visual sensor of alarms adjusted to the particular needs of the ship operator. To achieve this, a marine engineering simulator was employed to develop an image recognition tool that advises marine engineering officers when they are conducting their maintenance activities, with the aim to reduce their stress as a work risk prevention tool. Results showed adequate accuracy for three-layer Convolutional Neural Networks and balanced data, and the use of external cameras stands out for user-friendly applications.

Bidirectional Photovoltage-Driven Oxide Transistors for Neuromorphic Visual Sensors.

Biological vision is one of the most important parts of the human perception system. However, emulating biological visuals is challenging because it requires complementary photoexcitation and photoinhibition. Here, the study presents a bidirectional photovoltage-driven neuromorphic visual sensor (BPNVS) that is constructed by monolithically integrating two perovskite solar cells (PSCs) with dual-gate ion-gel-gated oxide transistors. PSCs act as photoreceptors, converting external visual stimuli into electrical signals, whereas oxide transistors generate neuromorphic signal outputs that can be adjusted to produce positive and negative photoresponses. This device mimics the human vision system's ability to recognize colored and color-mixed patterns. The device achieves a static color recognition accuracy of 96% by utilizing the reservoir computing system for feature extraction. The BPNVS mem-reservoir chip is also proposed for handing object movement and dynamic color recognition. This work is a significant step forward in neuromorphic sensing and complex pattern recognition.

Study of Contactless Computer Vision-Based Road Condition Estimation Methods Within the Framework of an Operational Design Domain Monitoring System

Deployment of Level 3 and Level 4 autonomous vehicles (AVs) in urban environments is significantly constrained by adverse weather conditions, limiting their operation to clear weather due to safety concerns. Ensuring that AVs remain within their designated Operational Design Domain (ODD) is a formidable challenge, making boundary monitoring strategies essential for safe navigation. This study explores the critical role of an ODD monitoring system (OMS) in addressing these challenges. It reviews various methodologies for designing an OMS and presents a comprehensive visualization framework incorporating trigger points for ODD exits. These trigger points serve as essential references for effective OMS design. The study also delves into a specific use case concerning ODD exits: the reduction in road friction due to adverse weather conditions. It emphasizes the importance of contactless computer vision-based methods for road condition estimation (RCE), particularly using vision sensors such as cameras. The study details a timeline of methods involving classical machine learning and deep learning feature extraction techniques, identifying contemporary challenges such as class imbalance, lack of comprehensive datasets, annotation methods, and the scarcity of generalization techniques. Furthermore, it provides a factual comparison of two state-of-the-art RCE datasets. In essence, the study aims to address and explore ODD exits due to weather-induced road conditions, decoding the practical solutions and directions for future research in the realm of AVs.

Lanthanide Functionalized Carbon Quantum Dots for White Light Emission, pH Sensing, and Co (II) Detection.

Carbon quantum dots (CQDs) with fluorescence emission have been widely studied for versatile applications, but facile tunability of the spectral properties of CQDs by doping remains to be further explored. Herein, employing lanthanide ion Eu3+ as a dopant and activator, a simple and efficient synthesis route for pure CQDs and Eu-CQDs was demonstrated using N, N-dimethylformamide, oleic acid, and oleylamine as precursors for carbon sources. In comparison, with the popular citric acid precursor, the as-prepared CQDs and Eu-CQDs exhibited an obviously smaller particle size (1.72 ± 0.29 nm) and a more uniform distribution. Through systematic optimization of Eu3+ doping for the Eu-CQD system, colorful light emissions in the visible range were first realized under excitation of ultraviolet (UV) ∼360 nm for promising application of UV-triggered light-emitting diodes. Most interestingly, this Eu-CQD solution with UV-excited colorful light emissions was utilized as a visual pH sensor, experimentally featuring good repeatabilities and stabilities (pH values switching between 7 and 14). Besides, the Eu-CQD solution exhibited highly sensitive detection characteristics for Co (II) ions. Due to the spectral overlapping, the quenching efficiency is as high as 96.3%, and the detection limit is obtained as 0.139 μM, which is much lower than the maximum concentration of Co (II) in the drinking water permitted by WHO. All of these relevant results significantly demonstrate the great potential of lanthanide doping in the multifunctionalization of CQDs for certain emerging applications like lighting, sensing, and detection.

Target-Triggered Ultrafast Chondroitin Gelation Enabled Power-Free and Point-of-Care Bioassays.

Point-of-care (POC) tests increasingly highlight the importance of portable, cost-effective, and visually quantitative detection of biomarkers. Herein, we developed a power-free and visual signal-readout POC sensor based on the target-triggered ultrafast gelation process. In the gelation process, the target triggered the cascade reaction catalyzed by oxidase and ferrous glycinate to produce carbon radicals that immediately initiated the rapid polymerization and cross-linking of acryloylated chondroitin sulfate and dimethylacrylamide. This highly efficient enzymatic polymerization process contributed to the ultrafast generation of chondroitin hydrogel within 1 min at 25 °C. The increase in viscosity of aqueous solution resulted from hydrogel formation was then visually measured according to the distance of solution migration on a tick-labeled pH test strip, which thus realized the quantification of a target. By utilizing glucose oxidase as an oxidase model during the gelation process, this POC sensor was successfully employed in the rapid quantitative detection of glucose without the need for any auxiliary instruments. Benefiting from the specificity and stability of the enzymatic polymerization reaction, the sensor exhibited excellent performance in the detection of glucose in clinical blood samples. Moreover, the sensor was further extended to uric acid detection and enabled accurate assay in clinical urine samples, which indicated the versatility and practicability of this sensor in the POC test.

Asynchronous Visual-Inertial Odometry for Event Cameras

Abstract. Event cameras are bio-inspired visual sensors that output changes in pixel-level brightness asynchronously instead of standard intensity frames at a fixed rate. These cameras offer reliable visual information in high-speed motion and high dynamic range (HDR) scenes, addressing the limitations of traditional cameras in such scenarios. Therefore, research of integrating event cameras into established visual algorithms holds significant value. In this study, based on traditional Visual-Inertial Odometry (VIO) frameworks, we proposed an innovative asynchronous monocular event-based inertial odometry method to fully exploit the benefits of event cameras. First, the corner features are extracted separately from the raw event stream and the time surface map, followed by uniform feature selection to accurately describe three-dimensional spatial geometry. Then, feature tracking is achieved by integrating these two event representation methods. In addition, our method obtains stable and high frequency state estimation by fusing event and IMU measurements through graph optimization. We validate the effectiveness of our proposed approach, comparing with several state-of-the-art EVIO systems and VIO systems.

A visual electrochemiluminescence sensor based on the combination of distance-readout strategy and nanozymes catalyzing precipitation

BackgroundNanozyme is a kind of biomimetic enzyme with unique properties and catalytic function of nanomaterials. Various nanozymes have been applied in the development of visual biosensors because of its excellent stability and availability. Nevertheless, the current applications of nanozymes primarily focus on colorimetric sensing based on color changes, which is susceptible to external factors interference including sample solution color and ambient light. Incorporating a distance-readout strategy with nanozymes presents a promising avenue for the advancement of innovative visual biosensing technologies in the future. ResultsA visual electrochemiluminescence (ECL) sensor was developed by combining the distance-readout strategy with nanozymes catalyzing precipitation for the first time. The detection is based on the ECL length change induced by nanozymes. This sensor chip comprises a square detection area and a visual ECL channel area. A peroxidase-mimic, core-shell hollow Fe3O4@PDA@ZIF-67, was modified in the detection region. The substrate 4-chloro-1-naphthol was rapidly catalyzed by nanozymes to form purple precipitate benzo-4-chlorohexanedione, leading to an increase in impedance within the detection region. Consequently, the detection area's partial voltage rose while the visual channel's dropped, resulting in a shorter ECL length. As a proof-of-concept experiment, the visual distance-readout ECL sensor was utilized for the quantitative detection of H2O2. The alteration in ECL length exhibited a good linear relationship with H2O2 concentration (logarithm) within the range of 300 nM to 300 μM, with the limit of detection (LOD) of 87 nM. SignificanceThis work developed a novel approach for the application of nanozymes in analytical chemistry. Compared with common colorimetric sensing, this method is more convenient and intuitive, which can effectively eliminate potential interference from humans and the environment during signal acquisition. Because biocatalytic precipitation can be formed through multiple sensing processes, this strategy holds great promise as a portable and cost-effective tool with diverse applications.

A high-precision visual detection method for rail profile based on Scheimpflug condition

Rail profile is an important indicator of rail service status, which should be always kept in good condition. How to efficiently collect rail profiles with high precision is a critical issue for their condition-based maintenance. This paper focuses on the method for accurate visual detection of rail full profile (RFP), including optical modeling, visual calibration, and error compensation. The proposed optical unit, featuring double line structured-light sensors, introduces an innovative method under the Scheimpflug condition to avoid the limitation of depth of field in RFP measurement. Subsequently, double visual sensors are calibrated together using the common reference target to extract the internal and external parameters respectively. The dynamic errors resulting from the impact of vehicle vibrations are ultimately investigated to rectify the distorted profiles to the normal. Both laboratory and field tests are performed to verify the effectiveness of the proposed system in terms of precision and efficiency.

Target detection and obstacle avoidance for UAV based on dynamic vision sensor

Aiming at the problem of UAV sensing dynamic targets and avoiding high-speed dynamic obstacles in dynamic environments, a target detection and obstacle avoidance algorithm based on dynamic vision sensor is designed. A filtering method and motion compensation algorithm are designed to filter out background noise and hot spot noise in the event stream as well as redundant events caused by the camera's own motion. A dynamic target fusion detection algorithm that fuses event images and RGB images is designed to ensure the reliability of detection. The target motion trajectory is estimated based on the detection results, and the speed obstacle method is improved to avoid dynamic obstacles by combining the obstacle motion characteristics and UAV dynamic constraints. A large number of simulation tests, handheld tests and flight tests verify the feasibility of the proposed algorithm.

Ultrasensitive dim-light neuromorphic vision sensing via momentum-conserved reconfigurable van der Waals heterostructure

Reconfigurable phototransistors featuring bipolar photoresponses are favorable for manipulating high-performance neuromorphic vision sensory. Here, we present a momentum-conserved reconfigurable phototransistor based on the van der Waals heterojunction between methylammonium lead iodide perovskite and two-dimensional Bi2O2Se semiconductor, which exhibits a synergistic interplay of interband hot-carrier transitions and reconfigurable heterointerface band alignments, eventually achieving the ultrahigh bipolar optoelectronic performances with the photoresponsivity of 6×107 AW−1, accompanied by the specific detectivity of 5.2×1011 Jones, and the dynamic range of 110 dB. Moreover, A 3×3 heterotransistor array is fabricated to perform in-sensor analog multiply-accumulate operations even under the challenging dim illumination of 0.1 μWcm−2 that comparable to natural moonlight. The reconfigurable heterotransistor array can be further adopted to enhance the traffic-light detection under dim-light conditions. Our advancement in momentum-conserved reconfigurable heterotransistor signifies a leap forward in real-time, energy-efficient, and low-light image processing for neuromorphic vision sensors.

Strong Robust LIO System Based on Event Camera Assistance

Abstract. At present, the LiDAR system can achieve high precision and is extensively used for indoor and outdoor mobile positioning and mapping. However, LiDAR systems still face issues in cluttered environments where strong features are absent, leading to a degradation of the LiDAR-based solution. When the carrier movement involves high-speed or prolonged exposure to the mirror wall, it can cause severe degradation issues or even positioning failures in the laser slam system. Event cameras are vision sensors inspired by biology that exhibit strong robustness in high dynamic and low texture environments, potentially leading to better performance in such environments. However, some issues with event cameras remain unresolved. In this paper, a multi-source fusion method based on EVIO, LIO and IMU trajectory layer post-processing method is proposed, which will fully consider the robustness of event camera in high dynamic environment and the high precision advantage of Lidar in conventional environment, and use an algorithm based on normalized uncertainty. The elastic multi-source fusion of event camera and LiDAR is realized and tested in realde environment. Experimental results show that the proposed algorithm can effectively improve the accuracy of event camera and LiDAR. Compared to the current more advanced algorithms, the proposed algorithm effectively addresses the problem of LIO in degraded environments. Additionally, it mitigates the scale inaccuracy and divergence of the EVIO trajectory to some extent. Compared to LIO, the algorithm can reduce the maximum position error by approximately 30% and increase the overall position accuracy by 32%. Additionally, it can significantly constrain the divergence of errors in the Y direction, improving its accuracy by about 75% and 65% compared to the LIO and EVIO algorithms, respectively.

Design of Connector Assembly Equipment for the Automotive Industry

The automotive industry is one of the most demanding sectors of all manufacturing industries due to its competitiveness. It is necessary to innovate through the implementation of automated and robotic equipment, leading to cycle time and labor cost reduction. This work aims to design semi-automatic equipment to assemble cabling connectors used in the automotive sector, replacing a manual process currently taking place in an automotive components company. In the proposed equipment, the operator places a connector in the equipment, and the components (pins and seals) are automatically inserted. A vision sensor with artificial intelligence then confirms the correct application. The equipment operation defined as Finite Element Method (FEM) was applied for structural verification; the materials and fabrication processes were detailed; the associated costs were calculated, and the equipment subsets were validated. The design was successfully accomplished, and the imposed requirements were fulfilled, with significant advantages over the current process, providing new knowledge on how semi-automatic systems can be deployed to enhance the productivity and quality of manufacturing processes. The design principles and insights gained from this work can be applied to other automation challenges, particularly where manual processes need to be replaced by more efficient semi-automatic or automatic systems. The modularity of the overall solution and the design concepts of the component inserter, component feeder, and assembly process allow for its use in different assembly scenarios beyond the automotive sector, such as electronics or aerospace, providing a contribution to increased competitiveness and survival in the global market.