Vision Sensor Research Articles

Bimodal Visual Sensors Based on Mechanoluminescence and Biosensing for Artificial Intelligence‐Assisted Orthodontics

AbstractFlexible visual sensors hold great potential for application in wearable electronics and health monitoring. However, the separate acquisition of various visual signals without mutual interference and their functional integration in internal biomedical devices remains a significant challenge. Here, a functional material that integrates mechanoluminescent units with biosensing units through supramolecular interactions, enabling bimodal non‐interfering visual detection of orthodontic force and bacterial infection is presented. The orthodontic force induces fluorescence by facilitating the release of electrons from trapped energy levels, caused by the disruption or rearrangement of defect structures within strontium aluminate crystals. Meanwhile, the seamlessly integrated biosensing component can effectively detect lactic acid in a dose‐dependent manner, underscoring its multifunctional capabilities in biosensing applications. Moreover, the development of a cloud‐based deep learning server can achieve a 97.7% accuracy in the precise decoupling of visual signals, facilitating remote monitoring and enabling early intervention during orthodontic treatment. The proposed artificial intelligence‐enhanced bimodal visual sensors represent a new paradigm for orthodontic monitoring, suitable for a wide range of biomedical applications.

Asymmetric and Ultrasensitive Structural Color Response in Nanochain‐Embedded Photonic Composites

AbstractMechanochromic photonic crystals, capable of altering structural colors in response to external forces, are emerging intelligent materials in various fields, such as visual sensors and anti‐counterfeiting technologies. However, conventional mechanical responses of photonic crystals mainly rely on stretching or compression, resulting in symmetric and limited mechanochromic effects due to restricted changes in lattice spacing, particularly limiting the recognition of the direction of force. Here, asymmetric and ultrasensitive structural color responses are reported in colloidal photonic composites featuring embedded one‐dimensional (1D) photonic nanochains of colloidal particles based on shear‐induced lattice orientation change. It is shown that shearing deformation tilts the photonic nanochains within the polymer matrix, leading to a remarkable and asymmetric color shift depending on shear force and observing directions. The asymmetric response encourages to develop ultrasensitive twist sensors capable of detecting subtle twisting movements as low as 0.2° and twist direction. Furthermore, by integrating magnetic and deformation orientation control, dynamic optical anti‐counterfeiting labels capable of displaying highly intricate patterns are devised. This unique shear‐induced color‐shifting mechanism expands the responsiveness of mechanochromic materials, with broad implications in many other areas, including structural health monitoring, soft robotics, and artificial skin.

Visual Piezoresistive Dual‐Response Sensor Based on CaZnOS: Mn Mechanoluminescence Materials

AbstractThe ability to achieve real‐time movement visualization in piezoresistive sensors remains a challenge. A visual piezoresistive sensor to perceive the intensity of mechanical stimuli and visible spatial location information is designed based on the mechanoluminescence material CaZnOS: Mn. The linearity, detection range, sensitivity, and stability of the sensor are tested, and the sensing mechanism of the sensor is discussed, and the relationship between force, resistance, and light intensity is established. A 5 × 5 sensor array is prepared to realize the visual detection of dynamic force trajectory, and combined with a convolutional neural network and random forest algorithm, the human writing number and pressure characteristics are recognized, and the writing path and handwriting of the robot arm are controlled by multi‐feature input. The experimental results show that the machine learning algorithm is very reliable with an accuracy of 98.33% for digit recognition and 97.21% for identity recognition. The visual piezoresistive pressure sensor provides a new idea for the visualization of flexible pressure and promotes the development of flexible sensors towards integration and intelligence.

Self-cure Dual-branch Network for Facial Expression Recognition Based on Visual Sensors

Self-cure Dual-branch Network for Facial Expression Recognition Based on Visual Sensors

DC characteristics of a dynamic vision sensor's photoreceptor circuit evaluated for event-based sensing in the mid-wave infrared

Forthcoming infrared event-based sensors will have utility in the space-based surveillance domain where they could potentially perform traditional sensing and tracking functions with significantly enhanced temporal resolution and reduced downstream datalink demands and power consumption. As a first step toward extending event-based sensing technology into the mid-wave infrared, a DC simulation of the conventional event-based sensor unit cell's photoreceptor circuit is performed, and the results are compared with measurements of a printed circuit board implementation of the same circuit to assess what design freedom is available to interface the photoreceptor with a mid-wave infrared photodetector. Detailed analysis of the circuit and measurements provides insight into which fundamental properties of the transistors drive the photoreceptor's dynamic range and demonstrates several characteristics that are relevant to mid-wave infrared sensing. These characteristics include the capability to produce a stable, low voltage bias on the infrared photodetector to maximize sensitivity, and that operating the circuit below room temperature increases the photoreceptor's dynamic range. Measurements show that even the simplest implementation of the photoreceptor circuit exhibits a dynamic range of logarithmic compression of 150 dB at room temperature. However, the dynamic range is ultimately limited by the mid-wave infrared photodetector's dark current activation energy, which is significantly lower than the threshold voltage (energy) of the photoreceptor's feedback transistor and, thus, there is an incentive for the photodetector's dark current to be diffusion-limited.

Fourier‐Based Action Recognition for Wildlife Behavior Quantification with Event Cameras

Event cameras are novel bioinspired vision sensors that measure pixel‐wise brightness changes asynchronously instead of images at a given frame rate. They offer promising advantages, namely, a high dynamic range, low latency, and minimal motion blur. Modern computer vision algorithms often rely on artificial neural network approaches, which require image‐like representations of the data and cannot fully exploit the characteristics of event data. Herein, approaches to action recognition based on the Fourier transform are proposed. The approaches are intended to recognize oscillating motion patterns commonly present in nature. In particular, the approaches are applied to a recent dataset of breeding penguins annotated for “ecstatic display,” a behavior where the observed penguins flap their wings at a certain frequency. It is found that the approaches are both simple and effective, producing slightly lower results than a deep neural network (DNN) while relying just on a tiny fraction of the parameters compared to the DNN (five orders of magnitude fewer parameters). They work well despite the uncontrolled, diverse data present in the dataset. It is hoped that this work opens a new perspective on event‐based processing and action recognition.

Visual and Radar Data Fusion: Complementary or Alternative Sensors in Intelligent Driving Systems

This article explores the role and advantages and disadvantages of visual and radar data fusion in intelligent driving systems. With the rapid development of intelligent driving technology, the comprehensive perception of the environment by car perception systems has become crucial, usually requiring the combination of low, medium, and high-precision sensors. Visual sensors perform excellently in acquiring high-resolution images and recognizing detailed information, while radar sensors have higher stability and penetration in harsh weather conditions. Due to the limitations of a single sensor in complex driving environments, data fusion has become a key strategy for improving the performance of perception systems. By comparing different perception schemes and application scenarios, this article analyzes the actual effects and potential advantages of visual and radar data fusion, revealing its important role in intelligent driving technology. Fusion technology can compensate for the shortcomings of various sensors, and improve the accuracy and reliability of environmental perception, but still faces challenges in data synchronization, algorithm selection, and real-time performance. This article aims to provide valuable insights for the development of future intelligent driving technology.

Cellulose-based halochromic sensor for real-time surveillance of spoilage of packed fish

The development of a biopolymer-based color-changing visual sensor for monitoring the spoilage of fishery products is described in this study. At first, a cellulose-based composite film containing mixed-indicator dyes was fabricated. The film was characterized using UV–visible and FT-IR spectroscopy, scanning electron microscopy, thermogravimetric analysis, lab, and RGB measurements. The halochromic properties of the film were evaluated through ammonia test and colorimetry. The developed film sensor is tested for monitoring the spoilage of packed seer fish fillets at room temperature for up to 48 h and at 4 °C for up to 10 days. The sensor displayed distinct color changes correlating with the spoilage of the packed fish. Analysis of total volatile basic nitrogen and pH levels of the samples revealed a correlation between the halochromism of the sensor and the spoilage of the fish. The color-changing pattern of the sensor was consistent with the results of microbial analysis. The present study reveals a promising material combination for fabricating simple visual sensors from biomaterials, which can be used to monitor the real-time deterioration of packaged fishery products.

Computer vision-based substructure isolation method for localized damage identification

In recent years, vision-based structural damage identification techniques have garnered significant attention due to their simplicity and cost-effectiveness. Nevertheless, dynamic response-based vision methods face challenges related to the limitations of the field of view (FOV), i.e., the extent of the FOV is often sacrificed in order to ensure sufficient monitoring accuracy, which reduces the amount of data available for structural health monitoring (SHM). This study presents a solution to this problem by employing the substructure isolation method (SIM), which focuses on local vibrations of the structure and enables local damage identification by isolating the area of interest. Additionally, a method combining kernelized correlation filter (KCF) with sub-pixel template matching is used to extract vibration information recorded by visual sensor. This strategy balances both the efficiency and accuracy of displacement extraction. In numerical simulations, the performance of the SIM based on acceleration response and displacement response is compared, demonstrating the potential compatibility of visual sensors with the SIM. Finally, the effectiveness of the proposed vision-based local damage identification method is validated through vibration testing of a shear frame model.

Identifying the Laser Stripes via Ray Model for Multiline Structured Light Stereo Vision Sensors

Identifying the Laser Stripes via Ray Model for Multiline Structured Light Stereo Vision Sensors

Analysis of modeling nasal narrow space based on visual slam

Abstract. Now, surgeries are becoming more stable, safe, efficient and low-cost. During the surgical treatment of nasal diseases, surgical robotic robotics can help operate accurately and reduce the discomfort after surgery. However, due to the internal space of the nasal cavity being relatively narrow, it is difficult for the nasal surgical robot to contain multiple vision sensors and the monocular camera could not get information about the depth of the 3D objects in the scene, so the existing surgical robots cannot accomplish the three-dimensional modeling about the internal space of nasal cavity well. In practice, doctors still have to analyze pictures from the robotic, which may decrease the efficiency of the surgery and increase the risk to patients. This article designed a SLAM algorithm framework based on a depth estimation network, it can simulate the internal structure of the nasal cavity more accurately through pictures, which come from monocular endoscopic on surgical robotic. The insights gained in this study verify that the method of image segmentation can also make the depth representation of the nasal internal space more accurate and this method may help robots realize their self-position in the narrow area of the nasal cavity, which lays the foundations for the development of fully autonomous surgical robots.

Design of Single-Camera Mirrored Binocular Vision Sensor With Single-Plane Mirror

Design of Single-Camera Mirrored Binocular Vision Sensor With Single-Plane Mirror

Spiking Neural Networks for Real-Time Pedestrian Street-Crossing Detection Using Dynamic Vision Sensors in Simulated Adverse Weather Conditions

This study explores the integration of Spiking Neural Networks (SNNs) with Dynamic Vision Sensors (DVSs) to enhance pedestrian street-crossing detection in adverse weather conditions—a critical challenge for autonomous vehicle systems. Utilizing the high temporal resolution and low latency of DVSs, which excel in dynamic, low-light, and high-contrast environments, this research evaluates the effectiveness of SNNs compared to traditional Convolutional Neural Networks (CNNs). The experimental setup involved a custom dataset from the CARLA simulator, designed to mimic real-world variability, including rain, fog, and varying lighting conditions. Additionally, the JAAD dataset was adopted to allow for evaluations using real-world data. The SNN models were optimized using Temporally Effective Batch Normalization (TEBN) and benchmarked against well-established deep learning models, concerning their accuracy, computational efficiency, and energy efficiency in complex weather conditions. This study also conducted a comprehensive analysis of energy consumption, highlighting the significant reduction in energy usage achieved by SNNs when processing DVS data. The results indicate that SNNs, when integrated with DVSs, not only reduce computational overhead but also dramatically lower energy consumption, making them a highly efficient choice for real-time applications in autonomous vehicles (AVs).

The influence of neighbor selection on self-organized UAV swarm based on finite perception vision.

Recently, vision-based unmanned aerial vehicle (UAV) swarming has emerged as a promising alternative that can overcome the adaptability and scalability limitations of distributed and communication-based UAV swarm systems. While most vision-based control algorithms are predicated on the detection of neighboring objects, they often overlook key perceptual factors such as visual occlusion and the impact of visual sensor limitations on swarm performance. To address the interaction problem of neighbor selection at the core of self-organizing UAV swarm control, a perceptually realistic finite perception visual (FPV) neighbor selection model is proposed, which is based on the lateral visual characteristics of birds, incorporates adjustable lateral visual field widths and orientations, and is able to ignore occluded agents. Based on the FPV model, a neighbor selection method based on the Acute Angle Test (AAT) is proposed,
which overcomes the limitation that the traditional neighbor selection mechanism can only interact with the nearest neighboring agents. A large number of Monte Carlo simulation comparison experiments show that the proposed FPV+AAT neighborselection mechanism can reduce the redundant communication burden between large scale self-organized UAV swarms, and outperforms the traditional neighbor selection method in terms of order, safety, union, connectivity, and noise resistance.

High-performance surface defect detection of aluminum substrate based on event camera

Abstract Traditional industrial surface defect detection often employs CCD/CMOS cameras, but they are difficult to detect the minute defects on aluminum substrates in highly dynamic industrial scenes due to their nature. Event camera is a novel high-resolution vision sensor which measures per-pixel brightness changes in an asynchronous manner and outputs as event information flow (EIF). Small and weak defects on aluminum substrate can be captured by event camera effectively, but the EIF contain a large amount of noise, making it difficult to perform accurate and high-precision defect detection. To address this problem, we propose a frame aggregation method to realize good event information flow processing (EIFP), and then use an improved circle detection method to locate the aluminum substrate in each frame, removing abundant event information outside the aluminum substrate. Subsequently, we enhance the event signals under different frames based on optical flow tracking using multiple features, and construct a semi-supervised detector based on pseudo-labels to achieve high-precision defect localization. Finally, considering the small inter-class differences in defects on the surface of aluminum substrates, we construct a defect class corrector based on ensemble learning to enhance the ability to determine defect classes, achieving high-precision automatic quality inspection of aluminum substrate surfaces. The performance of our method is compared with other advanced methods based on event camera data of aluminum substrates in real industrial scenarios. The experimental results show that our method has improved the detection accuracy by about 10% and the classification accuracy by about 25% compared to the original state-of-the-art methods.

A New Deep Learning Methodology for Alarm Supervision in Marine Power Stations.

Marine engineering officers operate and maintain the ship's machinery during normal navigation. Most accidents on board are related to human factors which, at the same time, are associated with the workload of the crew members and the working environment. The number of alarms is so high that, most of the time, instead of helping to prevent accidents, it causes more stress for crew members, which can result in accidents. Convolutional Neural Networks (CNNs) are being employed in the recognition of images, which depends on the quality of the images, the image recognition algorithm, and the very complex configuration of the neural network. This research study aims to develop a user-friendly image recognition tool that may act as a visual sensor of alarms adjusted to the particular needs of the ship operator. To achieve this, a marine engineering simulator was employed to develop an image recognition tool that advises marine engineering officers when they are conducting their maintenance activities, with the aim to reduce their stress as a work risk prevention tool. Results showed adequate accuracy for three-layer Convolutional Neural Networks and balanced data, and the use of external cameras stands out for user-friendly applications.

Bidirectional Photovoltage-Driven Oxide Transistors for Neuromorphic Visual Sensors.

Biological vision is one of the most important parts of the human perception system. However, emulating biological visuals is challenging because it requires complementary photoexcitation and photoinhibition. Here, the study presents a bidirectional photovoltage-driven neuromorphic visual sensor (BPNVS) that is constructed by monolithically integrating two perovskite solar cells (PSCs) with dual-gate ion-gel-gated oxide transistors. PSCs act as photoreceptors, converting external visual stimuli into electrical signals, whereas oxide transistors generate neuromorphic signal outputs that can be adjusted to produce positive and negative photoresponses. This device mimics the human vision system's ability to recognize colored and color-mixed patterns. The device achieves a static color recognition accuracy of 96% by utilizing the reservoir computing system for feature extraction. The BPNVS mem-reservoir chip is also proposed for handing object movement and dynamic color recognition. This work is a significant step forward in neuromorphic sensing and complex pattern recognition.

Study of Contactless Computer Vision-Based Road Condition Estimation Methods Within the Framework of an Operational Design Domain Monitoring System

Deployment of Level 3 and Level 4 autonomous vehicles (AVs) in urban environments is significantly constrained by adverse weather conditions, limiting their operation to clear weather due to safety concerns. Ensuring that AVs remain within their designated Operational Design Domain (ODD) is a formidable challenge, making boundary monitoring strategies essential for safe navigation. This study explores the critical role of an ODD monitoring system (OMS) in addressing these challenges. It reviews various methodologies for designing an OMS and presents a comprehensive visualization framework incorporating trigger points for ODD exits. These trigger points serve as essential references for effective OMS design. The study also delves into a specific use case concerning ODD exits: the reduction in road friction due to adverse weather conditions. It emphasizes the importance of contactless computer vision-based methods for road condition estimation (RCE), particularly using vision sensors such as cameras. The study details a timeline of methods involving classical machine learning and deep learning feature extraction techniques, identifying contemporary challenges such as class imbalance, lack of comprehensive datasets, annotation methods, and the scarcity of generalization techniques. Furthermore, it provides a factual comparison of two state-of-the-art RCE datasets. In essence, the study aims to address and explore ODD exits due to weather-induced road conditions, decoding the practical solutions and directions for future research in the realm of AVs.

Lanthanide Functionalized Carbon Quantum Dots for White Light Emission, pH Sensing, and Co (II) Detection.

Carbon quantum dots (CQDs) with fluorescence emission have been widely studied for versatile applications, but facile tunability of the spectral properties of CQDs by doping remains to be further explored. Herein, employing lanthanide ion Eu3+ as a dopant and activator, a simple and efficient synthesis route for pure CQDs and Eu-CQDs was demonstrated using N, N-dimethylformamide, oleic acid, and oleylamine as precursors for carbon sources. In comparison, with the popular citric acid precursor, the as-prepared CQDs and Eu-CQDs exhibited an obviously smaller particle size (1.72 ± 0.29 nm) and a more uniform distribution. Through systematic optimization of Eu3+ doping for the Eu-CQD system, colorful light emissions in the visible range were first realized under excitation of ultraviolet (UV) ∼360 nm for promising application of UV-triggered light-emitting diodes. Most interestingly, this Eu-CQD solution with UV-excited colorful light emissions was utilized as a visual pH sensor, experimentally featuring good repeatabilities and stabilities (pH values switching between 7 and 14). Besides, the Eu-CQD solution exhibited highly sensitive detection characteristics for Co (II) ions. Due to the spectral overlapping, the quenching efficiency is as high as 96.3%, and the detection limit is obtained as 0.139 μM, which is much lower than the maximum concentration of Co (II) in the drinking water permitted by WHO. All of these relevant results significantly demonstrate the great potential of lanthanide doping in the multifunctionalization of CQDs for certain emerging applications like lighting, sensing, and detection.

Target-Triggered Ultrafast Chondroitin Gelation Enabled Power-Free and Point-of-Care Bioassays.

Point-of-care (POC) tests increasingly highlight the importance of portable, cost-effective, and visually quantitative detection of biomarkers. Herein, we developed a power-free and visual signal-readout POC sensor based on the target-triggered ultrafast gelation process. In the gelation process, the target triggered the cascade reaction catalyzed by oxidase and ferrous glycinate to produce carbon radicals that immediately initiated the rapid polymerization and cross-linking of acryloylated chondroitin sulfate and dimethylacrylamide. This highly efficient enzymatic polymerization process contributed to the ultrafast generation of chondroitin hydrogel within 1 min at 25 °C. The increase in viscosity of aqueous solution resulted from hydrogel formation was then visually measured according to the distance of solution migration on a tick-labeled pH test strip, which thus realized the quantification of a target. By utilizing glucose oxidase as an oxidase model during the gelation process, this POC sensor was successfully employed in the rapid quantitative detection of glucose without the need for any auxiliary instruments. Benefiting from the specificity and stability of the enzymatic polymerization reaction, the sensor exhibited excellent performance in the detection of glucose in clinical blood samples. Moreover, the sensor was further extended to uric acid detection and enabled accurate assay in clinical urine samples, which indicated the versatility and practicability of this sensor in the POC test.