Advanced Vision Systems Research Articles

Versatile Graph Neural Networks Toward Intuitive Human Activity Understanding.

Benefiting from the advanced human visual system, humans naturally classify activities and predict motions in a short time. However, most existing computer vision studies consider those two tasks separately, resulting in an insufficient understanding of human actions. Moreover, the effects of view variations remain challenging for most existing skeleton-based methods, and the existing graph operators cannot fully explore multiscale relationship. In this article, a versatile graph-based model (Vers-GNN) is proposed to deal with those two tasks simultaneously. First, a skeleton representation self-regulated scheme is proposed. It is among the first trials that successfully integrate the idea of view adaptation into a graph-based human activity analysis system. Next, several novel graph operators are proposed to model the positional relationships and learn the abstract dynamics between different human joints and parts. Finally, a practical multitask learning framework and a multiobjective self-supervised learning scheme are proposed to promote both the tasks. The comparative experimental results show that Vers-GNN outperforms the recent state-of-the-art methods for both the tasks, with the to date highest recognition accuracies on the datasets of NTU RGB + D (CV: 97.2%), UWA3D (88.7%), and CMU (1000 ms: 1.13).

VO2/MoO3 Heterojunctions Artificial Optoelectronic Synapse Devices for Near-Infrared Optical Communication.

Artificial optoelectronic synapses (OES) have attracted extensive attention in brain-inspired information processing and neuromorphic computing. However, OES at near-infrared wavelengths have rarely been reported, seriously limiting the application in modern optical communication. Herein, high-performance near-infrared OES devices based on VO2/MoO3 heterojunctions are presented. The textured MoO3 films are deposited on the sputtered VO2 film by using the glancing-angle deposition technique to form a heterojunction device. Through tuning the oxygen defects in the VO2 film, the fabricated VO2/MoO3 heterojunction exhibits versatile electrical synaptic functions. Benefiting from the highly efficient light harvesting and the unique interface effect, the photonic synaptic characteristics, mainly including the short/long-term plasticity and learning experience behavior are successfully realized at the O (1342nm) and C (1550nm) optical communication wavebands. Moreover, a single OES device can output messages accurately by converting light signals of the Morse code to distinct synaptic currents. More importantly, a 3×3 artificial OES array is constructed to demonstrate the impressive image perceiving and learning capabilities. This work not only indicates the feasibility of defect and interface engineering in modulating the synaptic plasticity of OES devices, but also provides effective strategies to develop advanced artificial neuromorphic visual systems for next-generation optical communication systems.

Multifunctional optoelectronic memristor based on CeO2/MoS2 heterojunction for advanced artificial synapses and bionic visual system with nociceptive sensing

The emerging intelligent nanodevices make the artificial visual system a reality. Memristors, as artificial synapses equipped with the advantages of high parallelism and low power consumption, hold considerable promise for developing biological visual systems. In this study, an optoelectronic memristor composed of Ag/CeO2/MoS2/ITO is proposed, which actualizes neural synapse functions such as short/long-term plasticity by adjusting electrical excitation, and short/long-term learning and memory conversion is realized with the assistance of a letter “NANO” pattern of 50 devices. Moreover, light is capable of replacing voltage to induce advanced synaptic functions including short/long-term memory and learning-forgetting-relearning. In particular, the device benefits from the ability to produce significantly distinguishable photoconductive effects for multi-wavelengths (620, 580, and 520 nm) and exhibits memory properties for different wavelengths. A human visual system that is able to sense, memory, and color recognition of the “traffic signal” image is implemented on a 7 × 7 optoelectronic memristor array constructed with multi-wavelength modulation. Finally, an artificial visual system with pain-sensing capability is presented, where the four key features of human visual nociceptors, namely “threshold”, “no-adaption”, “relaxation” and “nociceptive sensitization”. This work provides a strategy to integrate sensing, memory, and visual pain devices for electronic eyes and humanoid robots in the future.

Multicolor vision perception of flexible optoelectronic synapse with high sensitivity for skin sunburn warning.

The development of flexible synaptic devices with multicolor signal response is important to exploit advanced artificial visual perception systems. The Sn vacancy-dominant memory and narrow gap characteristics of PEA2SnI4 make it suitable as a functional layer in ultraviolet-visible (UV-Vis) light-stimulated synaptic devices. However, such device tends to have high dark current and poor sensitivity, which is not conducive to subsequent information processing. Here, we proposed a self-powered flexible optoelectronic synapse based on PEA2SnI4 films. By introducing the electron transport layer (ETL), the dark current of the device is decreased by 5 orders of magnitude as compared to the Au/PEA2SnI4/ITO device, and the sensitivity is increased from 10.3% to 99.2% at 1.25 mW cm-2 light illumination (520 nm), indicating the vital role of the introduced ETL in promoting the separation of excitons in the interface and inhibiting the free carrier transfer. On this basis, the optoelectronic synaptic functions with integrated sensing, recognition, and memory features were realized. The array device exhibits UV-Vis light sensitivity and tunable synaptic plasticity, enabling its application for multicolor visual sensing and skin sunburn warning. This work provides an effective strategy for fabricating multicolor intelligent sensors and artificial vision systems, which facilitate the practical application of artificial optoelectronic synapses.

Evaluation of the common pauraque (Nyctidromus albicollis) cornea using light and scanning electron microscopy.

The common pauraque Nyctidromus albicollis (Gmelin, 1789) is a widespread avian species; however due to its nocturnal habits and reclusive behaviour, little is known about their vision and ecology. Most avian species are visually dependent with advanced visual systems providing high spatial resolution, on the species needs. Each ocular structure has a specific role in contributing towards high visual function, and the cornea is the first refractive structure in the visual process. However, the common pauraque cornea had not been described until the present data. Therefore, this study aims to describe the morphology and morphometry of the common pauraque cornea by means of light and scanning electron microscopy to evaluate the cross-sectional anatomy as well as the ultrastructure of the endothelial cells. Histological similarities with the cornea of other birds have been observed, but the thickness of the common pauraque cornea is much smaller than the other described corneas. A better understanding of the common pauraque cornea can help us better explain the physiology of vision and the visual requirements of this species. In turn, this will help us better understand how this species successfully interacts with its environment, and will improve our knowledge on how to interpret pathological changes in their cornea in a clinical setting.

Tetrachromatic vision-inspired neuromorphic sensors with ultraweak ultraviolet detection

Sensing and recognizing invisible ultraviolet (UV) light is vital for exploiting advanced artificial visual perception system. However, due to the uncertainty of the natural environment, the UV signal is very hard to be detected and perceived. Here, inspired by the tetrachromatic visual system, we report a controllable UV-ultrasensitive neuromorphic vision sensor (NeuVS) that uses organic phototransistors (OPTs) as the working unit to integrate sensing, memory and processing functions. Benefiting from asymmetric molecular structure and unique UV absorption of the active layer, the as fabricated UV-ultrasensitive NeuVS can detect 370 nm UV-light with the illumination intensity as low as 31 nW cm−2, exhibiting one of the best optical figures of merit in UV-sensitive neuromorphic vision sensors. Furthermore, the NeuVS array exbibits good image sensing and memorization capability due to its ultrasensitive optical detection and large density of charge trapping states. In addition, the wavelength-selective response and multi-level optical memory properties are utilized to construct an artificial neural network for extract and identify the invisible UV information. The NeuVS array can perform static and dynamic image recognition from the original color image by filtering red, green and blue noise, and significantly improve the recognition accuracy from 46 to 90%.

Semantic closed-loop based visual mapping algorithm for automated valet parking

Simultaneous localization and mapping (SLAM) is one of the core technologies to realize automated valet parking (AVP). Currently, advanced visual feature-based SLAM systems suffer from feature extraction failure and tracking loss due to the constraints of textureless scenes, unclear illumination, and dynamic conditions. To address these problems, this paper proposes a visual SLAM algorithm based on a semantic closed-loop detection algorithm using surround-view cameras and inertial measurement units (IMU) as sensors. The algorithm combines semantic features and the idea of inverse index to improve the traditional keyframes selection methods and the closed-loop detection algorithms, effectively avoiding the tedious and complicated feature point matching and improving the computational efficiency of the computer. Experiments show that the algorithm in this paper achieves better results in terms of precision and recall, absolute trajectory error (ATE), and relative pose error (RPE), and can meet the demand for SLAM and subsequent navigation in indoor parking lots.

Enhanced broadband Si-based optoelectronic synapse for artificial visual applications

Broadband sensibility and efficient image recognition are crucial for perception and memory storage in remote sensing, reconnaissance and environmental monitoring inspired by the human cognitive system. Although the amorphous silicon (a-Si) thin film is compatible to the complementary-metal–oxidesemiconductor (CMOS) technology, implementing in-sensor broadband recognition and memory based on intrinsic a-Si is still a challenge due to the band gap restrictions and the limited spectrum sensing ability. Herein, through doping of Ru (Ruthenium), optoelectronic devices based on the band engineered a-Si:Ru films have been fabricated with a silicon-oxide heterostructure, demonstrating comprehensive broadband image recognition, memory and convolutional processing simultaneously. Upon regulating the trapping and releasing of the photogenerated electron-hole pairs, the devices could mimic various synaptic functions such as excitatory postsynaptic current (EPSC), paired-pulse facilitation (PPF), transition from short- to long-term potentiation, lower switching energy and adjustable photocurrent analog states. Moreover, the in-sensor broadband convolutional processing has enhanced the high recognition accuracy for multi-band visuals, indicating a significant opportunity for future neuromorphic sensory applications and advanced robot visual systems.

Effect of shape on the physical properties of pharmaceutical tablets

Despite a well-established process understanding, quality issues for compressed oral solid dosage forms are frequently encountered during various drug product development and production stages. In the current work, a non-destructive contact ultrasonic experimental rig integrated with a collaborative robot arm and an advanced vision system is presented and employed to quantify the effect of the shape of a compressed tablet on its mechanical properties. It is observed that these properties are affected by the tablet geometric shapes and found to be linearly sensitive to the compaction pressures. It is demonstrated that the presented approach significantly improves the repeatability of the experimental waveform acquisition. In addition, with the increased confidence levels in waveform acquisition accuracy and corresponding pressure and shear wave speeds due to improved measurement repeatability, we conclude that pharmaceutical compact materials can indeed have a negative Poisson's ratio, therefore can be auxetic. The presented technique and instrument could find critical applications in continuous tablet manufacturing, and its real-time quality monitoring as measurement repeatability has been significantly improved, minimizing product quality variations.

Molecular-Layer-Defined Asymmetric Schottky Contacts in Organic Planar Diodes for Self-Powered Optoelectronic Synapses.

Optoelectronic synapses have been utilized as neuromorphic vision sensors for image preprocessing in artificial visual systems. Self-powered optoelectronic synapses, which can directly convert optical power into electrical power, are promising for practical applications. The Schottky junction tends to be a promising candidate as the energy source for electrical operations. However, fully utilizing the potential of Schottky barriers is still challenging. Herein, organic self-powered optoelectronic synapses with planar diode architecture are fabricated, which can simultaneously sense and process ultraviolet (UV) signals. The photovoltaic operations are facilitated by the built-in potential originating from the molecular-layer-defined asymmetric Schottky contacts. Diverse synaptic behaviors under UV light stimulation without external power supplies are facilitated by the interfacial carrier-capturing layer, which emulates the membranes of synapses. Furthermore, retina-inspired image preprocessing functions are demonstrated on the basis of synaptic plasticity. Therefore, our devices provide the potential for the development of power-efficient and advanced artificial visual systems.

Design of a deep learning visual system for the thickness measurement of each coating layer of TRISO-coated fuel particles

In the new generation of nuclear energy system, the thickness of the coating layer of tristructural isotropic (TRISO)-coated fuel particles is one of the most important parameters. Recently, some visual-based methods have been developed for the thickness measurement of each coating layer, but the existing method still lacks of practicality. In this study, an advanced visual system combined with the ceramographic section method and deep learning algorithms is designed to automatically measure the thickness values of each coating layer. In the designed visual system, an automatic image acquisition method is first achieved. After that, an accurate thickness measurement method is proposed based on the designed image segmentation model. Finally, to enhance the reliability and consistency of the measurement results, a tracing method is developed for the designed measurement system. The experimental results demonstrate that the designed system can accurately automatically measure the thickness values of each coating layer.

Retina-Inspired Self-Powered Artificial Optoelectronic Synapses with Selective Detection in Organic Asymmetric Heterojunctions.

The retina, the most crucial unit of the human visual perception system, combines sensing with wavelength selectivity and signal preprocessing. Incorporating energy conversion into these superior neurobiological features to generate core visual signals directly from incoming light under various conditions is essential for artificial optoelectronic synapses to emulate biological processing in the real retina. Herein, self‐powered optoelectronic synapses that can selectively detect and preprocess the ultraviolet (UV) light are presented, which benefit from high‐quality organic asymmetric heterojunctions with ultrathin molecular semiconducting crystalline films, intrinsic heterogeneous interfaces, and typical photovoltaic properties. These devices exhibit diverse synaptic behaviors, such as excitatory postsynaptic current, paired‐pulse facilitation, and high‐pass filtering characteristics, which successfully reproduce the unique connectivity among sensory neurons. These zero‐power optical‐sensing synaptic operations further facilitate a demonstration of image sharpening. Additionally, the charge transfer at the heterojunction interface can be modulated by tuning the gate voltage to achieve multispectral sensing ranging from the UV to near‐infrared region. Therefore, this work sheds new light on more advanced retinomorphic visual systems in the post‐Moore era.

Wafer-Scale Hybrid Integration of InP DFB Lasers on Si Photonics by Flip-Chip Bonding with sub-300nm Alignment Precision

InP DFB lasers are flip-chip bonded to 300mm Si photonic wafers using a pick-and-place tool with an advanced vision system, realizing high-precision and high-throughput passive assembly. By careful co-design of the InP-Si Photonics electrical, optical and mechanical interface, as well as dedicated alignment fiducials, sub-300nm post-bonding alignment precision is realized in a 25s cycle time. Optical coupling losses of -1.5±-0.5dB are achieved at 1550nm wavelength after epoxy underfill, with up to 40 mW of optical power coupled to the SiN waveguides on the Si photonics wafer. The bonding interface adds less than 10% to the series resistance of the laser diodes and post-bonding thermal resistance is measured to be 76K/W (or 27K.mm/W), mostly dominated by heat spreading resistance in the InP lasers as suggested by in-depth thermal modeling. Although the assembled lasers suffer from significant, unintentional optical backreflection from the fiber grating couplers used for optical characterization, laser linewidths well below 1MHz have been measured under specific drive conditions, as supported by a detailed laser noise analysis. Finally, we demonstrate the ability of bonded laser assemblies to pass early reliability tests.

Cooperative Light-Field Image Super-Resolution Based on Multi-Modality Embedding and Fusion With Frequency Attention

Light field (LF) imaging is an advanced visual perception system, which can record the intensity and direction information of light rays and provide multi-viewpoint images from a single capture. However, there is a trade-off between spatial and angular resolutions due to the restricted sensor size, which limits the wide applications of LF cameras. To address this problem, we propose a cooperative network to super-resolve LF sub-aperture images based on the multi-modality fusion. Specifically, in order to fully explore the LF information, we adopt various modalities and extract corresponding features to emphasise diverse LF characteristics. Then, we design a multi-scale fusion module to effectively integrate global and local LF features and apply frequency-aware attention mechanism to adaptively reinforce fused features. Extensive experiments demonstrate the superiority of our method on both qualitative and quantitative evaluations, with competitive execution efficiency.

МЕТОДИКА ИСПОЛЬЗОВАНИЯ BIM-ТЕХНОЛОГИЙ И ЛАЗЕРНОГО СКАНИРОВАНИЯ ДЛЯ РЕКОНСТРУКЦИИ И МОДЕРНИЗАЦИИ ОБЪЕКТОВ

The paper discusses the use of terrestrial laser scanning in order to obtain initial data in the design of complex objects using BIM technologies. BIM software products and ways of their use in a specific project for the reconstruction of Kostanay airport are indicated. The authors described the technology for obtaining source data using a Leica RTC360 laser scanner equipped with an advanced visual iner-tial system based on the SLAM algorithm. This allows automatically combining scans from different standing stations during the production process, and significantly saves time. The advantages of using such equipment in comparison with traditional measuring works are described. The root mean square errors of combining scans are presented. As a result a three-dimensional model of the building was obtained, which made it possible to restore the design documentation and the design of the new termi-nal building. At the same time, the design process was reduced by about 20%.

EHR STAR: The State‐Of‐the‐Art in Interactive EHR Visualization

AbstractSince the inception of electronic health records (EHR) and population health records (PopHR), the volume of archived digital health records is growing rapidly. Large volumes of heterogeneous health records require advanced visualization and visual analytics systems to uncover valuable insight buried in complex databases. As a vibrant sub‐field of information visualization and visual analytics, many interactive EHR and PopHR visualization (EHR Vis) systems have been proposed, developed, and evaluated by clinicians to support effective clinical analysis and decision making. We present the state‐of‐the‐art (STAR) of EHR Vis literature and open access healthcare data sources and provide an up‐to‐date overview on this important topic. We identify trends and challenges in the field, introduce novel literature and data classifications, and incorporate a popular medical terminology standard called the Unified Medical Language System (UMLS). We provide a curated list of electronic and population healthcare data sources and open access datasets as a resource for potential researchers, in order to address one of the main challenges in this field. We classify the literature based on multidisciplinary research themes stemming from reoccurring topics. The survey provides a valuable overview of EHR Vis revealing both mature areas and potential future multidisciplinary research directions.

Use of smartphones as optical metrology tools for surface wear detection

Proper wear level information and early wear detection are crucial goals in many engineering applications and industrial components in order to improve efficiency and reduce production, maintenance, or replacements costs. Furthermore, this should ideally be achieved with a user-friendly, low-cost, and easy to implement methodology for wear level monitoring and detection. In this work, we present the design of a new approach to accomplish early wear detection that is implemented by means of a stand-alone smartphone device and application providing real-time online metrology. The online monitoring is done by means of optical measurements and image processing based on the advanced smartphone vision system technology currently available in commercial devices. The developed mobile App works in continuous mode without interrupting the wear process. Specifically, it traces surface changes and monitors the progression of wear enabling just-in-time warning alarms for “significant wear” and “critical wear” detection. We demonstrate that critical wear of a surface prior to fatal rupture can be detected, which is the main objective in many industrial applications.

A Binocular MSCKF-Based Visual Inertial Odometry System Using LK Optical Flow

The odometry is an important part of intelligent mobile robots to achieve positioning and navigation functions. At present, the mainstream visual odometry locates only through the visual information obtained by camera sensors. Therefore, in the case of insufficient light, texture missing and camera jitter, the visual odometry is difficult to locate accurately. To solve the problem, we propose a binocular MSCKF-based visual inertial odometry system using Lucas-Kanade (LK) optical flow. Firstly, the Inertial Measurement Unit (IMU) is introduced to overcome the above problems. Moreover, LK optical flow algorithm is utilized to process the visual information obtained by the binocular camera, and MSCKF algorithm is employed to realize the fusion of visual information and inertial information, which improves the accuracy and efficiency of the visual inertial odometry system positioning. Finally, the proposed method is simulated on the European Robotics Challenge (EuRoc) dataset by Robot Operating System (ROS), and compared with two other advanced visual inertial odometry systems, ROVIO and MSCKF-mono. A large number of simulations verify that the proposed method can achieve accurate pose estimation, which is superior to the two existing advanced visual inertial odometry systems.

Automatic Optical Inspection System with Telecentric Optics and Phase-measuring Profilometry for Highly Accurate Localization of Electronic Packages

Typical visual placement inspection systems in semiconductor or electronics manufacturing industries use only highly-magnified two-dimensional imaging with controlled illumination for high accuracy. However, current semiconductor packaging technology needs more precise manufacturing processes for highly integrated semiconductor packages, in which high-precision alignment between the semiconductor die and the matched substrate is indispensable for successful electronic interconnection. To solve this problem, an advanced visual placement inspection system for 3D stacking of electronic components is proposed; it utilizes 3D profilometry and 2D telecentric optics. In addition to the introduction of the proposed system, the system calibration algorithms and the information processing algorithms for accurate positioning of the semiconductor die, as well as the implemented inspection system are described in detail. To verify the system performance, a series of real experiments on newly developed flip-chip packages for high-performance computing were performed, and the results are analyzed and discussed.

Evolution of neural processing for visual perception in vertebrates.

Visual perception requires both visual information and attention. This review compares, across classes of vertebrates, the functional and anatomical characteristics of (a) the neural pathways that process visual information about objects, and (b) stimulus selection pathways that determine the objects to which an animal attends. Early in the evolution of vertebrate species, visual perception was dominated by information transmitted via the midbrain (retinotectal) visual pathway, and attention was probably controlled primarily by a selection network in the midbrain. In contrast, in primates, visual perception is dominated by information transmitted via the forebrain (retinogeniculate) visual pathway, and attention is mediated largely by networks in the forebrain. In birds and nonprimate mammals, both the retinotectal and retinogeniculate pathways contribute critically to visual information processing, and both midbrain and forebrain networks play important roles in controlling attention. The computations and processing strategies in birds and mammals share some strikingly similar characteristics despite over 300 million years of independent evolution and being implemented by distinct brain architectures. The similarity of these functional characteristics suggests that they provide valuable advantages to visual perception in advanced visual systems. A schema is proposed that describes the evolution of the pathways and computations that enable visual perception in vertebrate species.