Visual Research Articles

Collective plasticity of binocular interactions in the adult visual system

Binocular visual plasticity can be initiated via either bottom-up or top-down mechanisms, but it is unknown if these two forms of adult plasticity can be independently combined. In seven participants with normal binocular vision, sensory eye dominance was assessed using a binocular rivalry task, before and after a period of monocular deprivation and with and without selective attention directed towards one eye. On each trial, participants reported the dominant monocular target and the inter-ocular contrast difference between the stimuli was systematically altered to obtain estimates of ocular dominance. We found that both monocular light- and pattern-deprivation shifted dominance in favour of the deprived eye. However, this shift was completely counteracted if the non-deprived eye’s stimulus was selectively attended. These results reveal that shifts in ocular dominance, driven by bottom-up and top-down selection, appear to act independently to regulate the relative contrast gain between the two eyes.

An empirical study of virtual museum based on dual-mode mixed visualization: the Sanxingdui bronzes

The design and implementation of virtual museums have evolved over decades, broadening the scope of traditional museum exhibitions and serving as an effective complement to physical museums. However, virtual visual effects have consistently fallen short of replicating the authentic exhibition experience and cannot substitute for physical museums. In this paper, we report a dual-mode mixed visualization system combining computational photography and binocular stereo vision techniques, and introduce an innovative virtual museum for Sanxingdui bronzes. This approach enables visitors to engage with the artifacts in a multifaceted manner-allowing for navigation, interaction, auditory guidance, and the examination of textural details as if handling the artifacts directly. Further analysis of participant responses in terms of visual cognition and aesthetic appreciation across different settings demonstrates that this dual-mode mixed visualization delivers not only a spatiotemporal experience consistent with the real world but also surreal visual effects. This immersive experience allows audiences to experience Sanxingdui art intimately, offering both high-resolution visuals and cost-effectiveness.

The experimental and numerical investigation of fracture behaviour in PMMA notched specimens under biaxial loading conditions – Tension with torsion

This paper presents the results of experimental fracture test of flat PMMA specimens under biaxial loading condition tension with torsion (proportional). The specimens were made in two thicknesses: 5 and 15 mm and were weakened with V-type edge notches with different root radii: 0.5; 2 and 10 mm. Thanks to the ARAMIS 3D 4 M non-contact vision system, measurement of the elongation and twist angle were recorded. During experimental part all of the deformation processes were observed using PHANTOM cameras. Obtained records made it possible to precisely indicate the moment of crack initiation (tensile force and torsional moment values). Using the microscopic observations the location of crack initiations were determined. Results obtained for biaxial loading were compared with those obtained for uniaxial tension and torsion. Based on experimental data the numerical calculation with FEM were carried out. The principal stress and plastic strain distribution under critical load, were obtained. The points of occurrence of stress maxima and plastic strain were indicated, which were taken as potential crack initiation sites. On the basis of the stress and plastic strain values measured at the critical points, a stress–strain fracture criterion was formulated, which was then positively verified. Additionally, new form of stress–strain fracture criterion was proposed. The bilinear form of the fracture criterion can be successfully used to predict fracture in PMMA flat specimens, regardless of the notch root radius, type of load or element thickness.

Gate-Tunable Positive and Negative Photoconductance in Near-Infrared Organic Heterostructures for In-Sensor Computing.

The rapid growth of sensor data in the artificial intelligence often causes significant reductions in processing speed and power efficiency. Addressing this challenge, in-sensor computing is introduced as an advanced sensor architecture that simultaneously senses, memorizes, and processes images at the sensor level. However, this is rarely reported for organic semiconductors that possess inherent flexibility and tunable bandgap. Herein, an organic heterostructure that exhibits a robust photoresponse to near-infrared (NIR) light is introduced, making it ideal for in-sensor computing applications. This heterostructure, consisting of partially overlapping p-type and n-type organic thin films, is compatible with conventional photolithography techniques, allowing for high integration density of up to 520 devices cm-2 with a 5µm channel length. Importantly, by modulating gate voltage, both positive and negative photoresponses to NIR light (1050nm) are attained, which establishes a linear correlation between responsivity and gate voltage and consequently enables real-time matrix multiplication within the sensor. As a result, this organic heterostructure facilitates efficient and precise NIR in-sensor computing, including image processing and nondestructive reading and classification, achieving a recognition accuracy of 97.06%. This work serves as a foundation for the development of reconfigurable and multifunctional NIR neuromorphic vision systems.

Image fusion algorithm for the collaborative use of unmanned aerial vehicles

Currently, the applicable scope of unmanned aircraft application is increasingly expanding. The promising field of unmanned aircraft enhancement is the implementation of some collaborative actions during controlled flight. This paper considers some issues of the group application of unmanned aerial vehicles (UAV) related to the coordinated planning and control of UAVs performing surveillance missions. Performing aerial search operations is technically complicated by the requirement to recognize a search object in arbitrary conditions, which can be both simple and severe environment. The search area is limited by the UAV capabilities, so, in order to improve the efficiency of search operations, UAVs are combined into groups. An algorithm for solving the problem of object search in arbitrary conditions by a group of unmanned aircraft is proposed. The advantage of search by a group of unmanned aircraft is the coverage of the larger search area in a conventional unit of time. This paper addresses the unmanned aircraft configuration, containing both the means of collaborative flight operation and a synthetic vision system. The image obtained by the synthetic vision system is both a source of navigation information and a means which reliably determines the result of search operations. Depending on the conditions of search operations, the image obtained by the synthetic vision system may require additional processing to use as intended. A fusion algorithm is proposed, which is characterized by adaptive adjustment of parameters in each frame individually for different image fragments. Based on the results obtained, it is planned to create a new product for commercial operation of unmanned aircraft.

Continuous lipreading based on acoustic temporal alignments

Visual speech recognition (VSR) is a challenging task that has received increasing interest during the last few decades. Current state of the art employs powerful end-to-end architectures based on deep learning which depend on large amounts of data and high computational resources for their estimation. We address the task of VSR for data scarcity scenarios with limited computational resources by using traditional approaches based on hidden Markov models. We present a novel learning strategy that employs information obtained from previous acoustic temporal alignments to improve the visual system performance. Furthermore, we studied multiple visual speech representations and how image resolution or frame rate affect its performance. All these experiments were conducted on the limited data VLRF corpus, a database which offers an audio-visual support to address continuous speech recognition in Spanish. The results show that our approach significantly outperforms the best results achieved on the task to date.

3D Heads-up digital filters for cataract surgery and corneal transplantation.

To describe the advantages of Ngenuity 3D digital filters for enhancing visualization during cataract and corneal transplant surgery. All surgeries were performed by the same experienced surgeon (L.M.) using the Ngenuity 3D heads-up visualization system connected to a microscope. Images were simultaneously captured with and without the filter in each of the following phases: endothelial evaluation, capsulorhexis, capsule rupture, vitreous leak, cortex removal, visco removal, corneal suture, descemetorhexis, DMEK graft preparation and insertion. In cataract surgery, green and monochrome filters facilitate capsulorhexis in difficult situations by highlighting the anterior capsule and can improve visualization of a posterior capsular tear. The enhanced contrast of the monochrome filter is also useful for cortex and viscoelastic removal. In corneal surgery, the green filter highlights the rim of the DMEK graft during tissue preparation, the yellow filter enhances the contrast of the stained DMEK graft in the anterior chamber, the monochrome filter simplifies descemetorhexis by improving visualization of the Descemet/endothelial layer and allows a better view in red saturated images while performing sutures. Ngenuity digital filters have the potential to enhance tissue visualization during cataract and corneal surgeries, especially in poor visibility conditions.

Systematic analysis of ocular features and responses of cultured spotted wolffish (Anarhichas minor).

A better understanding of unique anatomical and functional features of the visual systems of teleost fish could provide key knowledge on how these systems influence the health and survival of these animals in both wild and culture environments. We took a systematic approach to assess some of the visual systems of spotted wolffish (Anarhichas minor), a species of increasing importance in North Atlantic aquaculture initiatives. The lumpfish (Cyclopterus lumpus) was included in these studies in a comparative manner to provide reference. Histology, light and electron microscopy were used to study the spatial distribution and occurrence of cone photoreceptor cells and the nature of the retinal tissues, while immunohistochemistry was used to explore the expression patterns of two photoreceptor markers, XAP-1 and XAP-2, in both species. A marine bacterial infection paradigm in lumpfish was used to assess how host-pathogen responses might impact the expression of these photoreceptor markers in these animals. We define a basic photoreceptor mosaic and present an ultrastructural to macroscopic geographical configuration of the retinal pigment tissues in both animals. Photoreceptor markers XAP-1 and XAP-2 have novel distribution patterns in spotted wolffish and lumpfish retinas, and exogenous pathogenic influences can affect the normal expression pattern of XAP-1 in lumpfish. Live tank-side ophthalmoscopy and spectral domain optical coherence tomography (SD-OCT) revealed that normal cultured spotted wolffish display novel variations in the shape of the retinal tissue. These two complementary imaging findings suggest that spotted wolffish harbour unique ocular features not yet described in marine teleosts and that visual function might involve specific retinal tissue shape dynamics in these animals. Finally, extensive endogenous biofluorescence is present in the retinal tissues of both animals, which raises questions about how these animals might use retinal tissue in novel ways for visual perception and/or communication. This work advances fundamental knowledge on the visual systems of two economically important but now threatened North Atlantic teleosts and provides a basic foundation for further research on the visual systems of these animals in health versus disease settings. This work could also be useful for understanding and optimizing the health and welfare of lumpfish and spotted wolffish in aquaculture towards a one health or integrative perspective.

Low-grade glioma in children with neurofibromatosis type 1: surveillance, treatment indications, management, and future directions.

Neurofibromatosis type 1 (NF1) is an autosomal dominant cancer predisposition syndrome characterized by the development of both central and peripheral nervous system tumors. Low-grade glioma (LGG) is the most prevalent central nervous system tumor occurring in children with NF1, arising most frequently within the optic pathway, followed by the brainstem. Historically, treatment of NF1-LGG has been limited to conventional cytotoxic chemotherapy and surgery. Despite treatment with chemotherapy, a subset of children with NF1-LGG fail initial therapy, have a continued decline in function, or recur. The recent development of several preclinical models has allowed for the identification of novel, molecularly targeted therapies. At present, exploration of these novel precision-based therapies is ongoing in the preclinical setting and through larger, collaborative clinical trials. Herein, we review the approach to surveillance and management of NF1-LGG in children and discuss upcoming novel therapies and treatment protocols.

Heads-up 3-dimensional-visualization system and integrated intraoperative optical coherence tomography in infantile vitreoretinal surgeries

ObjectiveTo demonstrate initial experience of Heads-up 3-dimensional-visualization system and integrated intra operative optical coherence tomography (iOCT) in a spectrum of complex infantile vitreoretinal surgeries. MethodsTwo different 3-D-visualization systems were used during the surgery on case-to-case basis and iOCT was used as needed. The vitrectomy system used was the same for all the surgeries. The disease spectrum included retinal disorders due to retinopathy of prematurity, familial exudative vitreoretinopathy, and retinitis. DesignDescriptive case series of surgeries operated under 3-D visualization and conventional microscope. ResultsA total of 10 eyes of 7 patients were included; out of which 5 eyes of 4 unique cases were operated under 3-D-visualization system and remaining 5 eyes of other 3 cases under the conventional microscope. The cases present our initial experience of the 3-D visualization and iOCT system over the conventional microscope with appropriate imaging and surgical videos. ConclusionInfantile vitreoretinal surgeries are difficult to teach, require an ergonomic friendly surgical environment and unpredictable surgical adjustments, which are the chief advantages of the discussed technologies. Heads-up 3-D-visualization system and iOCT appeared to be powerful new tools that can be explored further for use in complex infantile vitreoretinal surgeries.

Distinct patterns of monocular advantage for facial emotions in social anxiety

Individuals with social anxiety often exhibit atypical processing of facial expressions. Previous research in social anxiety has primarily emphasized cognitive bias associated with face processing and the corresponding abnormalities in cortico-limbic circuitry, yet whether social anxiety influences early perceptual processing of emotional faces remains largely unknown. We used a psychophysical method to investigate the monocular advantage for face perception (i.e., face stimuli are better recognized when presented to the same eye compared to different eyes), an effect that is indicative of early, subcortical processing of face stimuli. We compared the monocular advantage for different emotional expressions (neutral, angry and sad) in three groups (N = 24 per group): individuals clinically diagnosed with social anxiety disorder (SAD), individuals with high social anxiety in subclinical populations (SSA), and a healthy control (HC) group of individuals matched for age and gender. Compared to SSA and HC groups, we found that individuals with SAD exhibited a greater monocular advantage when processing neutral and sad faces. While the magnitudes of monocular advantages were similar across three groups when processing angry faces, individuals with SAD performed better in this condition when the faces were presented to different eye. The former findings suggest that social anxiety leads to an enhanced role of subcortical structures in processing nonthreatening expressions. The latter findings, on the other hand, likely reflect an enhanced cortical processing of threatening expressions in SAD group. These distinct patterns of monocular advantage indicate that social anxiety altered representation of emotional faces at various stages of information processing, starting at an early stage of the visual system.

Spectral Characteristics Analysis of Images Matrix Masking Results

The article describes the results of a computational experiment to assess the capabilities of extracting useful information if an image masked by quasi-orthogonal matrices sent over an open channel became available to a third party. Hadamard and Mersenne matrices of symmetric and cyclic structure are considered. The results confirm the data that images masked by small-sized matrix leaves edges of the original image on the masked image. However, with an increase in the size of the masking matrix, all considered in the article matrices reliably hides the original image during visual analysis. Masking by symmetric Mersenne-Walsh matrices and cyclic Mersenne matrices based on modified M-sequences provides better spectral secrecy of masked images in comparison with Hadamard matrices. Mersenne matrices of cyclic structure, with equal sizes of the image and the masking matrix, bring the phase spectrum of the masked image to a form close in spectrum to uniform noise, which makes their use more preferable based on the considerations that the human visual system is extremely sensitive to phase-frequency distortions of the visual information.

A novel hybrid-view technique for accurate mass estimation of kimchi cabbage using computer vision

This study addresses the accurate estimation of kimchi cabbage mass, as cabbage leaves exhibit size variability and complex leaf structures. Conventional mass estimation methods, which rely solely on external imaging, often overlook leaf gaps. To improve accuracy, we propose an innovative computer vision system utilizing hybrid-view images and detailed saturation analysis. Our system quantifies the impact of leaf gaps on mass using features from the saturation channel of images of bisected cabbage. Our proposed method can be easily integrated into existing workflows and has the potential to improve labor efficiency. Our approach outperforms the conventional method (R2 of 0.66 and relative error of 8.68%), achieving a 0.92 R2 value and lowering the relative error to 4.22%. This advancement offers a robust solution for the mass estimation of kimchi cabbage and suggests potential applications for other foods and crops with internal voids.

Unsafe-Net: YOLO v4 and ConvLSTM based computer vision system for real-time detection of unsafe behaviours in workplace

AbstractUnsafe behaviour is a leading cause of death or injury in the workplace, including many accidents. Despite regular safety inspections in workplaces, many accidents occur as a result of breaches of occupational health and safety protocols. In these environments, despite efforts to prevent accidents and losses in hazardous environments, human error cannot be completely eliminated. In particular, in computer-based solutions, automated behaviour detection has low accuracy, is very costly, not real-time and requires a lot of time. In this study, we propose Unsafe-Net, a hybrid computer vision approach using deep learning models for real-time classification of unsafe behaviours in workplace. For the Unsafe-Net, a dataset is first specifically created by capturing 39 days of video footage from a factory. Using this dataset, YOLO v4 and ConvLSTM methods are combined for object detection and video understanding to achieve fast and accurate results. In the experimental studies, the classification accuracy of unsafe behaviours using the proposed Unsafe-Net method is 95.81% and the average time for action recognition from videos is 0.14 s. In addition, the Unsafe-Net has increased the real-time detection speed by reducing the average video duration to 1.87 s. In addition, the system is installed in a real-time working environment in the factory and employees are immediately alerted by the system, both audibly and visually, when unsafe behaviour occurs. As a result of the installation of the system in the factory environment, it has been determined that the recurrence rate of unsafe behaviour has been reduced by approximately 75%.

A non-image-forming visual circuit mediates the innate fear of heights in male mice

The neural basis of fear of heights remains largely unknown. In this study, we investigated the fear response to heights in male mice and observed characteristic aversive behaviors resembling human height vertigo. We identified visual input as a critical factor in mouse reactions to heights, while peripheral vestibular input was found to be nonessential for fear of heights. Unexpectedly, we found that fear of heights in naïve mice does not rely on image-forming visual processing by the primary visual cortex. Instead, a subset of neurons in the ventral lateral geniculate nucleus (vLGN), which connects to the lateral/ventrolateral periaqueductal gray (l/vlPAG), drives the expression of fear associated with heights. Additionally, we observed that a subcortical visual pathway linking the superior colliculus to the lateral posterior thalamic nucleus inhibits the defensive response to height threats. These findings highlight a rapid fear response to height threats through a subcortical visual and defensive pathway from the vLGN to the l/vlPAG.

Demonstrating computer vision to small‐ and medium‐sized enterprises in manufacturing: Toward overcoming costs and implementation challenges

AbstractComputer vision (CV) systems are crucial in various fields, such as manufacturing. However, the general feedback from small and medium‐sized enterprises (SMEs) in manufacturing indicates that implementing CV systems are out of reach due to their high costs, the need for expertise, and the complexities of Machine Learning (ML). To demonstrate the feasibility and applicability for SMEs, we present a cost‐effective portable CV‐based demonstrator for SMEs in manufacturing. Incorporating low‐cost hardware components, the system integrates open‐source software and employs Transfer Learning to adapt pre‐existing ML models. We present two illustrative use cases with fault detection and inventory management based on plastic brick datasets including various colors and shapes to demonstrate the system's effectiveness. Bringing this into a showable demonstrator, all components are integrated into a portable suitcase as a plug and play demonstration. We showcase the portable demonstrator at German television and four different industrial fairs, leveraging these dynamic platforms for direct interaction with SMEs and stakeholders. Firsthand insights and feedback from SMEs regarding our demonstration and their challenges, as well as opportunities for CV in manufacturing were received and summarized in this research.

Light‐adaptive Mimicking Retina with In Situ Image Memorization via Resistive Switching Photomemristor Arrays

AbstractArtificial visual memory systems have been of particular interest since the development of machine vision and bionic robots. Ordinarily, the conventional system architecture requires the complex integration of two functional modules, a photo‐sensor and a memory device, which greatly limits the operating efficiency and increases the extra energy consumption. Nonetheless, other simply configured optoelectronics memory devices generally face challenges of adaption in complex light environments. Here, a resistive switching (RS) perovskite‐based photomemristor is presented that mimics the retina function. The dual function of light perception and in situ storage are both achieved. In the dark condition, it exhibits impressive memory performance with a high ON/OFF ratio of 104, a long retention time of over 104 s, and a low operating voltage of 0.38 V. With illumination, it shows self‐powered, broadband photo‐detecting characteristics with responsivity of 70 mA W−1 and detectivity of 7.5 × 1010 Jones. More importantly, benefiting from the material dual‐phase configuration, the highly steady photo‐adjusted RS windows are achieved. Its light‐adaptive memory application in dynamic environments is further demonstrated using the mimicking retina for a machine eye. This work can provide a strategy for enhanced RS photomemristor and its application in changing and varied scenarios.

Parallel Chords: an audio-visual analytics design for parallel coordinates

AbstractOne of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

Conditions on detecting tripartite entangled state in psychophysical experiments

This paper explores the sensitivity of the human visual system to quantum entangled light. We examine the possibility of human subjects perceiving multipartite entangled state through psychophysical experiments. Our focus begins with a bipartite entangled state to make a comparative study with the literature by taking into account additive noise for false positive on bipartite entanglement perception by humans. After that, we limit our similar investigation to a tripartite entangled state for simplicity in higher dimensions. To model the photodetection by humans, we employ the probability of seeing determined for coherently amplified photons in Fock number states, including an additive noise. Our results indicate that detecting bipartite and tripartite entanglement with the human eye is possible for a certain range of additive noise levels and visual thresholds. Finally, we discuss several alternative amplification methods.Graphical abstract

Foveated pancake lens design for improved optical performance and eye rotation support.

We propose a novel, to our knowledge, approach to address the limitations of traditional pancake lenses for virtual reality headsets, such as low image contrast and poor performance when eyes rotate. The design leverages the foveated nature of human vision, achieving a superior modulation transfer function in the foveal area to enhance optical performance significantly. Furthermore, the pancake lens design is presented that considers the rotation of the user's pupil position, maintaining optimal image quality even when the user's eye rotates. The proposed method presents the parameters and optimization of a novel pancake lens that utilizes the characteristics of the human visual system and accounts for the rotation of the pupil position of the user, leading to improvements in image quality and user experience. The lens design and image simulation results are presented to demonstrate the effectiveness of the approach.