Sight Research Articles

Ground4Act: Leveraging visual-language model for collaborative pushing and grasping in clutter

The challenge in robotics is to enable robots to transition from visual perception and language understanding to performing tasks such as grasp and assembling objects, bridging the gap between “seeing” and “hearing” to “doing”. In this work, we propose Ground4Act, a two-stage approach for collaborative pushing and grasping in clutter using a visual-language model. In the grounding stage, Ground4Act extracts target features from multi-modal data via visual grounding. In the action stage, it embeds a collaborative pushing and grasping framework to generate the action's position and direction. Specifically, we propose a DQN-based reinforcement learning pushing policy that uses RGBD images as the state space to determine the push action's pixel-level coordinates and direction. Additionally, a least squares-based linear fitting grasping policy takes the target mask from the grounding stage as input to achieve efficient grasp. Simulations and real-world experiments demonstrate Ground4Act's superior performance. The simulation suite, source code, and trained models will be made publicly available.

Relating 2D isovists to audiovisual assessments of two urban spaces in a neighbourhood

Abstract In the field of urban design assessment, increasing attention to the nonexpert verbalizations is paid to engage social participation in planning processes, especially those not limited to static visual renderings. However, little has been explored on reproducible methods to collect these vocabularies and their relationship with architectural features. This study presents a hierarchical multifactor analysis to extract the main perceptual nonexpert clusters; and a linear correlation analysis to relate them with 2D isovist measures. In an on-line experiment, participants ( n = 20 n=20 ) elicited individual attributes ( n = 120 n=120 ) to describe their soundscape and visual perception and compared recorded urban environments ( n = 8 n=8 ). The results show that a percentage of nonexpert audio attributes correlate as well as the visual ones with isovist metrics.

The Development of a Smart Stick with SMS Emergency Location Feature for the Blind

Abstract: As humans tend to be reliant on their sense of sight to gather information about their surroundings, this allows them to feel safe and connected. White canes have been a common solution to visual impairment concerns however it was shown that such tools are still unable to resolve problems pertaining to obstructions on height level. This paper aims to create a smart stick which is able to resolve such concerns while including an emergency button feature that will inform emergency contacts via SMS of the location of the user once pressed. The gathered results will answer the following (a) reliability of the smart stick’s detection of obstructions, (b) effect of the environment to the performance of the device, and (c) effect of the distance between the user and emergency contact to the speed of transmission if the emergency button is pressed. It was found that the smart still is reliable in terms of detection with a high accuracy percentage and is able to determine if stairs are going up or down, the environment has little effect on the performance of the smart stick, and if the distance is farther the speed of transmission is more consistent.

Exploring the potential of pantomime games in medical education: Non-verbal exercise

The primary educational challenge in medical education is the lack of focus on non-verbal communication skills, such as body language, facial expressions, and gestures. This gap is crucial, particularly in multicultural and multilingual contexts where non-verbal cues can bridge language barriers and enhance understanding between healthcare professionals and patients. To address this, our wave explores the potential of integrating art, specifically pantomime games, into medical education as a non-verbal exercise. Inspired by ancient drama and role-play, pantomime games offer an innovative methodology for enhancing non-verbal communication skills. These games help students develop observation, empathy, and ambiguity tolerance—skills essential for effective healthcare delivery. Incorporating art and game-based learning into medical curricula has improved visual perception, personal reflection, and student engagement. The benefits of this integration extend globally, fostering emotional intelligence, creative thinking, and cross-cultural understanding. Educational games create a safe environment for students to experiment, learn from mistakes, and gain practical experience, ultimately reducing training time and instructor load. Moreover, non-verbal communication training can enhance professor-student interactions, improving feedback quality and learning outcomes. To implement these innovations, clear guidelines and effective facilitation are crucial. Educators should provide supportive environments for experimentation and learning, using minimal materials such as open spaces and simple props. High-resource settings can leverage AI technologies for feedback and create mobile applications to modernize the approach. This wave advocates for a student-centered, multimodal communication and learning environment, highlighting the transformative potential of integrating performing arts into medical education to prepare future healthcare providers better.

Bio-inspired deep neural local acuity and focus learning for visual image recognition

In the field of computer vision and image recognition, enabling the computer to discern target features while filtering out irrelevant ones poses a challenge. Drawing insights from studies in biological vision, we find that there is a local visual acuity mechanism and a visual focus mechanism in the initial conversion and processing of visual information, ensuring that the visual system can give ear to salient features of the target in the early visual observation phase. Inspired by this, we build a novel image recognition network to focus on the target features while ignoring other irrelevant features in the image, and further focus on the focus features based on the target features. Meanwhile, in order to comply with the output characteristics when similar features exist in different categories, we design a softer image label operation for similar features in different categories, which solves the correlation of labels between categories. Relevant experimental findings underscore the efficacy of our proposed method, revealing discernible advantages in comparison to alternative approaches. Visualization results further attest to the method’s capability to selectively focus on pertinent target features within the image, sidelining extraneous information.

Stereo depth‐fused 3D rendering for stacked stereo displays

AbstractWe propose a rendering method that enables the perception of 3D images at precise positions independent of the interocular distance of the observer by combining stacked stereo display and depth‐fused 3D (DFD) display and confirm its effectiveness through evaluation experiments. The depth range that can be expressed in stereo displays causes discomfort in observers because of the discrepancy between vergence and accommodation. However, the depth perceived differs depending on the interocular distance of the observer, resulting in discontinuity of the 3D image. Therefore, we propose a rendering method for stacked stereo displays that is independent of the observer's interocular distance using DFD display.

The tilt illusion arises from an efficient reallocation of neural coding resources at the contextual boundary.

The tilt illusion - a bias in the perceived orientation of a center stimulus induced by an oriented surround - illustrates how context shapes visual perception. While the tilt illusion has been the subject of quantitative study for over 85 years, we still lack a comprehensive account of the phenomenon that connects its neural and behavioral characteristics. Here, we demonstrate that the tilt illusion originates from a dynamic change in neural coding precision induced by the surround context. We simultaneously obtained psychophysical and fMRI responses from human subjects while they viewed gratings in the absence and presence of an oriented surround, and extracted sensory encoding precision from their behavioral and neural data. Both measures show that in the absence of a surround, encoding reflects the natural scene statistics of orientation. However, in the presence of an oriented surround, encoding precision is significantly increased for stimuli similar to the surround orientation. This local change in encoding is sufficient to accurately predict the behavioral characteristics of the tilt illusion using a Bayesian observer model. The effect of surround modulation increases along the ventral stream, and is localized to the portion of the visual cortex with receptive fields at the center-surround boundary. The pattern of change in coding accuracy reflects the surround-conditioned orientation statistics in natural scenes, but cannot be explained by local stimulus configuration. Our results suggest that the tilt illusion naturally emerges from a dynamic coding strategy that efficiently reallocates neural coding resources based on the current stimulus context.

A new test for evaluation of marginal cognitive function deficits in idiopathic normal pressure hydrocephalus through expressing texture recognition by sound symbolic words.

The number of dementia patients is increasing with population aging. Preclinical detection of dementia in patients is essential for access to adequate treatment. In previous studies, dementia patients showed texture recognition difficulties. Onomatopoeia or sound symbolic words (SSW) are intuitively associated with texture impressions and are less likely to be affected by aphasia and description of material perception can be easily obtained. In this study, we aimed to create a test of texture recognition ability expressed by SSW to detect the presence of mild cognitive disorders. The sound symbolic words texture recognition test (SSWTRT) is constructed from 12 close-up photos of various materials and participants were to choose the best SSW out of 8 choices to describe surface texture in the images in Japanese. All 102 participants seen in Juntendo University Hospital from January to August 2023 had a diagnosis of possible iNPH (age mean 77.9, SD 6.7). The answers were scored on a comprehensive scale of 0 to 1. Neuropsychological assessments included MMSE, FAB, and the Rey Auditory Verbal Learning Test (RAVLT), Pegboard Test, and Stroop Test from the EU-iNPH Grading Scale (GS). In study 1 the correlation between SSWTRT and the neuropsychological tests were analyzed. In study 2, participants were divided into two groups: the Normal Cognition group (Group A, n = 37) with MMSE scores of 28 points or above, and the Mild Cognitive Impairment group (Group B, n = 50) with scores ranging from 22 to 27 points, and its predictability were analyzed. In study 1, the total SSWTRT score had a moderate correlation with the neuropsychological test results. In study 2, there were significant differences in the SSWTRT scores between groups A and B. ROC analysis results showed that the SSWTR test was able to predict the difference between the normal and mildly impaired cognition groups. The developed SSWTRT reflects the assessment results of neuropsychological tests in cognitive deterioration and was able to detect early cognitive deficits. This test not only relates to visual perception but is likely to have an association with verbal fluency and memory ability, which are frontal lobe functions.

Visual Target Interaction Modeling with Multichannel Data Fusion

Facing the demand for accurate, fast, and natural human–computer interaction in the multi-screen control environment, the traditional exchange method of frequent switching control of multi-screen through mouse, keyboard, and other manual methods has problems of low efficiency, weak flexibility, and poor experience. In this paper, we research multi-screen precision control technology based on eye movement to break through the difficulties of precise recognition of human vision under complex environments and poor stability of vision estimation under frequent switching control of multi-screen. This study explores the technology for controlling multiple screens with high precision by tracking eye movements. It overcomes the challenges associated with reliably discerning human visual focus in intricate settings and the instability of visual assessments during rapid transitions between multiple screens. The experimental results of 1143 eye-movement capture tasks are studied, including the accurate recognition technology of human head posture and pupil gaze direction under complex backgrounds, illumination, and different visual field environments, and the theoretical modeling method of multi-screen precise target interaction based on eye-movement is explored. The expression data will be processed using convolutional neural networks, and the dataset will be trained through Python programming and tested on test samples collected by simulated flight crews in the laboratory with an accuracy rate of more than 85%.

Toward Enhanced Biomimetic Artificial Visual Systems Based on Organic Heterojunction Optoelectronic Synaptic Transistors

AbstractArtificial visual systems, inspired by the human eye, hold significant potential in artificial intelligence. Optoelectronic synapses, integrating image perception, processing, and memory in a single device, offer promising solutions. The human eye exhibits different recognition accuracies for objects under varying light conditions. Therefore, a more biomimetic visual system is needed to better fit actual application scenarios. Here, an organic heterojunction‐based optoelectronic synaptic transistor (OHOST) is proposed to enhance biomimetic artificial visual systems. By utilizing the excellent carrier capture ability of core‐multi‐shell quantum dots (QDs) and the high exciton dissociation efficiency of heterojunction interfaces, the device achieves a recognition capability under different light intensities closely resembling that of the human eye. Under optimal light intensity, the recognition accuracy for the modified national institute of standards and technology (MNIST) dataset can reach 91.52%. Nevertheless, under both low and high light intensities, the accuracy drops to a low level. This work pushes the development of artificial visual systems toward higher levels of biomimicry.

PHYSIOLOGICAL ASPECT OF AALOCHAK PITTA PERSPECTIVE OF THE MODERN ERA

Ayurvedic science is based on Dosha, Dhatu, and Mala Vijnana. The body's origin and maintenance depend upon the three primary and fundamental units or principles: Doshas Vata, Pitta, and Kapha. The first Aacharya to categorise Pitta Dosha into five categories was Sushruta. Rupa grahana, or the perception of vision, is attributed to the Pitta known as Alochaka Pitta, located in the retina (Drishti). Acharya Bhela has divided this Alochaka Pitta into two categories: intellectually (Buddhi) and visually or optically (Chakshu’s) differentiative. An example of an optically differentiative Pitta creates a differentiation by placing a picture. The visual process, also known as the physiology of vision, is the mechanism by which the outside world is perceived. Specialised cells called photoreceptors start the process that turns light rays into nerve impulses. There are two categories of photoreceptors: Rodes and cones.

Bright Structural-Phase-Pure CsPbI3 Core-PbSO4 Shell Nanoplatelets With Ultra-Narrow Emission Bandwidth of 77 meV at 630 nm.

Achieving a narrow emission bandwidth islong pursued for display applications. Among all primary colors, obtaining pure red emission with high visual perception is the most challenging. In this work, CsPbI3 halide perovskite nanoplatelets (NPLs) with rigorously controlled 2D [PbI6]4- octahedron layer number (n) are demonstrated. A perovskite core-PbSO4 shell structure is designed to prevent aggregation and fusion between NPLs, enabling consistent thickness and quantum confinement strength for each NPL. Consequently, exact n = 4 CsPbI3NPLs are demonstrated, exhibiting emission peaks around 630nm, with very narrow spectral bandwidths of <24nm and high absolute photoluminescence quantum yields up to 85%. The emission of n = 4 NPLs falls exactly within the pure-red region, closely aligning with the International Telecommunication Union Recommendation BT.2020 standard. Measurements suggest predominant stability and color homogeneity compared to traditional red-emitting CsPbIxBr3- x nanocrystals. Finally, proof-of-concept pure-red emissive light-emitting diodes (LEDs) are demonstrated by integrating n = 4 CsPbI3 NPLs films with a blue LED chip, showing an excellent external quantum efficiency of 18.3% and high brightness exceeding 3 × 106 nits. Stringent requirements for future display technologies, are satisfied based on the high color purity, stability, and brightness of CsPbI3 NPLs.

Image manipulation localization via dynamic cross-modality fusion and progressive integration

Image manipulation localization aims at distinguishing forged regions from the whole test image. Though many outstanding prior arts have been proposed for this task, there are still two issues that need to be further studied: (1) how to fuse diverse types of features with forgery clues; (2) how to progressively integrate multistage features for better localization performance. In this paper, we propose a tripartite progressive integration network (TriPINet) for end-to-end image manipulation localization. First, we extract both visual perception information, e.g., RGB input images, and visual imperceptible features, e.g., frequency and noise traces for forensic feature learning. Second, we develop a guided cross-modality dual-attention (gCMDA) module to fuse different types of forged clues. Third, we design a set of progressive integration squeeze-and-excitation (PI-SE) modules to improve localization performance by appropriately incorporating multiscale features in the decoder. Extensive experiments are conducted to compare our method with state-of-the-art image forensics approaches. The proposed TriPINet obtains competitive results on several benchmark datasets.

The Evolution of Simplifying Heuristics in Visual Cognition: Categorization, Specialization, and Visual Illusions.

Animals live in visually complex environments. As a result, visual systems have evolved mechanisms that simplify visual processing and allow animals to focus on the information that is most relevant to adaptive decision making. This review explores two key mechanisms that animals use to efficiently process visual information: categorization and specialization. Categorization occurs when an animal's perceptual system sorts continuously varying stimuli into a set of discrete categories. Specialization occurs when particular classes of stimuli are processed using distinct cognitive operations that are not used for other classes of stimuli. We also describe a nonadaptive consequence of simplifying heuristics: visual illusions, where visual perception consistently misleads the viewer about the state of the external world or objects within it. We take an explicitly comparative approach by exploring similarities and differences in visual cognition across human and nonhuman taxa. Considering areas of convergence and divergence across taxa provides insight into the evolution and function of visual systems and associated perceptual strategies.

Applications of Adaptive Optics Imaging for Studying Conditions Affecting the Fovea.

The fovea is a highly specialized region of the central retina, defined by an absence of inner retinal layers and the accompanying vasculature, an increased density of cone photoreceptors, a near absence of rod photoreceptors, and unique private-line photoreceptor to midget ganglion cell circuitry. These anatomical specializations support high-acuity vision in humans. While direct study of foveal shape and size is routinely performed using optical coherence tomography, examination of the other anatomical specializations of the fovea has only recently become possible using an array of adaptive optics (AO)-based imaging tools. These devices correct for the eye's monochromatic aberrations and permit cellular-resolution imaging of the living retina. In this article, we review the application of AO-based imaging techniques to conditions affecting the fovea, with an emphasis on how imaging has advanced our understanding of pathophysiology.

Predicting action type from visual perception: a kinematic study.

Predicting action type from visual perception: a kinematic study.

Linguistic and visual similarity judgements predict EEG representational dynamics in visual perception and sentence reading

Linguistic and visual similarity judgements predict EEG representational dynamics in visual perception and sentence reading

Parsing Pulses: Testing the Limits of Temporal Phase Perception in Human Vision

Parsing Pulses: Testing the Limits of Temporal Phase Perception in Human Vision

Spatial scrambling in human vision: investigating efficiency for discriminating scrambled letters using convolutional neural networks and confusion matrices

Spatial scrambling in human vision: investigating efficiency for discriminating scrambled letters using convolutional neural networks and confusion matrices

Inter-individual dependency in visual cognition: Observers adjust perceptual decisions based on others’ perception

Inter-individual dependency in visual cognition: Observers adjust perceptual decisions based on others’ perception