Conventional Vision Research Articles

Study of Visual Evoked Potentials in Schoolchildren: A Promising Aid to Pediatric Ophthalmology.

Background Visual evoked potential (VEP) is a noninvasive investigation conducted to identify abnormalities in the visual system. It is especially suitable for young children who are unable to express visual symptoms or participate in conventional vision tests. This study was undertaken to examine the VEP among schoolchildren to assess the functionality of their optic pathway. Methodology This short-term observational study was performed in the Clinical Neurophysiology Unit of the Physiology Department of a rural medical college. The study population consisted of 60 schoolchildren aged 7-12. Both eyes were examined for transient pattern reversal VEP recordingsusing a Recorders & Medicare Systems Electromyography-Evoked Potential recorder (RMS EMG-EP MARK-II Pvt. Ltd., Chandigarh, India). Results VEPs were analyzed for latency and amplitude of the main components, namely P100, N70, and N155. The results showedmarkedly extended P100 latency in33.33%, i.e., eight outof 24 eyes of standard (std.) III children. Similar latency prolongation was obtained in 36.36% (eight out of 22)eyes of std. IV, 30% (six of 20 eyes) of std. V, 13.63% (three of 22 eyes) of std. VI, and 50% (eight of 16 eyes) in std. VII and VIIIchildren. A markedly reduced P100 amplitude was observedin two of 20 eyes (10%) of std. V, two of 16 eyes (12.5%) in std. VII and VIIIchildren, amounting to a P100amplitude abnormality in5%eyesin toto. The interocular differences in all VEP parameters amongthe subjects were statistically insignificant. Conclusion In schoolchildren in whom normal latencies and amplitudes were obtained, the presence of reproducible VEPs indicated the normal functionalstatus of their visual pathway. On the other hand,in those children where alteredVEP findings were found, it hinted towardcomplementary informationthat they may have underlying ocular disordersthat were yet to be diagnosed. Hence, this study provides insight into the assessment of visual system function, which is primarily difficult in young children.

LCFE-X: Pluggable Local Channel Feature Extraction model enhancement method for hyperspectral image classification

Spectral information of hyperspectral images offers high-dimensional features for downstream tasks. Local spectral signatures in different wavelength ranges are critical for material identification and pixel classification. In line with conventional vision methods, recent studies have focused on bolstering the spatial modelling capabilities of transformers. However, the focus remains predominantly global when the channel dimension is considered. Current research on enhancing local channel feature extraction is primarily model specific and lacks generalisable methods that can be applied to various models. Therefore, this study introduces a Local Channel Feature Extraction (LCFE) module. It extracts spectral-spatial features and key features within each band group and then integrates the features to obtain local channel features. Subsequently, a pluggable LCFE-X model enhancement method is proposed. This involves performing LCFE at the front end of the model and fusing the extracted features with the original features. Three enhanced hyperspectral image classification models, namely, LCFE-ViT, LCFE-SF, and LCFE-MF, are constructed based on recently proposed models. Extensive experiments demonstrate that the LCFE module effectively extracts local spectral signatures, and the proposed method enhances the performance of the various basic models. The code is available at https://github.com/YiSu1997/LCFE-X.

An Enhanced Feature Extraction Framework for Cross-Modal Image–Text Retrieval

In general, remote sensing images depict intricate scenes. In cross-modal retrieval tasks involving remote sensing images, the accompanying text includes numerus information with an emphasis on mainly large objects due to higher attention, and the features from small targets are often omitted naturally. While the conventional vision transformer (ViT) method adeptly captures information regarding large global targets, its capability to extract features of small targets is limited. This limitation stems from the constrained receptive field in ViT’s self-attention layer, which hinders the extraction of information pertaining to small targets due to interference from large targets. To address this concern, this study introduces a patch classification framework based on feature similarity, which establishes distinct receptive fields in the feature space to mitigate interference from large targets on small ones, thereby enhancing the ability of traditional ViT to extract features from small targets. We conducted evaluation experiments on two popular datasets—the Remote Sensing Image–Text Match Dataset (RSITMD) and the Remote Sensing Image Captioning Dataset (RSICD)—resulting in mR indices of 35.6% and 19.47%, respectively. The proposed approach contributes to improving the detection accuracy of small targets and can be applied to more complex image–text retrieval tasks involving multi-scale ground objects.

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

Vision Transformer and its variants have demonstrated great potential in various computer vision tasks. However conventional vision transformers often focus on global dependency at a coarse level, which results in a learning challenge on global relationships and fine-grained representation at a token level. In this paper, we introduce Multi-scale Attention Fusion into transformer (MAFormer), which explores local aggregation and global feature extraction in a dual-stream framework for visual recognition. We develop a simple but effective module to explore the full potential of transformers for visual representation by learning fine-grained and coarse-grained features at a token level and dynamically fusing them. Our Multi-scale Attention Fusion (MAF) block consists of: i) a local window attention branch that learns short-range interactions within windows, aggregating fine-grained local features; ii) global feature extraction through a novel Global Learning with Down-sampling (GLD) operation to efficiently capture long-range context information within the whole image; iii) a fusion module that self-explores the integration of both features via attention. Our MAFormer achieves state-of-the-art results on several common vision tasks. In particular, MAFormer-L achieves 85.9% Top-1 accuracy on ImageNet, surpassing CSWin-B and LV-ViT-L by 1.7% and 0.6% respectively. On MSCOCO, MAFormer outperforms the prior art CSWin by 1.7% mAPs on object detection and 1.4% on instance segmentation with similar-sized parameters. With the performance, MAFormer demonstrates the ability to generalize across various visual benchmarks and prospects as a general backbone for different self-supervised pre-training tasks in the future.

DICAM: Deep Inception and Channel-wise Attention Modules for underwater image enhancement

In underwater environments, imaging devices suffer from water turbidity, attenuation of lights, scattering, and particles, leading to low quality, poor contrast, and biased color images. This has led to great challenges for underwater condition monitoring and inspection using conventional vision techniques. In recent years, underwater image enhancement has attracted increasing attention due to its critical role in improving the performance of current computer vision tasks in underwater object detection and segmentation. As existing methods, built mainly from natural scenes, have performance limitations in improving the color richness and distributions we propose a novel deep learning-based approach namely Deep Inception and Channel-wise Attention Modules (DICAM) to enhance the quality, contrast, and color cast of the hazy underwater images. The proposed DICAM model enhances the quality of underwater images, considering both the proportional degradations and non-uniform color cast. Extensive experiments on two publicly available underwater image enhancement datasets have verified the superiority of our proposed model compared with several state-of-the-art conventional and deep learning-based methods in terms of full-reference and reference-free image quality assessment metrics. The source code of our DICAM model is available at https://github.com/hfarhaditolie/DICAM.

MF 2 ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofing

In recent times, Face Anti-spoofing (FAS) has gained significant attention in both academic and industrial domains. Although various convolutional neural network (CNN)-based solutions have emerged, multimodal approaches incorporating RGB, depth, and information retrieval (IR) have exhibited better performance than unimodal classifiers. The increasing veracity of modern presentation attack instruments results in a persistent need to enhance the performance of such models. Recently, self-attention-based vision transformers (ViT) have become a popular choice in this field. Their fundamental aspects for multimodal FAS have not been thoroughly explored yet. Therefore, we propose a novel framework for FAS called MF 2 ShrT, which is based on a pretrained vision transformer. The proposed framework uses overlap patches and parameter sharing in the ViT network, allowing it to utilize multiple modalities in a computationally efficient manner. Furthermore, to effectively fuse intermediate features from different encoders of each ViT, we explore a T-encoder-based hybrid feature block enabling the system to identify correlations and dependencies across different modalities. MF 2 ShrT outperforms conventional vision transformers and achieves state-of-the-art performance on benchmarks CASIA-SURF and WMCA, demonstrating the efficiency of transformer-based models for presentation attack detection PAD).

Advanced visual components inspired by animal eyes.

Artificial vision systems pervade our daily lives as a foremost sensing apparatus in various digital technologies, from smartphones to autonomous cars and robotics. The broad range of applications for conventional vision systems requires facile adaptation under extreme and dynamic visual environments. However, these current needs have complicated individual visual components for high-quality image acquisition and processing, which indeed leads to a decline in efficiency in the overall system. Here, we review recent advancements in visual components for high-performance visual processing based on strategies of biological eyes that execute diverse imaging functionalities and sophisticated visual processes with simple and concise ocular structures. This review first covers the structures and functions of biological eyes (i.e., single-lens eyes and compound eyes), which contain micro-optic components and nanophotonic structures. After that, we focus on their inspirations in imaging optics/photonics, light-trapping and filtering components, and retinomorphic devices. We discuss the remaining challenges and notable biological structures waiting to be implemented.

Triple line-scan camera measurement for efficient and accurate narrow-space 3-D scanning with motion estimation.

Vision-based three-dimensional (3-D) shape measurement plays a crucial role in the inspection of vehicles and trains in the field of transportation. Consequently, the demand for more comprehensive narrow-space inspection has become an inevitable necessity and presents a great challenge to conventional vision methods. We propose the following efficient and accurate narrow-space 3-D scanning method based on triple line-scan cameras. First, the structure of coplanar dual line-scan cameras with a narrow reflector is proposed, making it easy for the optical path to enter a narrow space and obtain 3-D shape information. Efficient in-motion measurement with a large field of view is thereby achieved without the worries that often accompany narrow-space constraints. Secondly, an additional line-scan camera is attached to the coplanar dual cameras and creates a time-space constraint in point cloud stitching direction through the triple line-scan camera structure. With an efficient algorithm template including matching and motion estimation, accurate point cloud stitching is ensured. Lastly, point cloud texture mapping and light source optimization are realized. Our experimental results prove that the method realizes low-distortion in-motion reconstruction in narrow space despite motion variation.

Generative AI and neural networks towards advanced robot cognition

Enhancing autonomy and applicability of robotic systems across diverse scenarios, requires efficient environment perception. Conventional vision systems are highly performing but limited to simple tasks, while AI based ones require extensive data collection, processing and training. This paper presents a framework leveraging generative AI and Neural Networks to implement a dynamically updateable perception system. A multimodal conditional Generative Adversarial Network generates large image datasets which are automatically annotated by a Large Multimodal Model. A Convolutional Neural Network performs further dataset augmentation. A case study on the inspection of aircraft fuel tanks is used to showcase the potential of the approach.

Vision Transformers for Breast Cancer Human Epidermal Growth Factor Receptor 2 Expression Staging without Immunohistochemical Staining

Accurate staging of human epidermal growth factor receptor 2 (HER2) expression is vital for evaluating breast cancer treatment efficacy. However, it typically involves costly and complex immunohistochemical staining, along with hematoxylin and eosin staining. This work presents customized vision transformers for staging HER2 expression in breast cancer using only hematoxylin and eosin-stained images. The proposed algorithm comprised three modules: a localization module for weakly localizing critical image features using spatial transformers, an attention module for global learning via vision transformers, and a loss module to determine proximity to a HER2 expression level based on input images by calculating ordinal loss. Results, reported with 95% CIs, reveal the proposed approach's success in HER2 expression staging: area under the receiver operating characteristic curve, 0.9202±0.01; precision, 0.922±0.01; sensitivity, 0.876±0.01; and specificity, 0.959±0.02 over fivefold cross-validation. Comparatively, this approach significantly outperformed conventional vision transformer models and state-of-the-art convolutional neural network models (P<0.001). Furthermore, it surpassed existing methods when evaluated on an independent test data set. This work holds great importance, aiding HER2 expression staging in breast cancer treatment while circumventing the costly and time-consuming immunohistochemical staining procedure, thereby addressing diagnostic disparities in low-resource settings and low-income countries.

E2VIDX: improved bridge between conventional vision and bionic vision.

Common RGBD, CMOS, and CCD-based cameras produce motion blur and incorrect exposure under high-speed and improper lighting conditions. According to the bionic principle, the event camera developed has the advantages of low delay, high dynamic range, and no motion blur. However, due to its unique data representation, it encounters significant obstacles in practical applications. The image reconstruction algorithm based on an event camera solves the problem by converting a series of "events" into common frames to apply existing vision algorithms. Due to the rapid development of neural networks, this field has made significant breakthroughs in past few years. Based on the most popular Events-to-Video (E2VID) method, this study designs a new network called E2VIDX. The proposed network includes group convolution and sub-pixel convolution, which not only achieves better feature fusion but also the network model size is reduced by 25%. Futhermore, we propose a new loss function. The loss function is divided into two parts, first part calculates the high level features and the second part calculates the low level features of the reconstructed image. The experimental results clearly outperform against the state-of-the-art method. Compared with the original method, Structural Similarity (SSIM) increases by 1.3%, Learned Perceptual Image Patch Similarity (LPIPS) decreases by 1.7%, Mean Squared Error (MSE) decreases by 2.5%, and it runs faster on GPU and CPU. Additionally, we evaluate the results of E2VIDX with application to image classification, object detection, and instance segmentation. The experiments show that conversions using our method can help event cameras directly apply existing vision algorithms in most scenarios.

Refractive Laser Surgery for Vision Conditions

&#x0D; None of the identified studies were of sufficient quality to formulate conclusions on the clinical effectiveness of refractive laser surgeries compared to conventional vision correction for people with vision conditions.&#x0D; &#x0D; The identified evidence for the clinical effectiveness of photorefractive keratotomy (PRK) was limited to 1 low-quality study that did not detect a visual acuity difference between participants who had PRK and participants who wore contact lenses.&#x0D; Low-quality evidence from 1 study found participants who had undergone PRK had greater vision-related quality of life (QoL) than participants who had not had PRK. Another low-quality study found that participants who wore contact lenses had greater vision-related QoL than participants who had undergone a laser-assisted in situ keratomileusis (LASIK) procedure.&#x0D; Low-quality evidence suggested that contact lenses resulted in fewer incidences of vision loss events than LASIK.&#x0D; &#x0D; &#x0D; No evidence-based guidelines on best practices for refractive laser surgeries met the criteria for this review.&#x0D;

Memory impairment was ameliorated by corticolimbic microinjections of arachidonylcyclopropylamide (ACPA) and miRNA-regulated lentiviral particles in a streptozotocin-induced Alzheimer's rat model

The present study aimed to investigate the effect of corticolimbic cannabinoid CB1 receptors activity on memory impairment in the intracerebroventricular (ICV)-streptozotocin (STZ) animal model of Alzheimer's like-disease. This study also assessed whether the corticolimbic overexpression of miRNA-137 or -let-7a could increase the endocannabinoids by inhibiting the monoglyceride lipase (MAGL) to ameliorate STZ response. The results showed that ICV microinjection of STZ (3 mg/kg/10 μl) impaired passive avoidance memory retrieval. The chronic microinjection of arachidonylcyclopropylamide (ACPA; 10 ng/0.5 μl), a selective cannabinoid CB1 receptor agonist, into the hippocampal CA1 region, the central amygdala (CeA) or the medial prefrontal cortex (mPFC) ameliorated the amnesic effect of ICV-STZ. Intra-CA1 or -CeA microinjection of ACPA alone did not affect memory retrieval, while its microinjection into the mPFC impaired memory formation. Based on bioinformatics analysis and verification of the MAGL gene, miRNA-137 and -let-7a were chosen to target the expression levels of MAGL in the corticolimbic regions. The chronic corticolimbic microinjection of lentiviral particles containing miRNA-137 or -let-7a ameliorated ICV-STZ-induced memory impairment. The high transfection efficiency was determined for each virus using comparing fluorescent and conventional vision. Corticolimbic overexpression of miRNA-137 or -let-7a decreased the MAGL gene expression that encodes the MAGL enzyme to increase the endocannabinoids. Thus, among the molecular mechanisms and signaling pathways involved in the pathophysiology of Alzheimer's disease (AD), it is worth mentioning the role of endocannabinoids in the corticolimbic regions. CB1 receptor agonists, miRNA-137 or -let-7a, may be potential therapeutic targets against cognitive decline in AD.

Parkinsonian Tremor Detection with Compact Convolutional Transformer from Bispectrum Representation of tri-Axial Accelerometer Signals.

After the breakthroughs of Transformer networks in Natural Language Processing (NLP) tasks, they have led to exciting progress in visual tasks as well. Nonetheless, there has been a parallel growth in the number of parameters and the amount of training data, which led to the conclusion that Transformers are not suited for small datasets. This paper is the first to convey the feasibility of Compact Convolutional Transformers (CCT) for the prediction of Parkinsonian postural tremor based on the Bispectrum (BS) representation of IMU accelerometer time series. The dataset includes tri-axial accelerometer signals collected unobtrusively in-the-wild while subjects are on a phone call, and labelled by neurologists and signal processing experts. The BS is a noise-immune, higher-order representation that reflects a signal's deviation from Gaussianity and measures quadratic phase coupling. We performed comparative classification experiments using the CCT, pre-trained CNNs such as VGG-16 and ResNet-50, and the conventional Vision Transformer (ViT). Our model achieves competitive prediction accuracy and F1 score of 96% with only 1.016 M trainable parameters, compared to the ViT with 21.659 M trainable parameters, in a five-fold cross-validation scheme. Our model also outperforms pre-trained CNNs such as VGG-16 and ResNet-50. Furthermore, we show that the performance gains are maintained when training on a larger dataset of BS images. Our effort here is motivated by the hypothesis that data-efficient transformers outperform transfer learning using pre-trained CNNs, paving the way for promising deep learning architecture for small-scale, novel and noisy medical imaging datasets.Clinical relevance- Novel deep learning model for unobtrusive prediction of Parkinsonian Postural Tremor from Bispectrum image representation of tri-axial accelerometer signals collected in-the-wild.

Neuromorphic Artificial Vision Systems Based on Reconfigurable Ion‐Modulated Memtransistors

Conventional vision systems suffer from lots of data handling between memory and processing units. Inspired by how humans recognize noisy images and the flexible modulation on the timescale of ion dynamics inside an emerging memtransistor, a novel neuromorphic vision system is reported based on the ion‐modulated memtransistors. By controlling the ion‐doping processes under adequate stimuli strengths, both short‐term and long‐term ion dynamics can be utilized to deliver energy‐efficient data processing. When dealing with image reconstructions, the short‐term accumulation effect of the device can help filter noises in a set of received noisy images while enhancing the original pattern information. The increased contrast can help distinguish the actual contents. To demonstrate systematic performances with the reconfiguration of devices, the nonlinear relationship between channel conductance variation and the amplitude of gate pulses into the network‐level simulation is extracted. Also, with the nonvolatile conductance change characteristic, the task of recognizing noisy images is performed to verify the versatility of ion‐modulated memtransistors in the neuromorphic artificial vision systems. An interactive preprint version of the article can be found here: https://doi.org/10.22541/au.167939089.96499861/v1

Abstract 5437: Vision transformers for breast cancer human epidermal growth factor receptor 2 (HER2) expression staging without immunohistochemical (IHC) staining

Abstract Purpose: Staging of human epidermal growth factor receptor 2 (HER2) expression is crucial for evaluating the effectiveness of breast cancer treatment. However, it involves an expensive and challenging immunohistochemical (IHC) staining in addition to hematoxylin and eosin (H&amp;E) staining. Here, we argue that customized vision transformers are effective in breast cancer HER2 expression staging using only H&amp;E-stained images. Methods: Our algorithm has three modules: a localization module to weakly localize the necessary features of an image via spatial transformers, an attention module to learn global features from the image using vision transformers, and a loss module that determines the closest one to a HER2 expression level from input images by calculating the ordinal loss. We utilized a publicly available breast cancer HER2 expression dataset, the breast cancer immunohistochemical image (BCI) dataset, which contains paired H&amp;E and IHC stained images. Results were calculated with 95% confidence intervals and t-test was used to evaluate the significance of the proposed model against other models. Results: Our approach achieved area under receiver operating characteristics curve (AUC), precision, sensitivity, and specificity of 0.9202±0.01, 0.922±0.01, 0.876±0.01, and 0.959±0.02, respectively, averaged over five-fold cross-validation, in staging HER2 expression. The proposed method showed better performance than the conventional vision transformer model and state-of-the-art convolutional neural network models with statistical significance of p&amp;lt;0.01 in all cases. Conclusion: Our findings are important in assisting HER2 expression staging in breast cancer treatment, while avoiding the need for an expensive and time-consuming IHC staining procedure. Comparison of the proposed model with state-of-the-art models. CNN-convolutional neural network Study Neural network Model AUC Couture et al. (2018) CNN VGG16 0.75 Shamai et al. (2019) CNN ResNet50 0.74 Yang et al. (2021) CNN ResNet50 0.76 Naik et al. (2020) CNN ResNet50 0.78 Khater et al. (2021) CNN ShuffleNet 0.75 Conde-Sousa et al. (2022) CNN EfficientNet 0.88 Farahmand et al. (2022) CNN InceptionV3 0.90 Proposed Vision transformer vitb_16 0.92 Citation Format: Gelan Ayana, Eonjin Lee, Se-woon Choe. Vision transformers for breast cancer human epidermal growth factor receptor 2 (HER2) expression staging without immunohistochemical (IHC) staining. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5437.

Non-destructive discrimination of homochromatic foreign materials in cut tobacco based on VIS-NIR hyperspectral imaging.

The presence of foreign materials (FM) not only reduces the commercial value of tobacco and the quality of cigarette products, but also affects the aroma and flavor of cigarettes. Existing tobacco deblending equipment has received little study with respect to homochromatic FM. In the present study, visible-near infrared (VIS-NIR) hyperspectral imaging technique combined with chemometrics were used to identify and visualize the homochromatic FM on the surface of thining tobacco. A comparison with conventional vision method was made to analyze the feasibility of the method. The importance of detecting FM in cut tobacco was further demonstrated by first studying the volatile organic compounds produced in cigarette mixed FM smoke and their effects on human health before conducting hyperspectral experiments. The results indicated that solid-phase microextraction and gas chromatography mass spectrometry could detect volatile organic compounds in mainstream cigarette smoke that were not cigarette components and affected consumer health. Then, spectral features of the samples were extracted from hyperspectral images for building identification models to distinguish FM from cut tobacco. The visual RGB values of cut tobacco and FM were also used for the analysis of the recognition models. The results showed that the accuracy, precision and recall reached 100.00% using the back propagation artificial neural network classification model based on the principal component analysis raw wavelengths. The visualization results based on the optimal model produced clearer localization than conventional computer vision method. The present study revealed that the VIS-NIR hyperspectral imaging technology had advantage in the detection and localization of FM on the surface of thinning tobacco, which provided a foundation for improving the quality and safety of cut tobacco production. © 2023 Society of Chemical Industry.

Neuromorphic vision based control for the precise positioning of robotic drilling systems

The manufacturing industry is currently witnessing a paradigm shift with the unprecedented adoption of industrial robots, and machine vision is a key perception technology that enables these robots to perform precise operations in unstructured environments. However, the sensitivity of conventional vision sensors to lighting conditions and high-speed motion sets a limitation on the reliability and work-rate of production lines. Neuromorphic vision is a recent technology with the potential to address the challenges of conventional vision with its high temporal resolution, low latency, and wide dynamic range. In this paper and for the first time, we propose a novel neuromorphic vision based controller for robotic machining applications to enable faster and more reliable operation, and present a complete robotic system capable of performing drilling tasks with sub-millimeter accuracy. Our proposed system localizes the target workpiece in 3D using two perception stages that we developed specifically for the asynchronous output of neuromorphic cameras. The first stage performs multi-view reconstruction for an initial estimate of the workpiece’s pose, and the second stage refines this estimate for a local region of the workpiece using circular hole detection. The robot then precisely positions the drilling end-effector and drills the target holes on the workpiece using a combined position-based and image-based visual servoing approach. The proposed solution is validated experimentally for drilling nutplate holes on workpieces placed arbitrarily in an unstructured environment with uncontrolled lighting. Experimental results prove the effectiveness of our solution with maximum positional errors of less than 0.2 mm, and demonstrate that the use of neuromorphic vision overcomes the lighting and speed limitations of conventional cameras. The findings of this paper identify neuromorphic vision as a promising technology that can expedite and robustify robotic manufacturing processes in line with the requirements of the fourth industrial revolution. • We present a neuromorphic visual controller approach for precise robotic machining. • We devise an event-based 3D reconstruction method for the localization of workpieces. • We develop an event-based method for the detection and tracking of circular objects. • We preform experiments that validate the proposed method in a robotic drilling setup. • The presented system can perform drilling tasks with sub-millimeter precision. • Results demonstrate the performance and robustness advantages of neuromorphic vision.

CE-BART: Cause-and-Effect BART for Visual Commonsense Generation

"A Picture is worth a thousand words". Given an image, humans are able to deduce various cause-and-effect captions of past, current, and future events beyond the image. The task of visual commonsense generation has the aim of generating three cause-and-effect captions for a given image: (1) what needed to happen before, (2) what is the current intent, and (3) what will happen after. However, this task is challenging for machines, owing to two limitations: existing approaches (1) directly utilize conventional vision-language transformers to learn relationships between input modalities and (2) ignore relations among target cause-and-effect captions, but consider each caption independently. Herein, we propose Cause-and-Effect BART (CE-BART), which is based on (1) a structured graph reasoner that captures intra- and inter-modality relationships among visual and textual representations and (2) a cause-and-effect generator that generates cause-and-effect captions by considering the causal relations among inferences. We demonstrate the validity of CE-BART on the VisualCOMET and AVSD benchmarks. CE-BART achieved SOTA performance on both benchmarks, while an extensive ablation study and qualitative analysis demonstrated the performance gain and improved interpretability.

"Smart Optometry" phone-based application as a visual acuity testing tool among pediatric population.

To evaluate the validity of a smartphone-based application for visual acuity (VA) testing in children and to compare parent and clinician-performed VA to standard VA assessment. A cohort of 100 children aged <18 years old was recruited. Subjects were randomly assigned to either start with conventional distance VA chart screening or the smartphone VA assessment twice by both the clinician and the caregiver if applicable. Near VA scores were assessed using the near vision E chart. Accuracy and reliability values were analyzed. One hundred patients with an average age of 9.92 ± 3.0 years old were enrolled. The difference between conventional distance and application logMAR values was -0.023, and the difference between near vision and application logMAR values was -0.004. "Smart Optometry" had a sensitivity of 89.3% in detecting subnormal VA compared with conventional vision testing methods. Sensitivity in detecting subnormal VA was found to be higher in younger age groups up to 91.7% in comparison with the older age groups. The interclass correlation of application-measured VA scores by the caregivers and the clinician were 0.77 (95% CI; 0.67-0.83) using single measures and 0.87 (95%CI; 0.8-0.9) using average measures. "Smart Optometry" phone application was found to be an acceptable home-based VA testing tool with good inter-rater reliability for young children showing good sensitivity in detecting subnormal VA, but lower sensitivity in detecting amblyopia.