Visual Capabilities Research Articles

Home Automation System using OpenCV and Arduino

The research focuses on the integration of computer vision capabilities through OpenCV with Arduino-based hardware to create an efficient and versatile home automation system. The system employs cameras for image processing, allowing it to recognize and respond to various environmental changes within the home. The hardware setup utilizes Arduino microcontrollers as the central processing units, enabling seamless communication between the OpenCV algorithms and the connected devices. A user-friendly interface facilitates remote monitoring and control of home appliances through a mobile or web application. By combining the power of OpenCV for image processing and Arduino for real-time control, this home automation system provides a smart and adaptive solution for enhancing security, energy efficiency, and overall convenience within a household. The research’s modular design allows for future expansions and integration with emerging technologies, ensuring scalability and relevance in the rapidly evolving field of smart home automation.

Enhancing digital marketing of MSMEs through comprehensive photography training in Hendrosari Village, Gresik

MSMEs in Hendrosari Village have been struggling with digital marketing due to a lack of expertise in creating compelling visual content, which has led to poor online visibility and stagnating sales. Additionally, these businesses face challenges in keeping up with ever-evolving digital trends, limited access to affordable digital tools, and insufficient training opportunities, further hampering their ability to compete in a growing online marketplace. An extensive photography training program was launched, aimed at equipping 15 MSMEs in Hendrosari Village, the target groups of this service initiative, with essential skills in lighting, composition, and post-processing techniques. This program aims to enhance their visual content creation capabilities, helping them improve their online presence and market competitiveness. The program combined theoretical lessons with hands-on practice, allowing participants to immediately apply what they learned. The community service team conducted observations and interviews before and after the training to assess changes in product photo quality and the impact on sales. The average photo quality assessment score increased from 3 to 4.5. In addition, some participants reported a 15 percent increase in income after learning new photography techniques. Based on the evaluation process, it shows that the photography training has strong potential to improve the competitiveness of MSMEs in Hendrosari village, which can inspire similar programs in other areas.

All-in-One 2D Molecular Crystal Optoelectronic Synapse for Polarization-Sensitive Neuromorphic Visual System.

Neuromorphic visual systems (NVSs) hold the potential to not only preserve but also enhance human visual capabilities. One such augmentation lies in harnessing polarization information from light reflected or scattered off surfaces like bees, which can disclose unique characteristics imperceptible to the human eyes. While creating polarization-sensitive optoelectronic synapses presents an intriguing avenue for equipping NVS with this capability, integrating functions like polarization sensitivity, photodetection, and synaptic operations into a singular device has proven challenging. This integration typically necessitates distinct functional components for each performance metric, leading to intricate fabrication processes and constraining overall performance. Herein, a pioneering linear polarized light sensitive synaptic organic phototransistor (OPT) based on 2D molecular crystals (2DMCs) with highly integrated, all-in-one functionality, is demonstrated. By leveraging the superior crystallinity and molecular thinness of 2DMC, the synaptic OPT exhibits comprehensive superior performance, including a linear dichroic ratio up to 3.85, a high responsivity of 1.47 × 104 A W-1, and the adept emulation of biological synapse functions. A sophisticated application in noncontact fingerprint detection achieves a 99.8% recognition accuracy, further highlights its potential. The all-in-one 2DMC optoelectronic synapse for polarization-sensitive NVS marks a new era for intelligent perception systems.

Abnormal activation patterns in MT+ during visual motion perception in major depressive disorder.

Previous studies have found that patients with Major Depressive Disorder (MDD) exhibit impaired visual motion perception capabilities, and multi-level abnormalities in the human middle temporal complex (MT+), a key brain area for processing visual motion information. However, the brain activity pattern of MDD patients during the perception of visual motion information is currently unclear. In order to study the effect of depression on the activity and functional connectivity (FC) of MT+ during the perception of visual motion information, we conducted a study combining task-state fMRI and psychophysical paradigm to compare MDD patients and healthy control (HC). Duration threshold was examined through a visual motion perception psychophysical experiment. In addition, a classic block-design grating motion task was utilized for fMRI scanning of 24 MDD patients and 25 HC. The grating moved randomly in one of eight directions. We examined the neural activation under visual stimulation conditions compared to the baseline and FC. Compared to HC group, MDD patients exhibited increased duration threshold. During the task, MDD patients showed decreased beta value and percent signal change in left and right MT+. In the sample comprising MDD and HC, there was a significant negative correlation between beta value in right MT+ and duration threshold. And in MDD group, activation in MT+ were significantly correlated with retardation score. Notably, no such differences in activation were observed in primary visual cortex (V1). Furthermore, when left MT+ served as the seed region, compared to the HC, MDD group showed increased FC with right calcarine fissure and surrounding cortex and decreased FC with left precuneus. Overall, the findings of this study highlight that the visual motion perception function impairment in MDD patients relates to abnormal activation patterns in MT+, and task-related activity are significantly connected to the retardation symptoms of the disease. This not only provides insights into the potential neurobiological mechanisms behind visual motion perception disorder in MDD patients from the aspect of task-related brain activity, but also supports the importance of MT+ as a candidate biomarker region for MDD.

OpenECAD: An efficient visual language model for editable 3D-CAD design

Computer-aided design (CAD) tools are utilized in the manufacturing industry for modeling everything from cups to spacecraft. These programs are complex to use and typically require years of training and experience to master. Structured and well-constrained 2D sketches and 3D constructions are crucial components of CAD modeling. A well-executed CAD model can be seamlessly integrated into the manufacturing process, thereby enhancing production efficiency. Deep generative models of 3D shapes and 3D object reconstruction models have garnered significant research interest. However, most of these models produce discrete forms of 3D objects that are not editable. Moreover, the few models based on CAD operations often have substantial input restrictions. In this work, we fine-tuned pre-trained models to create OpenECAD models (0.55B, 0.89B, 2.4B and 3.1B), leveraging the visual, logical, coding, and general capabilities of visual language models. OpenECAD models can process images of 3D designs as input and generate highly structured 2D sketches and 3D construction commands, ensuring that the designs are editable. These outputs can be directly used with existing CAD tools’ APIs to generate project files. To train our network, we created a series of OpenECAD datasets. These datasets are derived from existing public CAD datasets, adjusted and augmented to meet the specific requirements of vision language model (VLM) training. Additionally, we have introduced an approach that utilizes dependency relationships to define and generate sketches, further enriching the content and functionality of the datasets.

Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges

Multimodal Large Language Models (MLLMs) harness comprehensive knowledge spanning text, images, and audio to adeptly tackle complex problems. This study explores the ability of MLLMs in visually solving the Traveling Salesman Problem (TSP) and Multiple Traveling Salesman Problem (mTSP) using images that portray point distributions on a two-dimensional plane. We introduce a novel approach employing multiple specialized agents within the MLLM framework, each dedicated to optimizing solutions for these combinatorial challenges. We benchmarked our multi-agent model solutions against the Google OR tools, which served as the baseline for comparison. The results demonstrated that both multi-agent models—Multi-Agent 1, which includes the initializer, critic, and scorer agents, and Multi-Agent 2, which comprises only the initializer and critic agents—significantly improved the solution quality for TSP and mTSP problems. Multi-Agent 1 excelled in environments requiring detailed route refinement and evaluation, providing a robust framework for sophisticated optimizations. In contrast, Multi-Agent 2, focusing on iterative refinements by the initializer and critic, proved effective for rapid decision-making scenarios. These experiments yield promising outcomes, showcasing the robust visual reasoning capabilities of MLLMs in addressing diverse combinatorial problems. The findings underscore the potential of MLLMs as powerful tools in computational optimization, offering insights that could inspire further advancements in this promising field.

Ultralow-Energy-Consumption Photosynaptic Transistor Utilizing Conjugated Polymers/Perovskite Quantum Dots Nanocomposites With Ligand Density Optimization.

The photosynaptic transistor stands as a promising contender for overcoming the von Neumann bottleneck in the realm of photo-communication. In this context, photonic synaptic transistors is developed through a straightforward solution process, employing an organic semiconducting polymer with pendant-naphthalene-containing side chains (PDPPNA) in combination with ligand-density-engineered CsPbBr3 perovskite quantum dots (PQDs). This fabrication approach allows the devices to emulate fundamental synaptic behaviors, encompassing excitatory postsynaptic current, paired-pulse facilitation, the transition from short-to-long-term memory, and the concept of "learning experience." Notably, the phototransistor, incorporating the blend of the PDPPNA and CsPbBr3 PQDs washed with ethyl acetate, achieved an exceptional memory ratio of 104. Simultaneously, the same device exhibited an impressive paired-pulse facilitation ratio of 223% at a moderate operating voltage of -4V and an extraordinarily low energy consumption of 0.215 aJ at an ultralow operating voltage of -0.1mV. Consequently, these low-voltage synaptic devices, constructed with a pendant side-chain engineering of organic semiconductors and a ligand density engineering of PQDs through a simple fabrication process, exhibit substantial potential for replicating the visual memory capabilities of the human brain.

Comparing Vision-Capable Models, GPT-4 and Gemini, With GPT-3.5 on Taiwan's Pulmonologist Exam.

Introduction The latest generation of large language models (LLMs) features multimodal capabilities, allowing them to interpret graphics, images, and videos, which are crucial in medical fields. This study investigates the vision capabilities of the next-generation Generative Pre-trained Transformer 4 (GPT-4) and Google's Gemini. Methods To establish a comparative baseline, we used GPT-3.5, a model limited to text processing, and evaluated the performance of both GPT-4 and Gemini on questions from the Taiwan Specialist Board Exams in Pulmonary and Critical Care Medicine. Our dataset included 1,100 questions from 2012 to 2023, with 100 questions per year. Of these, 1,059 were in pure text and 41 were text with images, with the majority in a non-English language and only six in pure English. Results For each annual exam consisting of 100 questions from 2013 to 2023, GPT-4 achieved scores of 66, 69, 51, 64, 72, 64, 66, 64, 63, 68, and 67, respectively. Gemini scored 45, 48, 45, 45, 46, 59, 54, 41, 53, 45, and 45, while GPT-3.5 scored 39, 33, 35, 36, 32, 33, 43, 28, 32, 33, and 36. Conclusions These results demonstrate that the newer LLMs with vision capabilities significantly outperform the text-only model. When a passing score of 60 was set, GPT-4 passed most exams and approached human performance.

Rational design of a dual-mode fluorescent probe for portable detection of pyriproxyfen in the environment and food

The development of a fluorescent probe for pyriproxyfen (PPF) is crucial due to its potential threat to human health. However, the chemical inertness and low solubility of PPF present significant challenges for the detection of PPF in aqueous solutions using fluorescent probes. Herein, we have originally proposed a complex based on 2-(4-(dimethylamino)phenyl)−3-hydroxy-6,7-dimethoxy-4 H-chromen-4-one (HOF) and serum albumin (SA) as a dual-mode fluorescent probe, HOF@SA. This probe utilizes an indicator displacement assay (IDA) to release the dye HOF from the probe at low PPF concentrations (< 10 µM) and embeds the free dye HOF into the micelle of PPF at high concentrations (> 10 µM). This results in dual-mode fluorescent response characteristics for PPF: a turn-off response at low concentrations and a ratiometric response at high concentrations. An investigation of sensing behavior of HOF@SA for PPF detection exhibits rapid response (< 60 s), high sensitivity (LOD ∼4.7 ppb), high selectivity, and excellent visual detection capability (from cyan to yellow). Moreover, with the aid of a portable device, this method enables to analyze PPF in environmental and food samples. These results promote the advancement of a fluorescent probe approach for PPF analysis in environment and food.

Decoding the Developmental Trajectory of Energy Trading in Power Markets through Bibliometric and Visual Analytics

This research leverages bibliometric methodologies, enhanced by the visual analytics capabilities of CiteSpace, to meticulously examine the evolution and current trends in energy trading within power markets, analyzing 642 scholarly articles from the Web of Science Core Collection spanning from 1996 to 2023. The study aims to illuminate the prevailing research landscape, growth patterns, and future directions in energy trading dynamics. Key findings include: (1) A noticeable escalation in the volume of publications, especially from 2021 to 2023, indicating a burgeoning interest and rapid evolution in this research area; (2) The author and institutional collaboration networks are in a nascent stage, with a predominantly China-centric international collaboration pattern, including significant partnerships with the United States, Australia, and the United Kingdom; (3) The focal points of research are centered around themes such as “energy management”, “demand-side innovation”, “decentralized energy trading”, and “strategic optimization”, covering areas such as intelligent grid technologies, energy market dynamics, and sustainable energy solutions. The study recommends enhancing collaborative networks, fusing technological and strategic dimensions in research, increasing focus and funding for emerging technologies, and promoting wider international and cross-disciplinary collaborations to enrich the understanding of energy trading dynamics in the context of electricity markets.

Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion Model

Visual In-Context Learning (ICL) has emerged as a promising research area due to its capability to accomplish various tasks with limited example pairs through analogical reasoning. However, training-based visual ICL has limitations in its ability to generalize to unseen tasks and requires the collection of a diverse task dataset. On the other hand, existing methods in the inference-based visual ICL category solely rely on textual prompts, which fail to capture fine-grained contextual information from given examples and can be time-consuming when converting from images to text prompts. To address these challenges, we propose Analogist, a novel inference-based visual ICL approach that exploits both visual and textual prompting techniques using a text-to-image diffusion model pretrained for image inpainting. For visual prompting, we propose a self-attention cloning (SAC) method to guide the fine-grained structural-level analogy between image examples. For textual prompting, we leverage GPT-4V's visual reasoning capability to efficiently generate text prompts and introduce a cross-attention masking (CAM) operation to enhance the accuracy of semantic-level analogy guided by text prompts. Our method is out-of-the-box and does not require fine-tuning or optimization. It is also generic and flexible, enabling a wide range of visual tasks to be performed in an in-context manner. Extensive experiments demonstrate the superiority of our method over existing approaches, both qualitatively and quantitatively. Our project webpage is available at https://analogist2d.github.io.

Wavelength‐Selective Photodetector and Neuromorphic Visual Sensor Utilizing Intrinsic Defect Semiconductor

AbstractWith the rapid developments of Artificial Intelligence (AI) and the Internet of Things (IoT), increasingly intricate and expanding application scenarios are placing higher demands on current machine vision capabilities. Therefore, there is a pressing need to simultaneously achieve diverse functionalities, simple designs, and efficient computing in vision devices. Here, the study develops a two‐terminal optoelectronic device utilizing a single 2D intrinsic defect semiconductor In2S3. The device demonstrates wavelength‐selective photodetection and neuromorphic visual capabilities, attributed to defect‐related charge‐trapping/de‐trapping processes. As a photodetector, the device exhibits a high photoresponsivity of 473.6 A W−1, a high external quantum efficiency of 1.6 × 105%, and a fast rise/fall time of 0.3/1.4 ms at the wavelength of 359 nm. As an all‐in‐one neuromorphic visual device, optoelectronic‐driven fundamental synaptic functions, including paired‐pulse facilitation (PPF), short‐term plasticity (STP), long‐term plasticity (LTP), and “learning‐experience”, are successfully mimicked at the wavelength of 671 nm. Moreover, one‐shot recognition of the 12‐letter image “SHAN XI NORMAL” is achieved through an artificial convolutional neural network. This study provides a new strategy for developing compact high‐level intelligence systems for complex application scenarios.

Real-Time SLAM and Faster Object Detection on a Wheeled Lifting Robot with Mobile-ROS Interaction

Wheeled lifting robots have found widespread applications in various industrial and logistical environments. However, traditional robots are far from adequate in terms of visual perception capabilities. Additionally, their remote control methods suffer from inefficiencies, which tend to bring safety concerns. To address these issues, this work proposes an autonomous multi-sensor-enabled wheeled lifting robot system, i.e., AMSeWL-R, to facilitate remote autonomous operations. Specifically, AMSeWL-R integrates real-time simultaneous localization and mapping with object detection on a wheeled lifting robot. Additionally, a novel mobile-ROS interaction method is proposed to achieve real-time communication and control between a mobile device and a ROS host. Furthermore, a lightweight object detection algorithm based on YOLOv8, i.e., YOLOv8-R, is proposed to achieve faster detection. Experimental results validate the effectiveness of the AMSeWL-R system for accurately detecting objects and mapping its surroundings. Furthermore, TensorRT acceleration is employed during practical testing on a Jetson Nano to achieve real-time detection using the proposed YOLOv8-R, demonstrating its efficacy in real-world scenarios.

Enhancing end-to-end control in autonomous driving through kinematic-infused and visual memory imitation learning

This paper presents an exploration, study, and comparison of various alternatives to enhance the capabilities of an end-to-end control system for autonomous driving based on imitation learning by adding visual memory and kinematic input data to the deep learning architectures that govern the vehicle. The experimental comparison relies on fundamental error metrics (MAE, MSE) during the offline assessment, supplemented by several external complementary fine-grain metrics based on the behavior of the ego vehicle at several urban test scenarios in the CARLA reference simulator in the online evaluation. Our study focuses on a lane-following application using different urban scenario layouts and visual bird-eye-view input. The memory addition involves architectural modifications and different sensory input types. The kinematic data integration is managed with a modified input. The experiments encompass both typical driving scenarios and extreme never-seen conditions. Additionally, we conduct an ablation study examining various memory lengths and densities. We prove experimentally that incorporating visual memory capabilities and kinematic input data makes the driving system more robust and able to handle a wider range of challenging situations, including those not encountered during training, in terms of reduction of collisions and speed self-regulation, resulting in a 75% enhancement. All the work we present here, including model architectures, trained model weights, comparison tool, and the dataset, is open-source, facilitating replication and extension of our findings.

A novel infrared and visible image fusion algorithm based on global information-enhanced attention network

The fusion of infrared and visible images aims to extract and fuse thermal target information and texture details to the fullest extent possible, enhancing the visual understanding capabilities of images for both humans and computers in complex scenes. However, existing methods have difficulties in preserving the comprehensiveness of source image feature information and enhancing the saliency of image texture information. Therefore, we put forward a novel infrared and visible image fusion algorithm based on global information-enhanced attention network (GIEA). Specifically, we develop an attention-guided Transformer module (AGTM) to make sure the fused images have enough global information. This module combines the convolutional neural network and Transformer to perform adequate feature extraction from shallow to deep layers, and utilize the attention network for multi-level feature-guided learning. Then, we build the contrast enhancement module (CENM), which enhances the feature representation and contrast of the image so that the fused image contains significant texture information. Furthermore, our network is driven to fully preserve the texture and structure details of the source images with a loss function that consists of content loss and total variance loss. Numerous experiments demonstrate that our fusion approach outperforms other fusion approaches in both subjective and objective assessments.

Oral mucosal lesions triage via YOLOv7 models

Background/purposeThe global incidence of lip and oral cavity cancer continues to rise, necessitating improved early detection methods. This study leverages the capabilities of computer vision and deep learning to enhance the early detection and classification of oral mucosal lesions. MethodsA dataset initially consisting of 6903 white-light macroscopic images collected from 2006 to 2013 was expanded to over 50,000 images to train the YOLOv7 deep learning model. Lesions were categorized into three referral grades: benign (green), potentially malignant (yellow), and malignant (red), facilitating efficient triage. ResultsThe YOLOv7 models, particularly the YOLOv7-E6, demonstrated high precision and recall across all lesion categories. The YOLOv7-D6 model excelled at identifying malignant lesions with notable precision, recall, and F1 scores. Enhancements, including the integration of coordinate attention in the YOLOv7-D6-CA model, significantly improved the accuracy of lesion classification. ConclusionThe study underscores the robust comparison of various YOLOv7 model configurations in the classification to triage oral lesions. The overall results highlight the potential of deep learning models to contribute to the early detection of oral cancers, offering valuable tools for both clinical settings and remote screening applications.

Bidirectional Braille-speech Communication System for Deafblind Students

Deafblindness is a type of dual disability wherein both visual and auditory capabilities are significantly impaired. Special communication methods have been developed for the deafblind community, yet these methods require that both people involved have prior knowledge and training to successfully communicate; limiting deafblind people’s social interactions, particularly in academic settings. This paper describes the development of a device that enables two-way communication between a severe to profoundly deafblind user and a hearing person with no prior knowledge of Braille and no additional intermediaries. A Convolutional Neural Network (CNN) scheme for speech recognition was designed and implemented along with the development of an algorithm capable of developing both text-to-speech and Finger-Braille-to-text conversion. Lastly, a system integration via 3D modelling and additive manufacturing was carried out to deliver a functional prototype. The resulting device aims to allow deafblind students to send and receive information entirely in finger Braille, using buttons and vibrotactile feedback, while the hearing tutor receives auditory messages and speaks to reply, making the educational experience as familiar as possible for both parties. Users testing the device achieved an average typing accuracy of over 95% and demonstrated understanding of commands transmitted through the device’s components.

TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model

Multi-modal large language models (MLLMs), such as GPT-4, exhibit great comprehension capabilities on human instruction, as well as zero-shot ability on new downstream multi-modal tasks. To integrate the different modalities within a unified embedding space, previous MLLMs attempted to conduct visual instruction tuning with massive and high-quality image-text pair data, which requires substantial costs in data collection and training resources. In this article, we propose TOMGPT (Text-Only training Multi-modal GPT), a cost-effective MLLM tuned solely on easily accessible text data with much fewer resources. Along with pre-trained visual-linguistic coupled modality space (e.g., CLIP and ALIGN model), a text-only training strategy is devised to further project the aligned multi-modal latent space to that of LLM, endowing the LLM with visual comprehension capabilities in an efficient manner. Instead of enormous image-text training data required by previous MLLMs, we find that TOMGPT can be well-tuned with fewer yet diverse GPT-generated free-form text data, as we establish the semantic connection between LLM and pre-trained vision-language model. A quantitative evaluation is conducted on both MME and LVLM, which are recently released and extensively utilized MLLM benchmarks. The experiments reveal that TOMGPT achieved reliable performance compared to numerous models trained on a large amount of image-text pair data. Case studies are also presented, demonstrating TOMGPT’s broad understanding and dialogue capabilities across diverse image categories.

Mobile AR Interaction Design Patterns for Storytelling in Cultural Heritage: A Systematic Review

The recent advancements in mobile technologies have enabled the widespread adoption of augmented reality (AR) to enrich cultural heritage (CH) digital experiences. Mobile AR leverages visual recognition capabilities and sensor data to superimpose digital elements into the user’s view of their surroundings. The pervasive nature of AR serves several purposes in CH: visitor guidance, 3D reconstruction, educational experiences, and mobile location-based games. While most literature reviews on AR in CH focus on technological aspects such as tracking algorithms and software frameworks, there has been little exploration of the expressive affordances of AR for the delivery of meaningful interactions. This paper (based on the PRISMA guidelines) considers 64 selected publications, published from 2016 to 2023, that present mobile AR applications in CH, with the aim of identifying and analyzing the (mobile) AR (interaction) design patterns that have so far been discussed sporadically in the literature. We identify sixteen (16) main UX design patterns, as well as eight (8) patterns with a single occurrence in the paper corpus, that have been employed—sometimes in combination—to address recurring design problems or contexts, e.g., user navigation, representing the past, uncovering hidden elements, etc. We analyze each AR design pattern by providing a title, a working definition, principal use cases, and abstract illustrations that indicate the main concept and its workings (where applicable) and explanation with respect to examples from the paper corpus. We discuss the AR design patterns in terms of a few broader design and development concerns, including the AR recognition approach, content production and development requirements, and affordances for storytelling, as well as possible contexts and experiences, including indoor/outdoor settings, location-based experiences, mobile guides, and mobile games. We envisage that this work will thoroughly inform AR designers and developers abot the current state of the art and the possibilities and affordances of mobile AR design patterns with respect to particular CH contexts.

Синаптичні процеси гангліозних клітин сітківки ока ссавців

To date, mammalian retinal ganglion cells have been studied quite well, and their diversity, development, and connections have been clarified. Fifty subtypes of these cells have already been identified, each of which optimally encodes unique visual capabilities and projects visual information to various targets in the brain, where it is decoded. Some types of ganglion cells express the photopigment melanopsin (Opn4), which makes them directly sensitive to light. In this review, we discuss the diversity of ganglion cells, their characteristics and differences, and describe the structure of retinal neurons and the synaptic processes that occur in them. Since the quality of human life is seriously affected by retinal diseases that can lead to vision loss, scientists are paying close attention to the causes of visual clarity and quality and the role of retinal ganglion cells in this process, conducting multidirectional research using new methods and different objects for study.