Scene Description Research Articles

From Detection to Action: A Multimodal AI Framework for Traffic Incident Response

With the rising incidence of traffic accidents and growing environmental concerns, the demand for advanced systems to ensure traffic and environmental safety has become increasingly urgent. This paper introduces an automated highway safety management framework that integrates computer vision and natural language processing for real-time monitoring, analysis, and reporting of traffic incidents. The system not only identifies accidents but also aids in coordinating emergency responses, such as dispatching ambulances, fire services, and police, while simultaneously managing traffic flow. The approach begins with the creation of a diverse highway accident dataset, combining public datasets with drone and CCTV footage. YOLOv11s is retrained on this dataset to enable real-time detection of critical traffic elements and anomalies, such as collisions and fires. A vision–language model (VLM), Moondream2, is employed to generate detailed scene descriptions, which are further refined by a large language model (LLM), GPT 4-Turbo, to produce concise incident reports and actionable suggestions. These reports are automatically sent to relevant authorities, ensuring prompt and effective response. The system’s effectiveness is validated through the analysis of diverse accident videos and zero-shot simulation testing within the Webots environment. The results highlight the potential of combining drone and CCTV imagery with AI-driven methodologies to improve traffic management and enhance public safety. Future work will include refining detection models, expanding dataset diversity, and deploying the framework in real-world scenarios using live drone and CCTV feeds. This study lays the groundwork for scalable and reliable solutions to address critical traffic safety challenges.

Text2NeRF: Text-Driven 3D Scene Generation With Neural Radiance Fields.

Text-driven 3D scene generation is widely applicable to video gaming, film industry, and metaverse applications that have a large demand for 3D scenes. However, existing text-to-3D generation methods are limited to producing 3D objects with simple geometries and dreamlike styles that lack realism. In this work, we present Text2NeRF, which is able to generate a wide range of 3D scenes with complicated geometric structures and high-fidelity textures purely from a text prompt. To this end, we adopt NeRF as the 3D representation and leverage a pre-trained text-to-image diffusion model to constrain the 3D reconstruction of the NeRF to reflect the scene description. Specifically, we employ the diffusion model to infer the text-related image as the content prior and use a monocular depth estimation method to offer the geometric prior. Both content and geometric priors are utilized to update the NeRF model. To guarantee textured and geometric consistency between different views, we introduce a progressive scene inpainting and updating strategy for novel view synthesis of the scene. Our method requires no additional training data but only a natural language description of the scene as the input. Extensive experiments demonstrate that our Text2NeRF outperforms existing methods in producing photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of natural language prompts. Our code and model are available at https://github.com/eckertzhang/Text2NeRF.

An audiovisual cognitive optimization strategy guided by salient object ranking for intelligent visual prothesis systems

Objective.Visual prostheses are effective tools for restoring vision, yet real-world complexities pose ongoing challenges. The progress in AI has led to the emergence of the concept of intelligent visual prosthetics with auditory support, leveraging deep learning to create practical artificial vision perception beyond merely restoring natural sight for the blind.Approach.This study introduces an object-based attention mechanism that simulates human gaze points when observing the external world to descriptions of physical regions. By transforming this mechanism into a ranking problem of salient entity regions, we introduce prior visual attention cues to build a new salient object ranking (SaOR) dataset, and propose a SaOR network aimed at providing depth perception for prosthetic vision. Furthermore, we propose a SaOR-guided image description method to align with human observation patterns, toward providing additional visual information by auditory feedback. Finally, the integration of the two aforementioned algorithms constitutes an audiovisual cognitive optimization strategy for prosthetic vision.Main results.Through conducting psychophysical experiments based on scene description tasks under simulated prosthetic vision, we verify that the SaOR method improves the subjects' performance in terms of object identification and understanding the correlation among objects. Additionally, the cognitive optimization strategy incorporating image description further enhances their prosthetic visual cognition.Significance.This offers valuable technical insights for designing next-generation intelligent visual prostheses and establishes a theoretical groundwork for developing their visual information processing strategies. Code will be made publicly available.

Automated physics-based modeling of construction equipment through data fusion

Physics-based simulations are essential for designing autonomous construction equipment, but preparing models is time-consuming, requiring the integration of mechanical and geometric data. Current automatic modeling methods for modular robots are inadequate for construction equipment. This paper explores automating the modeling process by integrating mechanical data into 3D computer-aided design (CAD) models. A template library is developed with hierarchy and joint templates specific for equipment. During model generation, appropriate templates are selected based on the equipment type. Unspecified joint template data is extracted from technical specifications using a large language model (LLM). The 3D CAD model is then converted into a Universal Scene Description (USD) model. Users can adjust the part names and hierarchy within the USD model to align with the hierarchy template, and joint data is automatically integrated, resulting in a simulation-ready model. This method reduces modeling time by over 87 % compared to manual methods, while maintaining accuracy.

From Single Words to Sentence Production: Shared Cortical Representations but Distinct Temporal Dynamics.

Sentence production is the uniquely human ability to transform complex thoughts into strings of words. Despite the importance of this process, language production research has primarily focused on single words. It remains an untested assumption that insights from this literature generalize to more naturalistic utterances like sentences. Here, we investigate this using high-resolution neurosurgical recordings (ECoG) and an overt production experiment where patients produce six words in isolation (picture naming) and in sentences (scene description). We trained machine learning models to identify the unique brain activity pattern for each word during picture naming, and used these patterns to decode which words patients were processing while they produced sentences. Our findings reveal that words share cortical representations across tasks. In sensorimotor cortex, words were consistently activated in the order in which they were said in the sentence. However, in inferior and middle frontal gyri (IFG and MFG), the order in which words were processed depended on the syntactic structure of the sentence. This dynamic interplay between sentence structure and word processing reveals that sentence production is not simply a sequence of single word production tasks, and highlights a regional division of labor within the language network. Finally, we argue that the dynamics of word processing in prefrontal cortex may impose a subtle pressure on language evolution, explaining why nearly all the world's languages position subjects before objects.

Vision Aid: Developing An Assistive Mobile Application for Visually Impaired Indivuals

Vision Aid is a cutting edge smartphone software that combines several assistive technologies into a unified platform to help visually impaired individuals. Integrating Computer Vision algorithms this project makes use of Bing Maps API for Step by Step navigation, Tesseract OCR for text recognition, YOLO (You Only Look Once) for real time object detection and CLIP (Contrastive Language-Image Pre training) for scene description. It offers voice commands, audio feedback for a more improved day to day life hence improving self-confidence while empowering independence to the users. This journal provides a thorough description of Vision Aid the development process, methodology, evaluation results and shows the potential for future improvements with a goal to advance mobile accessibility for visually impaired individuals

Word2Scene: Efficient remote sensing image scene generation with only one word via hybrid intelligence and low-rank representation

To address the numerous challenges existing in current remote sensing scene generation methods, such as the difficulty in capturing complex interrelations among geographical features and the integration of implicit expert knowledge into generative models, this paper proposes an efficient method for generating remote sensing scenes using hybrid intelligence and low-rank representation, named Word2Scene, which can generate complex scenes with just one word. This approach combines geographic expert knowledge to optimize the remote sensing scene description, enhancing the accuracy and interpretability of the input descriptions. By employing a diffusion model based on hybrid intelligence and low-rank representation techniques, this method endows the diffusion model with the capability to understand remote sensing scene concepts and significantly improves the training efficiency of the diffusion model. This study also introduces the geographic scene holistic perceptual similarity (GSHPS), a novel evaluation metric that holistically assesses the performance of generative models from a global perspective. Experimental results demonstrate that our proposed method outperforms existing state-of-the-art models in terms of remote sensing scene generation quality, efficiency, and realism. Compared to the original diffusion models, LPIPS decreased by 18.52% (from 0.81 to 0.66), and GSHPS increased by 28.57% (from 0.70 to 0.90), validating the effectiveness and advancement of our method. Moreover, Word2Scene is capable of generating remote sensing scenes not present in the training set, showcasing strong zero-shot capabilities. This provides a new perspective and solution for remote sensing image scene generation, with the potential to advance the development of remote sensing, geographic information systems, and related fields. Our code will be released at https://github.com/jaycecd/Word2Scene.

Number Attraction in Pronoun Production.

Pronoun production involves at least two processes: (i) deciding to refer to a referent with a pronoun instead of a full NP and (ii) determining the pronoun's form. In the present study, we assess whether the second of these processes occurs as a by-product of the first process-namely, does accessing the message-level representation of the referent provide access to the features required to determine pronoun form, meaning that pronouns should be robust to errors, or are pronoun features determined through an agreement operation with the antecedent, in which case they may be susceptible to agreement attraction, similar to subject-verb agreement. Prior lab experiments suggest that pronouns display number attraction at a similar rate to verbs. However, in contrast to verb attraction errors, there is no documentation of systematic pronoun attraction errors in corpora of natural speech. Our study builds upon prior lab work by eliciting pronoun sentences using a scene description paradigm that engages the pronominalization processes involved in natural speech. Across three experiments, we observed small but reliable number attraction effects for pronouns, suggesting that pronoun form is not always determined from the message-level representation of the referent. The elicited error rates were smaller than those previously observed for verbs in a similar scene-description paradigm; this smaller error rate helps to reconcile the apparent discrepancy between pronoun number attraction error rates observed in and outside the lab. The results suggest that pronoun form is determined (at least at times) through an agreement process referencing the features of the linguistic antecedent.

The theme of love and beauty in the works by Sergei Moskvitin

The theme of love and beauty is key in the work of Sergei Moskvitin. According to Moskvitin, poetry, beauty and love are the three pillars on which the entire planet rests. At the same time, the concepts of “woman”, “beauty”, “inspiration” are inseparable for the author. Woman is identical with love, the author deifies her, belittling himself. The image of a woman is slightly blurry and not specified in Moskvitin’s works, unlike landscape lyrics, where every detail is carefully described. At the same time, landscape lyrics and love lyrics are inseparable: through the prism of describing nature, one can guess the author’s mood, some milestones in the biography. For example, when a romantic relationship with a lady begins, the description of nature is positive; when disagreements occur, the hero and heroine are tormented by doubts, nature and natural phenomena echo these feelings. The author is no stranger to the sensual side of the relationship between a man and a woman. However, platonic love predominates in creativity The description of intimate scenes is always complemented by the chanting of love, the beloved woman, which fully conveys a respectful attitude towards the heroine. The author emphasizes that women are different, but each is beautiful and worthy of love, its own poet who sings of her beauty. The author appreciates not only external beauty, but more spiritual beauty. Moskvitin's love lyrics are autobiographical; they are expressed in poems and messages to the woman he loves, as evidenced by the conclusions drawn by the researchers

A Survey of Robot Intelligence with Large Language Models

Since the emergence of ChatGPT, research on large language models (LLMs) has actively progressed across various fields. LLMs, pre-trained on vast text datasets, have exhibited exceptional abilities in understanding natural language and planning tasks. These abilities of LLMs are promising in robotics. In general, traditional supervised learning-based robot intelligence systems have a significant lack of adaptability to dynamically changing environments. However, LLMs help a robot intelligence system to improve its generalization ability in dynamic and complex real-world environments. Indeed, findings from ongoing robotics studies indicate that LLMs can significantly improve robots’ behavior planning and execution capabilities. Additionally, vision-language models (VLMs), trained on extensive visual and linguistic data for the vision question answering (VQA) problem, excel at integrating computer vision with natural language processing. VLMs can comprehend visual contexts and execute actions through natural language. They also provide descriptions of scenes in natural language. Several studies have explored the enhancement of robot intelligence using multimodal data, including object recognition and description by VLMs, along with the execution of language-driven commands integrated with visual information. This review paper thoroughly investigates how foundation models such as LLMs and VLMs have been employed to boost robot intelligence. For clarity, the research areas are categorized into five topics: reward design in reinforcement learning, low-level control, high-level planning, manipulation, and scene understanding. This review also summarizes studies that show how foundation models, such as the Eureka model for automating reward function design in reinforcement learning, RT-2 for integrating visual data, language, and robot actions in vision-language-action models, and AutoRT for generating feasible tasks and executing robot behavior policies via LLMs, have improved robot intelligence.

Multimodality-guided Visual-Caption Semantic Enhancement

Video captions generated with single modality, e.g. video clips, often suffer from insufficient event discovery and inadequate scene description. Therefore, this paper aims to improve the quality of captions by addressing these issues through the integration of multi-modal information. Specifically, We first construct a multi-modal dataset and introduce the triplet annotations of video, audio and text, fostering a comprehensive exploration about the associations between different modalities. Build upon this, We propose to explore the collaborative perception of audio and visual concepts to mitigate inaccuracies and incompleteness in captions in vision-based benchmarks by incorporating audio-visual perception priors. To achieve this, we extract effective semantic features from visual and auditory modalities, bridge the semantic gap between audio-visual modalities and text, and form a more precise knowledge graph multimodal coherence checking and information pruning mechanism. Exhaustive experiments demonstrate that the proposed approach surpasses existing methods and generalizes well with the assistance of ChatGPT.

A Synthetic Aperture Radar Imaging Simulation Method for Sea Surface Scenes Combined with Electromagnetic Scattering Characteristics

Synthetic aperture radar (SAR) simulation is a vital tool for planning SAR missions, interpreting SAR images, and extracting valuable information. SAR imaging is essential for analyzing sea scenes, and the accuracy of sea surface and scattering models is crucial for effective SAR simulations. Traditional methods typically employ empirical formulas to fit sea surface scattering, which are not closely aligned with the principles of electromagnetic scattering. This paper introduces a novel approach by constructing multiple sea surface models based on the Pierson–Moskowitz (P-M) sea spectrum, integrated with the stereo wave observation projection (SWOP) expansion function to thoroughly account for the influence of wave fluctuation characteristics on radar scattering. Utilizing the shooting and bouncing ray-physical optics (SBR-PO) method, which adheres to the principles of electromagnetic scattering, this study not only analyzes sea surface scattering characteristics under various sea conditions but also facilitates the computation of scattering coupling between multiple targets. By constructing detailed scattering distribution data, the method achieves high-precision SAR simulation results. The scattering model developed using the SBR-PO method provides a more nuanced description of sea surface scenes compared to traditional methods, achieving an optimal balance between efficiency and accuracy, thus significantly enhancing sea surface SAR imaging simulations.

Феноменология жестокости и преступления в творчестве японского писателя Мураками Рю

Murakami Ryu’s works are characterized by detailed description of violent and cruel scenes, which does not correlate with the realities of Japan. The purpose of this study is an attempt to identify the functions of violence and crime phenomenology in the works of the author. To achieve the goal, the following tasks are solved: consideration of semantic content of violence and justification concept, its necessity for the plot, analysis of a real criminal case that became the basis of the story “In the Park”, determination of its role in the work. In the course of the work it was revealed that through the description of cruelty the author wants to show the reader the need to spill out aggression. A number of social problems were highlighted, which in the writer's opinion deserve the indignation of the Japanese. The study showed that by mentioning real crimes the author expresses his opinion about them, thus emphasizing the social vices of Japan.

Robust Shape Fitting for 3D Scene Abstraction.

Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

A Lightweight Visual Understanding System for Enhanced Assistance to the Visually Impaired Using an Embedded Platform

Visually impaired individuals often face significant challenges in navigating their environments due to limited access to visual information. To address this issue, a portable, cost-effective assistive tool is proposed to operate on a low-power embedded system such as the Jetson Nano. The novelty of this research lies in developing an efficient, lightweight video captioning model within constrained resources to ensure its compatibility with embedded platforms. This research aims to enhance the autonomy and accessibility of visually impaired people by providing audio descriptions of their surroundings through the processing of live-streaming videos. The proposed system utilizes two distinct lightweight deep learning modules: an object detection module based on the state-of-the-art YOLOv7 model, and a video captioning module that utilizes both the Video Swin Transformer and 2D-CNN for feature extraction, along with the Transformer network for caption generation. The goal of the object detection module is for providing real-time multiple object identification in the surrounding environment of the blind while the video captioning module is to provide detailed descriptions of the entire visual scenes and activities including objects, actions, and relationships between them. The user interacts via a headphone with the proposed system using a specific audio command to trigger the corresponding module even object detection or video captioning and receiving an audio description output for the visual contents. The system demonstrates satisfactory results, achieving inference speeds between 0.11 to 1.1 seconds for object detection and 0.91 to 1.85 seconds for video captioning, evaluated through both quantitative metrics and subjective assessments.

A Multidimensional Interpretation of A New Story of the Stone Based on Corpus Analysis

As a traditional science fiction novel, A New Story of the Stone blends old and new elements. This study employs quantitative descriptions to reveal that the novel's language is rich, vivid, and concise. In terms of narrative features, the novel inherits the traditional Chinese chapter-novel structure and maintains a strong focus on scene descriptions. However, it also reflects Western influences by paying attention to the portrayal of characters' psychological activities. The novel's social utility is pronounced, as it celebrates the dissemination of advanced technology, pursues the ideal society, reflects on societal improvements, and critiques the current society and backward traditional culture.

Automatic Generation Algorithm of Movie and TV Scripts Based on ChatGPT

A movie or TV script is a meticulously crafted document that serves as the blueprint for a visual storytelling project. It outlines the dialogue, actions, and scene descriptions that will unfold on screen, guiding actors, directors, and crew members in bringing the story to life. Each script is divided into scenes and acts, with clear instructions on character entrances and exits, camera angles, and pacing. This paper introduces a novel framework for enhancing the quality of movie and TV scripts through the integration of the Combinational Multi-Stage Genetic Optimization (CMSGO) model with ChatGPT, a state-of-the-art language generation model. The CMSGO model utilizes iterative optimization techniques to systematically refine and enhance script elements such as coherence, dialogue flow, character development, and overall narrative structure. The proposed CMSGO model comprises the Combinational model with the genetic optimization function. The function CMSGO model examines the fitness function with the Multi-Stage Optimization process. The proposed CMSGO model uses the estimation of features in the Multi-stage optimization model with the computation of features related to the scripts. Through 20 generations of optimization, the CMSGO model demonstrates its effectiveness in improving script quality, as evidenced by a steady increase in average script quality scores. Additionally, the multi-stage optimization approach targets specific aspects of script quality, allowing for targeted adjustments to parameters related to character motivations, plot coherence, and tone. Viewer opinions further validate the efficacy of the generated scripts, with positive evaluations across various aspects such as audience engagement, coherence, emotional impact, and originality. The proposed framework offers a robust and data-driven approach to scriptwriting, enabling the creation of high-quality movie and TV scripts that captivate and resonate with audiences, thus enriching the overall viewing experience.

Leveraging the Apple Ecosystem: Easy Viewing and Sharing of Three-dimensional Perforator Visualizations via iPad/iPhone-based Augmented Reality

Summary: We introduce a novel technique using augmented reality (AR) on smartphones and tablets, making it possible for surgeons to review perforator anatomy in three dimensions on the go. Autologous breast reconstruction with abdominal flaps remains challenging due to the highly variable anatomy of the deep inferior epigastric artery. Computed tomography angiography has mitigated some but not all challenges. Previously, volume rendering and different headsets were used to enable better three-dimensional (3D) review for surgeons. However, surgeons have been dependent on others to provide 3D imaging data. Leveraging the ubiquity of Apple devices, our approach permits surgeons to review 3D models of deep inferior epigastric artery anatomy segmented from abdominal computed tomography angiography directly on their iPhone/iPad. Segmentation can be performed in common radiology software. The models are converted to the universal scene description zipped format, which allows immediate use on Apple devices without third-party software. They can be easily shared using secure, Health Insurance Portability and Accountability Act–compliant sharing services already provided by most hospitals. Surgeons can simply open the file on their mobile device to explore the images in 3D using “object mode” natively without additional applications or can switch to AR mode to pin the model in their real-world surroundings for intuitive exploration. We believe patient-specific 3D anatomy models are a powerful tool for intuitive understanding and communication of complex perforator anatomy and would be a valuable addition in routine clinical practice and education. Using this one-click solution on existing devices that is simple to implement, we hope to streamline the adoption of AR models by plastic surgeons.

Facilitating self-directed language learning in real-life scene description tasks with automated evaluation

Engaging children in describing real-life scenes provides an effective approach to fostering language production and developing their language skills, enabling them to establish meaningful connections between their language proficiency and authentic contexts. However, for such learning tasks, there has been a lack of research focusing on promoting self-directed language learning by using artificial intelligence techniques, primarily due to the challenges of handling multimodal information involved in such tasks. To address this gap, this study introduced a two-stage automated evaluation method that employed emerging cross-modal matching AI techniques. Firstly, an automated scoring model was developed to evaluate the quality of students' responses to scene description tasks. Compared with manually assigned human scores, our model scored students' descriptions accurately, as evidenced by a small testing mean absolute error of 0.3969 for a total score of 10 points. Based on the scoring results, immediate feedback was then provided to students by generating targeted comments and suggestions. The goal of this feedback was to assist students in progressively improving their descriptions of daily-life scenes, thereby enabling them to practice their language skills independently. To assess the effectiveness of the feedback, a comprehensive investigation was conducted involving 157 students from middle schools in China, and both qualitative and quantitative experimental data were collected from the students. It is found that the quality of students' descriptions was improved significantly with the assistance of immediate feedback. On average, students achieved an increase of 1.48 points in their scores after making revisions based on the feedback. In addition, students reported positive learning experiences and expressed favorable opinions regarding the language learning tasks with the automated evaluation. The findings of this study have significant implications for future research and educational practice. They not only highlighted the potential of emerging cross-modal matching AI techniques in automatically evaluating learning tasks involving multimodal data but also suggested that providing immediate targeted feedback based on automated scoring results can effectively promote students’ self-directed language learning.

Генрогенные ситуации (на примере болтовни)

The article deals with genrogenic situations (GgS), i.e. situations marked by speech genres, indicative of the representation of a particular speech genre in a given national culture. The G gS as a cognitive image corresponding to a speech genre in the human mind includes, on the one hand, a set of components of a communication situation, necessary and sufficient for the generation of this genre, and, on the other hand, the memory of the GgS, real or reconstructed (by writers, psychologists), in which this genre was generated and developed. These factors determine the theory and practice of speech genres methodology based on GgS: it, unlike the theory of speech genres in general, has a more distinct applied focus: namely, it is focused on GgS and the compilation of an anthology/encyclopedia of GgS. This technique includes two aspects: firstly, a model according to which GgS are isolated from the speech continuum and systematized (GgS as a model of article structure); secondly, a model according to which speech genres are studied using the material of the GgS and taking into account their characteristics (model of filling the «body» of articles with the material). Much attention is paid to reflection (by communication specialists, word artists, journalists, bloggers) on G gS and genres themselves. The article also considers more general aspects of the GgS, their place in speech genres theory and linguistic pragmatics in general: GgS and the integral description of speech genres, GgS and genre role scenes, G gS and speech genre primary-secondary nature, GgS and the relationship between invariant and prototype approaches to the study of text and genre, GgS and speech genre variantology. As an illustration of the discussed approach the author offers GgS for speech genre boltovnya ‘≈chatter’.