Description Language Research Articles

Image Captioning Based on Semantic Scenes.

With the development of artificial intelligence and deep learning technologies, image captioning has become an important research direction at the intersection of computer vision and natural language processing. The purpose of image captioning is to generate corresponding natural language descriptions by understanding the content of images. This technology has broad application prospects in fields such as image retrieval, autonomous driving, and visual question answering. Currently, many researchers have proposed region-based image captioning methods. These methods generate captions by extracting features from different regions of an image. However, they often rely on local features of the image and overlook the understanding of the overall scene, leading to captions that lack coherence and accuracy when dealing with complex scenes. Additionally, image captioning methods are unable to extract complete semantic information from visual data, which may lead to captions with biases and deficiencies. Due to these reasons, existing methods struggle to generate comprehensive and accurate captions. To fill this gap, we propose the Semantic Scenes Encoder (SSE) for image captioning. It first extracts a scene graph from the image and integrates it into the encoding of the image information. Then, it extracts a semantic graph from the captions and preserves semantic information through a learnable attention mechanism, which we refer to as the dictionary. During the generation of captions, it combines the encoded information of the image and the learned semantic information to generate complete and accurate captions. To verify the effectiveness of the SSE, we tested the model on the MSCOCO dataset. The experimental results show that the SSE improves the overall quality of the captions. The improvement in scores across multiple evaluation metrics further demonstrates that the SSE possesses significant advantages when processing identical images.

The Pedagogical Tipiṭaka: OER & the Three Baskets of Ancient Language Instruction

Ancient languages present a unique teaching challenge: for spoken languages, common pedagogy recommends engaging students via dialogue; for ancient languages, no speakers survive with whom to practise. This paper highlights how the Linguistics Research Center at the University of Texas at Austin has approached this challenge by creating the Early Indo-European OnLine (EIEOL) collection, an online educational resource whose lesson series present early languages directly through original, unsimplified ancient texts. Currently accessed by over 20,000 users per month, EIEOL spans 18 languages, from Greek and Latin to Old Church Slavonic, Sanskrit, and other important languages of ancient Asia such as Hittite, Classical Armenian, Avestan, and Tocharian. Each series presents extensively annotated excerpts of original texts in the target language, with accompanying modules explaining grammar and context. The text-centred approach affords learners a direct path to understanding that suits a variety of experience levels and minimises the conceptual grammatical apparatus necessary to begin interpreting original texts. This format fosters theoretical flexibility, adaptable to different approaches and grammatical descriptions of ancient languages. It is also useful for languages whose grammatical structures have shifted dramatically over their history, like Tocharian, or remain hotly debated or under-described by experts. Finally, it facilitates applications to typologically diverse languages and language families, with early Mesoamerican, Semitic, and Sino-Tibetan language series already under development. The EIEOL infrastructure therefore provides a robust platform for free, text-centred, self-paced introductions to ancient languages from a variety of language families.

Hardware-accelerated neural network model for early prediction of sudden cardiac arrest based on heart rate variability metrics

AbstractSudden Cardiac Arrest (SCA) constitutes a dire medical condition, marked by the abrupt cessation of effective blood circulation due to the heart's failure to contract properly. This leads to acute circulatory collapse, often culminating in loss of consciousness within an hour and potentially resulting in fatality within minutes if left unattended. Heart rate variability (HRV) serves as a critical biometric, derived from electrocardiogram (ECG) signals with ventricular depolarization waves to further calculate the R-R Intervals (RRIs). These intervals provide the basis for extracting various characteristics of cardiac rhythm, encompassing time-domain, frequency-domain, and nonlinear features. This study presents a neural network-based classification algorithm that leverages HRV metrics to categorize patients into SCA and Normal Sinus Rhythm (NSR) cohorts. The proposed neural network (NN) model showcased impressive results, achieving an accuracy of 96.23%, a sensitivity of 94.35%, and a specificity of 98.12% in detecting SCA, as evaluated through leave-one-subject-out analysis. In order to harness the benefits of hardware acceleration, the algorithm is implemented on a Field-Programmable Gate Array (FPGA). Its computational efficiency is subsequently benchmarked against traditional software-based methodologies. The hardware-level implementation is made possible in Verilog hardware description language (HDL) and was verified successfully with expected performance by register-transfer level (RTL) simulation via Vivado 2020.2.

The Connection Between Literary Images and Visual Perception in Meditative Practice

The study of practice based on visualization and the role of visualization in shaping the outcomes of contemplative practice is an overlooked research niche. The aim of this article is to contribute to a deeper understanding of the dynamics of the process that connects literary images and visual perception in relevant meditative practices. The key question is which elements contribute to forming this connection in contemplative practices like meditation. To gain a deeper understanding of the dynamics involved, the study employed a mix of primary and secondary data sources, along with analytical, descriptive, and phenomenological methods, drawing from an interdisciplinary approach that includes psychology, neuroscience, and literature. The following meditation practices were selected: contemplative meditation, creative visualization, koan meditation, imaginative meditation, creative workshops with haiku poetry as an outcome, and meditative storytelling. The identified elements in the connection between literary depictions and visual perception include mental imagery and visualization, cognitive processes, sensory processes, and emotional processes. The importance of literary techniques is particularly highlighted – using descriptive language with metaphors, similes, allegories, and other stylistic figures that create strong visual images. This connection has a neurological basis – neuroscience studies show that reading/listening to texts describing visual experiences activates brain areas involved in actual visual perception. This overlap suggests that the brain processes literary descriptions similarly to how it processes real visual stimuli. The connection between literary depictions and visual perception significantly enhances the quality of meditative practice and promotes deeper understanding and emotional-volitional engagement with the text and personal development of the meditators. Both secondary and primary data sources were used in the paper.

Text-and-Image Learning Transformer for Cross-modal Person Re-identification

Text-based person re-identification aims to find the target person from a large pedestrian gallery with the given natural language description. Previous works mainly focus on embedding salient textual and visual representations in a common latent space by utilizing the dual-path structure or parameter-shared network. However, they still lack the ability to effectively extract fine-grained unimodal features as well as fuse the cross-modal data, leading to the increase of misaligned cases. To settle these issues, we propose a text-and-image implicit learning Transformer (TILT) to eliminate textual anisotropy and enhance the cross-modal alignment from both domains based on the bi-direction multimodal encoders. Specifically, we apply the pre-trained multimodal embedding module to overcome the unimodal anisotropy problem with contrastive learning, and map fine-grained features with dual encoder in bidirectional masking. Then, we design the cross-modal interaction encoder to comprehensively mine implicit cross-modal relations by reconstructing masked tokens, and fuse rich multi-modal knowledge in a common space. In addition, the cross-modal similarity matching module is proposed to optimize the intra-domain classification and decrease the inter-domain divergence. Extensive experiments are conducted on three public benchmarks CUHK-PEDES, ICFG-PEDES and RSTPReid to verify the effectiveness of our proposed framework. And results prove that our model outperforms state-of-the-art methods on all metrics.

Increasing the speed of digital devices using the ASMD-FSMD method using non-blocking operators

The ASMD-FSMD method for designing digital devices is considered, which consists of constructing a block diagram of an algorithmic state machine with a data path (ASMD), which describes the behavior of the device, and creating project code in Verilog in the form of a finite state machine with a data processing path. (finite state machine with datapath – FSMD). One of the directions for the development of the ASMD-FSMD methodology is the use of features of the hardware description language (HDL). A hypothesis has been put forward: in the ASMD-FSMD technique, it is possible to apply several non-blocking assignment operators to the same variable in one synchronization cycle, which will lead to an increase in device performance. The hypothesis put forward was investigated in the design of synchronous multipliers that implement the classical multiplication algorithms c and d. Experimental studies have confirmed the validity of the put forward hypothesis, while the speed of the multipliers increases two to three times, and the cost of implementation in most cases decreases compared to the traditional approach.

Semantic-aware frame-event fusion based pattern recognition via large vision–language models

Pattern recognition through the fusion of RGB frames and Event streams has emerged as a novel research area in recent years. Current methods typically employ backbone networks to individually extract the features of RGB frames and event streams, and subsequently fuse these features for pattern recognition. However, we posit that these methods may suffer from two key issues: (1). They attempt to directly learn a mapping from the input vision modality to the semantic labels. This approach often leads to sub-optimal results due to the disparity between the input and semantic labels; (2). They utilize small-scale backbone networks for the extraction of RGB and Event input features, thus these models fail to harness the recent performance advancements of large-scale visual-language models. In this study, we introduce a novel pattern recognition framework that consolidates the semantic labels, RGB frames, and event streams, leveraging pre-trained large-scale vision–language models. Specifically, given the input RGB frames, event streams, and all the predefined semantic labels, we employ a pre-trained large-scale vision model (CLIP vision encoder) to extract the RGB and event features. To handle the semantic labels, we initially convert them into language descriptions through prompt engineering and polish using ChatGPT, and then obtain the semantic features using the pre-trained large-scale language model (CLIP text encoder). Subsequently, we integrate the RGB/Event features and semantic features using multimodal Transformer networks. The resulting frame and event tokens are further amplified using self-attention layers. Concurrently, we propose to enhance the interactions between text tokens and RGB/Event tokens via cross-attention. Finally, we consolidate all three modalities using self-attention and feed-forward layers for recognition. Comprehensive experiments on the HARDVS and PokerEvent datasets fully substantiate the efficacy of our proposed SAFE model. The source code has been released at https://github.com/Event-AHU/SAFE_LargeVLM.

UART IP core Design and Verification using WISHBONE Interface

The organization of communication between the devices is greatly aided by the communication protocol. These protocols have a clear set of guidelines that have been agreed upon by the tools needed for effective communication. A popular protocol for serial communication is UART. The hardware description language for the SV/Verilog UART functional module is designed in this paper. The serial connection capabilities offered by this UART IP CORE enable interaction with modems and other external devices. The Universal Verification Technique is used to undertake functional verification of the UART. By comparing the collected response to the intended response via the scoreboard mechanism, the reusable UVM test bench architecture is meant to push the randomized stimuli to the unit under test to assess the functional correctness. This core is made to be as compatible as possible with commercially accepted designs. The WISHBONE INTERFACE WITH 8-BIT OR 32-BIT SELECTABLE DATA BUS MODES is one of this design's primary characteristics. Interface debugging in 32-bit data bus mode. Functionality and register-level compatibility. FIFO procedure. The UVM methodology is used to validate the design. For optimal functional coverage, the test bench is written with regression test cases. The overall execution time will be shortened if the stimulus may be reused.

MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools

ABSTRACT The mapping process generally involves intricate operations, such as symbol design, layout design, and text annotation, demanding a high level of professional expertise. The high requirement for map producers hinders the promotion and widespread adoption of mapping. Consequently, researchers are concentrating on techniques to automate and enhance the intelligence of the mapping process. For example, some studies attempt to train deep learning models for mapping, including methods like map style transfer. However, these approaches typically treat the entire map as a global input and generate a new map as output, lacking the flexibility to consider and control detailed elements within a map. Therefore, in this paper, we propose a large language model-based intelligent mapping framework, termed MapGPT, which can be used for mapping by considering the map as an integration of various map elements. Specifically, multiple professional mapping tools are designed in MapGPT, and each tool is designed to control a corresponding map element. With these tools, a large language model is used to first understand the demand of users based on mere natural language descriptions, and subsequently automatically invoke appropriate tools in sequence to generate a map. Furthermore, by utilizing a memory component to store interaction information, users can interact with MapGPT through conversation to adjust map elements such as color and position. In conclusion, MapGPT offers user-friendly mapping experience, showing potential to be a mapping assistant for professional map producers. A comprehensive demonstration of this framework is provided in a visual case study video, accessible at https://github.com/AGI-GIS/MapGPT.

Algorithm-Driven Robotic Discovery of Polyoxometalate-Scaffolding Metal-Organic Frameworks.

The experimental exploration of the chemical space of crystalline materials, especially metal-organic frameworks (MOFs), requires multiparameter control of a large set of reactions, which is unavoidably time-consuming and labor-intensive when performed manually. To accelerate the rate of material discovery while maintaining high reproducibility, we developed a machine learning algorithm integrated with a robotic synthesis platform for closed-loop exploration of the chemical space for polyoxometalate-scaffolding metal-organic frameworks (POMOFs). The eXtreme Gradient Boosting (XGBoost) model was optimized by using updating data obtained from the uncertainty feedback experiments and a multiclass classification extension based on the POMOF classification from their chemical constitution. The digital signatures for the robotic synthesis of POMOFs were represented by the universal chemical description language (χDL) to precisely record the synthetic steps and enhance the reproducibility. Nine novel POMOFs including one with mixed ligands derived from individual ligands through the imidization reaction of POM amine derivatives with various aldehydes have been discovered with a good repeatability. In addition, chemical space maps were plotted based on the XGBoost models whose F1 scores are above 0.8. Furthermore, the electrochemical properties of the synthesized POMOFs indicate superior electron transfer compared to the molecular POMs and the direct effect of the ratio of Zn, the type of ligands used, and the topology structures in POMOFs for modulating electron transfer abilities.

Automating Comment Generation for Smart Contract from Bytecode

Recently, smart contracts have played a vital role in automatic financial and business transactions. To help end users without programming background to better understand the logic of smart contracts, previous studies have proposed models for automatically translating smart contract source code into their corresponding code summaries. However, in practice, only 13% of smart contracts deployed on the Ethereum blockchain are associated with source code. The practical usage of these existing tools is significantly restricted. Considering that bytecode is always necessary when deploying smart contracts, in this paper, we first introduce the task of automatically generating smart contract code summaries from bytecode. We propose a novel approach, named S mart BT ( Smart contract B ytecode T ranslator) for automatically translating smart contract bytecode into fine-grained natural language description directly. Two key challenges are posed for this task: structural code logic hidden in bytecode and the huge semantic gap between bytecode and natural language descriptions. To address the first challenge, we transform bytecode into CFG (Control-Flow Graph) to learn code structural and logic details. Regarding the second challenge, we introduce an information retrieval component to fetch similar comments for filling the semantic gap. Then the structural input and semantic input are used to build an attentional sequence-to-sequence neural network model. The copy mechanism is employed to copy rare words directly from similar comments and the coverage mechanism is employed to eliminate repetitive outputs. The automatic evaluation results show that SmartBT outperforms a set of baselines by a large margin, and the human evaluation results show the effectiveness and potential of SmartBT in producing meaningful and accurate comments for smart contract code from bytecode directly.

Flight Arrival Scheduling via Large Language Model

The flight arrival scheduling problem is one of the critical tasks in air traffic operations, aiming to ensure that the flight arrive in the correct sequence safely. Existing methods primarily focus on the terminal area and often overlook the presence of training flight at the airport. Due to the limited generalization of traditional methods and varying control practices at different airports, training flight at airports still rely on manual control for arrival sorting. To effectively address these issues, we propose a novel method for slot allocation that leverages the strong reasoning capabilities and generalization potential of large language models (LLMs). Our method conceptualizes the dynamic scheduling problem for training flight as a language modeling problem, a perspective not previously explored. Specifically, we represent the allocator’s inputs and outputs as language tokens, utilizing LLMs to generate conflict-free results based on a language description of requested landing information and assigned training flight information. Additionally, we employ a reset strategy to create a small dataset for scenario-specific samples, enabling LLMs to quickly learn allocation schemes from the dataset. We demonstrated the capability of LLMs in addressing time conflicts by evaluating metrics such as answer accuracy, conflict rate, and total delay time (without the wrong answer). These findings underscore the feasibility of employing LLMs in the field of air traffic control.

An early parenting intervention focused on enriched parent-child interactions improves effortful control in the early years of school.

This study examined long-term mediating effects of the smalltalk parenting intervention on children's effortful control at school age (7.5 years; 2016-2018). In 2010-2012, parents (96% female) of toddlers (N = 1201; aged 12-36 months; 52% female) were randomly assigned to either: standard playgroup, smalltalk playgroup (group-only), or smalltalk playgroup with additional home coaching (smalltalk plus). Multi-informant data indicated that smalltalk plus had unique indirect effects on children's effortful control, through parents' capacity to 'maintain and extend' children's focus during joint interactions. Possible mediating pathways via parent verbal responsivity, home learning activities, and descriptive language use were not supported. When parents received a structured playgroup program with additional home coaching, sustainable benefits were evident in children's self-regulation, assessed in the early school years.

Advancing Zebrafish Husbandry: Takeaways From the 2024 Husbandry Workshop and Husbandry Summit.

The 2024 Zebrafish Husbandry Workshop and Summit held during the World Aquaculture Society Meeting in San Antonio focused on key areas for improving zebrafish husbandry research. Discussions highlighted the need for comprehensive literature on husbandry, better communication and collaboration between researchers and facility staff, and the adoption of a standardized reference diet. Current literature lacks comprehensive data and often overlooks crucial factors such as housing density and space requirements for fish development. Collaborative efforts between researchers and facility managers are essential for acquiring accurate husbandry data and minimizing pathogen risks. Standardizing descriptive language and parameter lists in publications and enhancing communication between facilities can improve research quality. Action items proposed include better communication of incoming fish information, standardization of pathogen monitors, transparency in husbandry practices, and fostering a spirit of collaboration among organizations. The summit emphasized the importance of increased PI awareness about husbandry, testing existing standardized diets, forming consortia to oversee diet standardization, creating unified repositories and forums, and conducting evidence-based husbandry studies.

Interactive Surgical Training in Neuroendoscopy: Real-Time Anatomical Feature Localization Using Natural Language Expressions.

This study addresses challenges in surgical education, particularly in neuroendoscopy, where the demand for optimized workflow conflicts with the need for trainees' active participation in surgeries. To overcome these challenges, we propose a framework that accurately identifies anatomical structures within images guided by language descriptions, facilitating authentic and interactive learning experiences in neuroendoscopy. Utilizing the encoder-decoder architecture of a conventional transformer, our framework processes multimodal inputs (images and language descriptions) to identify and localize features in neuroendoscopic images. We curate a dataset from recorded endoscopic third ventriculostomy (ETV) procedures for training and evaluation. Utilizing evaluation metrics, including "R@n," "IoU= θ," "mIoU," and top-1 accuracy, we systematically benchmark our framework against state-of-the-art methodologies. The framework demonstrates excellent generalization, surpassing the compared methods with 93.67 % accuracy and 76.08 % mIoU on unseen data. It also exhibits better computational speed compared with other methods. Qualitative results affirms the framework's effectiveness in precise localization of referred anatomical features within neuroendoscopic images. The framework's adeptness at localizing anatomical features using language descriptions positions it as a valuable tool for integration into future interactive clinical learning systems, enhancing surgical training in neuroendoscopy. The exemplary performance reinforces the framework's potential in enhancing surgical education, leading to improved skills and outcomes for trainees in neuroendoscopy.

Extraction of logical networks during decompiling transistor-level CMOS circuit descriptions

Objectives. The problem of restoring the functional description of digital VLSI devices presented at the transistor level is considered. The objective of the work is to develop means for extraction of blocks representing logical networks from two-level descriptions of CMOS circuits at the transistor level, which were obtained as a result of recognition (extraction) of subcircuits that implement logic elements.Methods. Graph based methods and software tools are proposed for extracting a connected blocks representing a logical network from two-level descriptions of a transistor circuits in SPICE format. In the graph interpretation, the task is reduced to constructing a labeled directed graph of a logical network based on a labeled undirected bipartite graph specifying a two-level description of the transistor circuit.Results. The proposed method makes it possible to identify lexicographically ranked logical networks, from which a transition is made to logical equations that specify the functions implemented at the outputs of the resulting networks. Software tools have been developed that provide the generation of a hierarchical description in SPICE format that implements the original circuit at the transistor level, as well as descriptions of found logical networks in the SF language of hierarchical structural and functional descriptions of discrete devices and in high-level languages (VHDL and Verilog).Conclusion. The developed methods are implemented in C++, included in the program for decompiling transistor CMOS circuits and tested within it on practical examples of transistor-level circuits. The paper provides examples of reverse engineering of some practical transistor circuits.

A Transfer Learning Approach for Arabic Image Captions

Background: Arabic image captioning (AIC) is the automatic generation of text descriptions in the Arabic language for images. Applies a transfer learning approach in deep learning to enhance computer vision and natural language processing. There are many datasets in English reverse other languages. Instead of, the Arabs researchers unanimously agreed that there is a lack of Arabic databases available in this field. Objective: This paper presents the improvement and processing of the available Arabic textual database using Google spreadsheets for translation and creation of AR. Flicker8k2023 dataset is an extension of the Arabic Flicker8k dataset available, it was uploaded to GitHub and made public for researches. Methods: An efficient model proposed using deep learning techniques by including two pre-training models (VGG16 and VGG19), to extract features from the images and build (LSTM and GRU) models to process textual prediction sequence. In addition to the effect of pre-processing the text in Arabic. Results: The adopted model outperforms better compared to the previous study in BLEU-1 from 33 to 40. Conclusions: This paper concluded that the biggest problem is the database available in the Arabic language. This paper has worked to increase the size of the text database from 24,276 to 32,364 thousand captions, where each image contains 4 captions.

An efficient XOR-free implementation of polar encoder for reconfigurable hardware

— This paper presents a novel approach to implementing an XOR-Free architecture of the non-systematic polar encoder (NSPE) for 5G radio. The optimization of XOR logic for hardware (HW) implementation is essential to reduce delay and power consumption. The proposed architecture for NSPE replaces XOR operations with combinational logical patterns, and some redundant patterns are removed with the help of bit manipulation to make it more efficient. The design infers multiplexers (2:1 or 4:1) and inverters as its functional units, making the design adequate and effective in alleviating HW complexity. The XOR-Free encoder performs the same functionality as the XOR-based conventional encoder. We have written a MATLAB script that generates Verilog hardware description language (HDL) code for fully or partially parallel polar encoders tailored to specific code lengths (N) and degrees of parallelism (M). A comparative analysis of various fully and partially parallel encoders with the XOR-Free algorithm is presented. The implementation results show that the proposed architectures are more efficient in terms of HW cost, power consumption, throughput, and latency.

Referring Image Segmentation with Multi-Modal Feature Interaction and Alignment Based on Convolutional Nonlinear Spiking Neural Membrane Systems.

Referring image segmentation aims to accurately align image pixels and text features for object segmentation based on natural language descriptions. This paper proposes NSNPRIS (convolutional nonlinear spiking neural P systems for referring image segmentation), a novel model based on convolutional nonlinear spiking neural P systems. NSNPRIS features NSNPFusion and Language Gate modules to enhance feature interaction during encoding, along with an NSNPDecoder for feature alignment and decoding. Experimental results on RefCOCO, RefCOCO[Formula: see text], and G-Ref datasets demonstrate that NSNPRIS performs better than mainstream methods. Our contributions include advances in the alignment of pixel and textual features and the improvement of segmentation accuracy.

Linguacodus: a synergistic framework for transformative code generation in machine learning pipelines

In the ever-evolving landscape of machine learning, seamless translation of natural language descriptions into executable code remains a formidable challenge. This article introduces Linguacodus, an innovative framework designed to tackle this challenge by deploying a dynamic pipeline that iteratively transforms natural language task descriptions into code through high-level data-shaping instructions. The core of Linguacodus is a fine-tuned large language model, empowered to evaluate diverse solutions for various problems and select the most fitting one for a given task. This article details the fine-tuning process and sheds light on how natural language descriptions can be translated into functional code. Linguacodus represents a substantial leap towards automated code generation, effectively bridging the gap between task descriptions and executable code. It holds great promise for advancing machine learning applications across diverse domains. Additionally, we propose an algorithm capable of transforming a natural description of an ML task into code with minimal human interaction. In extensive experiments on a vast machine learning code dataset originating from Kaggle, we showcase the effectiveness of Linguacodus. The investigations highlight its potential applications across diverse domains, emphasizing its impact on applied machine learning in various scientific fields.