Markup Language Research Articles

Adapting Vision-Language Models via Learning to Inject Knowledge.

Pre-trained vision-language models (VLM) such as CLIP, have demonstrated impressive zero-shot performance on various vision tasks. Trained on millions or even billions of image-text pairs, the text encoder has memorized a substantial amount of appearance knowledge. Such knowledge in VLM is usually leveraged by learning specific task-oriented prompts, which may limit its performance in unseen tasks. This paper proposes a new knowledge injection framework to pursue a generalizable adaption of VLM to downstream vision tasks. Instead of learning task-specific prompts, we extract task-agnostic knowledge features, and insert them into features of input images or texts. The fused features hence gain better discriminative capability and robustness to intra-category variances. Those knowledge features are generated by inputting learnable prompt sentences into text encoder of VLM, and extracting its multi-layer features. A new knowledge injection module (KIM) is proposed to refine text features or visual features using knowledge features. This knowledge injection framework enables both modalities to benefit from the rich knowledge memorized in the text encoder. Experiments show that our method outperforms recently proposed methods under few-shot learning, base-to-new classes generalization, cross-dataset transfer, and domain generalization settings. For instance, it outperforms CoOp by 4.5% under the few-shot learning setting, and CoCoOp by 4.4% under the base-to-new classes generalization setting. Our code will be released.

Text to Video using GANs and Diffusion Models

The challenging endeavour of text-to-video creation requires transforming text descriptions into realistic and cohesive videos. This field of study has made substantial progress in recent years, with the development of diffusion models and generative adversarial networks (GANs). This study examines the most modern text-to-video generation models, as well as the various steps involved in text-to-video generation,including temporal coherence, video generation, and text encoding. We additionally emphasise the challenges involved with text-to-video generation, as well as recent advances to overcome these issues. The most frequently used datasets and metrics in this field are also analysed and reviewed

Towards Fast and Accurate Image-Text Retrieval With Self-Supervised Fine-Grained Alignment

Image-text retrieval requires the system to bridge the heterogenous gap between vision and language for accurate retrieval while keeping the network lightweight-enough for efficient retrieval. Existing trade-off solutions mainly study from the view of incorporating cross-modal interactions with the independent-embedding framework or leveraging stronger pre-trained encoders, which still demand time-consuming similarity measurement or heavyweight model structure in the retrieval stage. In this work, we propose an image-text alignment module SelfAlign on top of the independent-embedding framework, which improves the retrieval accuracy while maintains the retrieval efficiency without extra supervision. SelfAlign contains two collaborative sub-modules that force image-text alignment at both the concept level and context level by self-supervised contrastive learning. It doesn't require cross-modal embedding interactions during training while maintaining independent image and text encoders during retrieval. With comparable time cost, SelfAlign consistently boosts the accuracy of state-of-the-art non-pre-training independent-embedding models respectively by 9.1%, 4.2%, and 6.6% in terms of R@sum score on Flickr30K, MS-COCO 1K and MS-COCO 5K datasets. The retrieval accuracy also outperforms most existing interactive-embedding models with orders of magnitude decrease in retrieval time. The source code is available at: https://github.com/Zjamie813/SelfAlign.

The Establishment of Chinese Culture Teaching Resources by Knowledge Graph Applied to Chinese International Education

Abstract With the development of Chinese international education, the construction of Chinese culture teaching resources has become a priority. This study uses knowledge mapping technology to optimize Chinese culture teaching resources in Chinese international education and improve the usability and efficiency of teaching resources. The study adopts a top-down approach to construct a knowledge graph of Chinese culture teaching resources, covering named entity identification, relationship extraction and knowledge meta-extraction. BERT and BiLSTM techniques are used for feature extraction, text encoding, and conditional random field algorithm for knowledge meta-extraction and relationship mining. The experimental results show that the information utilization rate of intelligent retrieval of teaching resources using this knowledge graph is 82.67% and shows good adaptability in applications between different geographical regions. In addition, through comparative experiments, the proposed method in this study performs better than the traditional method in improving the utilization efficiency of teaching resources. This study proves the effectiveness and practicality of knowledge mapping in constructing Chinese culture teaching resources in Chinese international education.

Adversarial Modality Alignment Network for Cross-Modal Molecule Retrieval

The cross-modal molecule retrieval (Text2Mol) task aims to bridge the semantic gap between molecules and natural language descriptions. A solution to this non-trivial problem relies on graph convolutional network (GCN) and cross-modal attention with contrastive learning for reasonable results. However, there exist the following issues: 1) the cross-modal attention mechanism is only in favor of text representations and can not provide helpful information for molecule representations. 2) the GCN-based molecule encoder ignores edge features and the importance of various substructures of a molecule. 3) the retrieval learning loss function is rather simplistic. This paper further investigates the Text2Mol problem and proposes a novel Adversarial Modality Alignment Network (AMAN)-based method to sufficiently learn both description and molecule information. Our method utilizes a SciBERT as a text encoder and a graph transformer network as a molecule encoder to generate multimodal representations. Then an adversarial network is used to align these modalities interactively. Meanwhile, a triplet loss function is leveraged to perform retrieval learning and further enhance the modality alignment. Experiments on the ChEBI-20 dataset show the effectiveness of our AMAN method compared with baselines.

BoostXML: Gradient Boosting for Extreme Multilabel Text Classification With Tail Labels.

Multilabel learning involving hundreds of thousands or even millions of labels is referred to as extreme multilabel learning (XML), in which the labels often follow a power-law distribution with the majority occurring in very few data points as tail labels. Recent years have witnessed the intensive use of deep-learning methods for high-performance XML, but they are typically optimized for the head labels with abundant training instances and less consider the performance on tail labels, which, however, like the needles in haystacks, are often the focus of attention in real-life applications. In light of this, we present BoostXML, a deep learning-based XML method for extreme multilabel text classification, enhanced greatly by gradient boosting. In BoostXML, we pay more attention to tail labels in each Boosting Step by optimizing the residual mostly from unfitted training instances with tail labels. A Corrective Step is further proposed to avoid the mismatching between the text encoder and weak learners during optimization, which reduces the risk of falling into local optima and improves model performance. A Pretraining Step is also introduced in the initial stage of BoostXML to avoid exorbitant bias to tail labels. Extensive experiments on five benchmark datasets with state-of-the-art baselines demonstrate the advantage of BoostXML in tail-label prediction.

A Virtual Laboratory for Control System Engineering

The use of virtual laboratory is becoming increasingly common. These virtual laboratories are beneficial in such situations when practical laboratories are inaccessible or when there is lack of expensive hardware equipment. This paper presents a virtual Control System Laboratory that can be used in Electrical Engineering education for undergraduate students and researchers. A set of experiments have been introduced that have been developed using free-open-source-software (Hyper Text Markup Language 5 (HTML5), Cascading Style Sheets 3 (CSS3), and JavaScript) and uploaded into web for remote access. The case studies of these experiments are presented. This laboratory can be accessed in (http://vlabs.iitkgp.ac.in/psac/newlabs2024/ctrl/).

An innovative Bridge Surveillance Methodological Framework: the case of the A3 Highway (Southern Italy)

The road networks represent one of the main services necessary for the economic and social development of a country. Undoubtedly, the weak points of the network are represented by infrastructures such as bridges and tunnels, in fact, the lack of an overpass could paralyze different areas. With the aim to support the managing companies in a decision-making process, in order to ensure service in full safety, the acquisition of an appropriate knowledge of the current health state and its evolution over time of the overall buildings is necessary. The Italian “Guidelines for Risk Classification and Management, Safety Assessment and Monitoring of Existing Bridges – LG20” presents a multilevel methodology as a management process from on-site survey to the bridges’ condition assessment. In 2022, CUGRI launched an innovative surveillance management model established on a multidisciplinary approach based on Geography Markup Language, BIM tools, on-site interdisciplinary inspections, and multi-hazard analysis. The methodology provides the on-site training of inspectors, the elaboration of suitable BIM models according to the LG20 and AINOP (National Archive of Public Infrastructures) requirements, and an expert judgement process for preliminary assessment and data validation to support the maintenance managing process. The paper briefly illustrates the experimental methodology performed on the A3 highway, which connects Naples to Salerno (Southern Italy), and the preliminary considerations.

HoBACDSL: HoBAC-focused Access Control Domain Specific Language

Access control (AC) in IoT (Internet of Things) systems presents significant challenges. These systems are evolving, heterogeneous, and dynamic, combined with the lack of appropriate tools to specify and design Access Control Policies (ACP). Despite various proposals, complexity persists, especially in restrictive environments like IoT. In this perspective, Higher-Order Attribute-Based Access Control (HoBAC) was proposed as a general AC model that extends Attribute Based AC (ABAC). It allows the design of flexible AC models and policies applicable to IoT and non-IoT systems. The work presented in this paper focuses on addressing the need for tools to support the adoption of HoBAC for IoT systems by proposing HoBACDSL, a Domain Specific Language (DSL). HoBACDSL abstracts the complexities of HoBAC and makes its concepts more accessible. We illustrate how this DSL is concretely used to specify, and generate AC policies in the Extensible Access Control Markup Language (XACML) standard for a Smart Home use case.

X3D-ПРОТОТИПИРОВАНИЕ МОДЕЛЕЙ ТЕЛ МАНИПУЛЯЦИОННЫХ РОБОТОВ

The aim is toinvestigate the possibilities and methods for separate X3D modeling of the structure of manipulation robots and external X3D prototyping of their body models, without requiring knowledge of X3D programming of virtual worlds for their practical use. The research methods are related to 3D modeling, body system kinematics and robotics. Typical angular manipulative robots consisting of a rotating platform, a shoulder, a forearm, and a three-linked hand with a two-fingered grip at the end are considered. The results of the study contain a description of the developed ways of marking up the struc-ture and geometry of manipulation robots and three ways (simple, simple and typical) of creating a digital code, based on which an X3D model of the body is automatically generated. For each of these methods, advantages, disadvantages and limitations are noted, and cases of their effective use are recommended. Practical application of the methods described in detail will be possible for any user who has mastered digital coding of body models. This is especially important for those who do not know declarative xml-programming and in particular the extensible markup language X3D, for the study of which books and articles in Russian are very few. For those who wish to deepen their knowledge of X3D in practice here are presented and described developed X3D and JS-codes for modeling manipulation robots and their bodies. Examples of X3D layout of the structure of typical manipulation robots with six rotational kinematic pairs, which cover most angular industrial robots, are given. Conclusion. The obtained results are used for external X3D prototyping of manipulation robots and their grippers as part of X3D models of robotic technological complexes of inter-operational transportation, painting, welding, assembly in order to simulate the performance of the corresponding technological processes.

Multimodal Machine Translation

In recent years, neural network machine translation, especially in the field of multimodality, has developed rapidly. It has been widely used in natural languages processing tasks such as event detection and sentiment classification. The existing multimodal neural network machine translation is mostly based on the autoencoder framework of the attention mechanism, which further integrates spatial-visual features. However, due to the ubiquitous lack of corpus and the semantic interaction between multimodalities, the quality of machine translation is difficult to guarantee. Therefore, this paper proposes a multi-modal machine translation model that integrates external linguistic knowledge. Specifically, on the encoder side, we adopt the pre-trained Bert model to be used as an additional encoder to integrate with the original text encoder and picture encoder. Under the cooperation of the three encoders, a better text representation and picture representation at the source end is generated. Besides, the decoder decodes and generates a translation based on the image and text representation of the source. To sum up, this paper studies the visual-text semantic interaction on the encoder side and the visual-text semantic interaction on the decoder side, and further improves the quality of translation by introducing external linguistic knowledge. We compared the performance of the multimodal neural network machine translation model with pre-trained Bert and other baseline models in English German translation tasks on the multi30k data sets. The results show that the model can significantly improve the quality of multimodal neural network machine translation, which also verifies the importance of integrating external knowledge and visual text semantic interaction.

DCTP-Net: Dual-Branch CLIP-Enhance Textual Prompt-Aware Network for Acute Ischemic Stroke Lesion Segmentation From CT Image.

Detecting early ischemic lesions (EIL) in computed tomography (CT) images is crucial for reducing diagnostic time and minimizing neuron loss due to oxygen deprivation. This paper introduces DCTP-Net, a dual-branch network for segmenting acute ischemic stroke lesions in CT images, consisting of a segmentation branch and a prompt-aware branch. The segmentation branch uses an encoder-decoder network as the backbone to identify lesions, where the encoder fuses CT image features with prompt features from the prompt-aware branch. To enhance semantic feature extraction and reduce the impact of cerebral structural details, we introduce a cross-collaboration dynamic connection (CCDC) module to link the encoder and decoder. The prompt-aware branch includes a learnable prompt (LP) block to incorporate cerebral prior knowledge, and the prompt-aware encoder (PAE) combines the LP block with multi-level features from the segmentation branch for more precise representation. Additionally, we propose a CLIP-enhance textual prompt (CETP) module that utilizes the CLIP text encoder to generate specialized convolutional parameters for the segmentation head. These parameters are tailored to the unique characteristics of each input image, improving segmentation performance. Qualitative and quantitative studies reveal that DCTP-Net outperforms the current state-of-the-art, IS-Net, with Dice score increases of 3.9% on AISD and 3.8% on ISLES2018, demonstrating its superiority in EIL segmentation.

Prospects and Obstacles on the Path of Chinese’s Rise as a Global Language in the Context of the Internet

Abstract In this paper, the video retrieval technique based on model migration makes full use of the migration learning technique to build a text encoder and video encoder to obtain text and video features. The text is preprocessed based on the BERT model, and text features are extracted by combining TF-IDF and mutual information. The logistic regression algorithm is applied to achieve text classification. The text mining techniques above are used to parse YouTube Chinese learning resources to extract their dissemination features. The results show that 5,286 (61.48%) of YouTube Chinese learning resources are from the category of “education,” and 2,294 (26.68%) are from the category of “people and blogs.” The video resources category is the most popular and loved category among learners due to its highest average number of views (1238214.3). The dissemination is significantly impacted by teachers’ performance, vivid language, and interaction during the teaching process. The indirect effect value test between online interaction, technical support, and resource quality, as well as satisfaction and intention to continue using, met the criterion of significance (p<0.05). It indicates that perceived usefulness has a significant indirect effect (P<0.01) on the impact of knowledge dissemination. This study clarifies the dissemination influences that led to the rise of Chinese resources in the global context.

PEMBANGUNAN LAMAN SESAWANG BAGI PENGURUSAN JAWATANKUASA OBE POLITEKNIK KUCHING SARAWAK DENGAN KERANGKA DJANGO

OBE Committee Management (Outcome Based Learning) must be organized and easily accessible by certain parties such as management and committee members themselves. Among the information that can be obtained is the committee organization chart, committee information, activities carried out as well as data management for the committee. The difficulty of managing activity data is the main reason this website was created. The process of developing this particular website uses the Django web development framework. Website display requires basic knowledge of HTML (Hypertext Markup Language) and CSS (Cascading Style Sheet). The result of the development of the OBE Committee website gives the management the opportunity to access the database of activities carried out by the department's OBE coordinator.

DEVELOPMENT OF A LEARNING MANAGEMENT SYSTEM FOR PHYTON PROGRAMMING LANGUAGE

An Electronic Learning System is an educational platform that takes place over the Internet. Due to its convenience and flexibility, learning through an Electronic Learning Management System (LMS) results in effective learning experiences. The purpose of this research is to design and develop a LMS for programming education. Collaborative, microlearning and scaffolding programming learning approach were incorporated into the design and development of the learning management system for programming language. The software was developed using waterfall model, and evaluated by analyzing user’s feedback. The tools used in building the learning management system are visual studio code, Hyper Text Markup Language (HTML), cascading style sheet, javascript, python, django, Django rest framework, Jason, ajax, git, tinyMCE. The evaluation of the newly implemented online learning platform based on user feedback was overwhelmingly positive. Users appreciated the platform's intuitive interface, which allowed for easy navigation and quick access to resources. The clear and concise instructions provided were also highly commended, as they helped to ensure that users understood the system's various features and capabilities. Users also noted the platform's support for various types of learning content, including text, audio, video, and interactive materials, which helped to create a more engaging and dynamic learning experience. Furthermore, the platform's effective tools for monitoring student progress and managing course completion were praised for their efficiency and ease of use. The platform's support for collaborative learning was also highly valued, with users citing the ability to work together and exchange ideas as a valuable aspect of the platform. In conclusion, the newly implemented online learning platform is a valuable tool that can significantly enhance the learning experience of students. The positive feedback from users suggests that the platform's support for micro-learning and collaborative learning can effectively promote engagement, collaboration, and successful learning outcomes. These findings underscore the importance of designing and implementing effective online learning platforms that support the needs of both students and instructors.

Optimizing Science Question Ranking through Model and Retrieval-Augmented Generation

This paper delves into the challenges of discerning optimal answers from science-based questions generated by large language models (LLM), particularly emphasizing the intricate task of ranking. Employing the MAP@3 evaluation metric and drawing from the OpenBookQA dataset, the study explores modeling strategies and highlights the exceptional performance of the Platypus2-70B model. Equipped with a state-of-the-art text encoder, Platypus2-70B achieves an impressive score of 0.909904, setting a benchmark for excellence in future large language model competitions. The paper goes beyond a mere description of model architectures and experimental results, offering a comprehensive journey that envisions the transformative impact of large-scale language models on the landscape of natural language understanding, especially within the intricate domains of scientific exploration.

Implementation of User-Oriented Geovisualization Web Dashboard for Monitoring Access to Improved Water using Satellite Imageries Data

This study aims to develop an engaging, web-based visualization dashboard for improved water access in Indonesia. The dashboard map was made using three technologies: the Qgis2web Python plugin for producing two-dimensional (2D) dashboard maps, JavaScript leaflets for map visualization, and Hypertext Markup Language (HTML), Cascade Stylesheet (CSS), and JavaScript for the user interface. The built-in map dashboard has several features, including grid click, legend, zoom, search, and measure distance, which are meant to help users determine the location of the nearest water treatment facilities, identify geographical features, and keep track of areas that have poor access to improved water. Evaluation using the system usability scale (SUS) concludes the dashboard is acceptable with an excellent rating. Our results reiterate and enhance support for government institution and relevant stakeholders in providing sustainable access to public water.

Design and Implementation of an Interactive Visualization Dashboard for Monitoring the Flood Vulnerability and Mapping

This study aims to build a web-based interactive visualization dashboard from granular flood vulnerability index estimation maps using data from satellite imagery. The approach used to build this visualization dashboard is a two-dimensional (2D) approach created with the qgis2web python plugin facilitated with a JavaScript leaflet. Raw data from satellite imagery consisting of indicators of the causes of flooding are extracted in comma-separated value (CSV) format. Furthermore, the data is integrated based on its spatial attributes and stored in Geographic JavaScript Object Notation (GeoJSON) format to produce a visualization of the flood vulnerability index map. In web views, dashboards are built by utilizing hypertext markup language (HTML), cascading style sheets (CSS), and JavaScript (JS). This interactive dashboard has several useful features in helping the process of monitoring the flood vulnerability of an area such as zoom, "show me where I am", measure distance, search, legend, and change year. Thus, the flood vulnerability estimation map dashboard is expected to assist the government in monitoring areas with extreme flood vulnerability and support the decision-making process related to mitigation of areas that have high flood vulnerability.

A Geovisualization Dashboard of Granular Food Security Index Map using GIS for Monitoring the Provincial Level Food Security Status

This study aims to build a web-based interactive geovisualization dashboard from a granular food security index map using satellite imagery and other geospatial big data. The map dashboard is built using a two-dimensional (2D) data visualization approach. Making a two-dimensional map using QuantumGIS (QGIS) tools, displayed in the form of WebGIS with the plugin used "Qgis2web" based on javascript leaflets. Once included in WebGIS, interactive visualizations are displayed on websites with interfaces based on hypertext markup language (HTML), cascading style sheets (CSS), and JavaScript (JS). The dashboard map is equipped with interactive features such as legend, click grid, zoom, show me where I am, measure distance, and search. Therefore, the dashboard map can be used to monitor the food security index, search for food security index areas, as well as geographical identification of food security index areas which are useful for supporting the analysis of decision-making or policies by the government regarding food security strategies.

Extending 3D geometric file formats for geospatial applications

This study addresses the representation and exchange of geospatial geometric 3D models, which is a common requirement in various applications like outdoor mixed reality, urban planning, and disaster risk management. Over the years, multiple file formats have been developed to cater to diverse needs, offering a wide range of supported features and target areas of application. However, classic exchange formats like the JavaScript Object Notation and the Extensible Markup Language have been predominantly favored as a basis for exchanging geospatial information, leaving out common geometric information exchange formats such as Wavefront’s OBJ, Stanford’s PLY, and OFF. To bridge this gap, our research proposes three novel extensions for the mentioned geometric file formats, with a primary focus on minimizing storage requirements while effectively representing geospatial data and also allowing to store semantic meta-information. The extensions, named GeoOBJ, GeoOFF, and GeoPLY, offer significant reductions in storage needs, ranging from 14 to 823% less compared to standard file formats, while retaining support for an adequate number of semantic features. Through extensive evaluations, we demonstrate the suitability of these proposed extensions for geospatial information representation, showcasing their efficacy in delivering low storage overheads and seamless incorporation of critical semantic features. The findings underscore the potential of GeoOBJ, GeoOFF, and GeoPLY as viable solutions for efficient geospatial data representation, empowering various applications to operate optimally with minimal storage constraints.