Computer Natural Language Research Articles

TL-CStrans Net: a vision robot for table tennis player action recognition driven via CS-Transformer

Currently, the application of robotics technology in sports training and competitions is rapidly increasing. Traditional methods mainly rely on image or video data, neglecting the effective utilization of textual information. To address this issue, we propose: TL-CStrans Net: A vision robot for table tennis player action recognition driven via CS-Transformer. This is a multimodal approach that combines CS-Transformer, CLIP, and transfer learning techniques to effectively integrate visual and textual information. Firstly, we employ the CS-Transformer model as the neural computing backbone. By utilizing the CS-Transformer, we can effectively process visual information extracted from table tennis game scenes, enabling accurate stroke recognition. Then, we introduce the CLIP model, which combines computer vision and natural language processing. CLIP allows us to jointly learn representations of images and text, thereby aligning the visual and textual modalities. Finally, to reduce training and computational requirements, we leverage pre-trained CS-Transformer and CLIP models through transfer learning, which have already acquired knowledge from relevant domains, and apply them to table tennis stroke recognition tasks. Experimental results demonstrate the outstanding performance of TL-CStrans Net in table tennis stroke recognition. Our research is of significant importance in promoting the application of multimodal robotics technology in the field of sports and bridging the gap between neural computing, computer vision, and neuroscience.

Image Captioning Based on Semantic Scenes.

With the development of artificial intelligence and deep learning technologies, image captioning has become an important research direction at the intersection of computer vision and natural language processing. The purpose of image captioning is to generate corresponding natural language descriptions by understanding the content of images. This technology has broad application prospects in fields such as image retrieval, autonomous driving, and visual question answering. Currently, many researchers have proposed region-based image captioning methods. These methods generate captions by extracting features from different regions of an image. However, they often rely on local features of the image and overlook the understanding of the overall scene, leading to captions that lack coherence and accuracy when dealing with complex scenes. Additionally, image captioning methods are unable to extract complete semantic information from visual data, which may lead to captions with biases and deficiencies. Due to these reasons, existing methods struggle to generate comprehensive and accurate captions. To fill this gap, we propose the Semantic Scenes Encoder (SSE) for image captioning. It first extracts a scene graph from the image and integrates it into the encoding of the image information. Then, it extracts a semantic graph from the captions and preserves semantic information through a learnable attention mechanism, which we refer to as the dictionary. During the generation of captions, it combines the encoded information of the image and the learned semantic information to generate complete and accurate captions. To verify the effectiveness of the SSE, we tested the model on the MSCOCO dataset. The experimental results show that the SSE improves the overall quality of the captions. The improvement in scores across multiple evaluation metrics further demonstrates that the SSE possesses significant advantages when processing identical images.

Multi-Task Diffusion Learning for Time Series Classification

Current deep learning models for time series often face challenges with generalizability in scenarios characterized by limited samples or inadequately labeled data. By tapping into the robust generative capabilities of diffusion models, which have shown success in computer vision and natural language processing, we see potential for improving the adaptability of deep learning models. However, the specific application of diffusion models in generating samples for time series classification tasks remains underexplored. To bridge this gap, we introduce the MDGPS model, which incorporates multi-task diffusion learning and gradient-free patch search (MDGPS). Our methodology aims to bolster the generalizability of time series classification models confronted with restricted labeled samples. The multi-task diffusion learning module integrates frequency-domain classification with random masked patches diffusion learning, leveraging frequency-domain feature representations and patch observation distributions to improve the discriminative properties of generated samples. Furthermore, a gradient-free patch search module, utilizing the particle swarm optimization algorithm, refines time series for specific samples through a pre-trained multi-task diffusion model. This process aims to reduce classification errors caused by random patch masking. The experimental results on four time series datasets show that the proposed MDGPS model consistently surpasses other methods, achieving the highest classification accuracy and F1-score across all datasets: 95.81%, 87.64%, 82.31%, and 100% in accuracy; and 95.21%, 82.32%, 78.57%, and 100% in F1-Score for Epilepsy, FD-B, Gesture, and EMG, respectively. In addition, evaluations in a reinforcement learning scenario confirm MDGPS’s superior performance. Ablation and visualization experiments further validate the effectiveness of its individual components.

Modular Co-attention Networks in Nepali Visual Question Answering Systems

Visual question answering (VQA) has been regarded as a challenging task requiring a perfect blend of computer vision and natural language processing. As no dataset was available to train such a model for the Nepali language, a new dataset was developed during the research by translating the VQAv2 dataset. Then the dataset consisting of 202,577 images and 886,560 questions was used to train an attention-based VQA model. The dataset consists of yes/no, counting, and other questions with primarily one-word answers. Modular Co-attention Network (MCAN) was applied to the visual features extracted using the Faster RCNN framework and question embeddings extracted using the Nepali GloVe model. After co-attending the visual and language features for a few cascaded MCAN layers, the features are fused to train the whole network. During evaluation, an overall accuracy of 69.87% was obtained with 81.09% accuracy in yes/no type questions. The results surpassed the performance of models developed for Hindi and Bengali languages. Overall, novel research has been done in the Nepali Language VQA domain paving the way for further advancements.

Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects.

Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks. The most prominent advantage of SSL is that it reduces the dependence on labeled data. Based on the pre-training and fine-tuning strategy, even a small amount of labeled data can achieve high performance. Compared with many published self-supervised surveys on computer vision and natural language processing, a comprehensive survey for time series SSL is still missing. To fill this gap, we review current state-of-the-art SSL methods for time series data in this article. To this end, we first comprehensively review existing surveys related to SSL and time series, and then provide a new taxonomy of existing time series SSL methods by summarizing them from three perspectives: generative-based, contrastive-based, and adversarial-based. These methods are further divided into ten subcategories with detailed reviews and discussions about their key intuitions, main frameworks, advantages and disadvantages. To facilitate the experiments and validation of time series SSL methods, we also summarize datasets commonly used in time series forecasting, classification, anomaly detection, and clustering tasks. Finally, we present the future directions of SSL for time series analysis.

A Transfer Learning Approach for Arabic Image Captions

Background: Arabic image captioning (AIC) is the automatic generation of text descriptions in the Arabic language for images. Applies a transfer learning approach in deep learning to enhance computer vision and natural language processing. There are many datasets in English reverse other languages. Instead of, the Arabs researchers unanimously agreed that there is a lack of Arabic databases available in this field. Objective: This paper presents the improvement and processing of the available Arabic textual database using Google spreadsheets for translation and creation of AR. Flicker8k2023 dataset is an extension of the Arabic Flicker8k dataset available, it was uploaded to GitHub and made public for researches. Methods: An efficient model proposed using deep learning techniques by including two pre-training models (VGG16 and VGG19), to extract features from the images and build (LSTM and GRU) models to process textual prediction sequence. In addition to the effect of pre-processing the text in Arabic. Results: The adopted model outperforms better compared to the previous study in BLEU-1 from 33 to 40. Conclusions: This paper concluded that the biggest problem is the database available in the Arabic language. This paper has worked to increase the size of the text database from 24,276 to 32,364 thousand captions, where each image contains 4 captions.

Comparative evaluation of interpretation methods in surface-based age prediction for neonates

Significant changes in brain morphology occur during the third trimester of gestation. The capability of deep learning in leveraging these morphological features has enhanced the accuracy of brain age predictions for this critical period. Yet, the opaque nature of deep learning techniques, often described as "black box" approaches, limits their interpretability, posing challenges in clinical applications. Traditional interpretable methods developed for computer vision and natural language processing may not directly translate to the distinct demands of neuroimaging. In response, our research evaluates the effectiveness and adaptability of two interpretative methods—regional age prediction and the perturbation-based saliency map approach—for predicting the brain age of neonates. Analyzing 664 T1 MRI scans with the NEOCIVET pipeline to extract brain surface and cortical features, we assess how these methods illuminate key brain regions for age prediction, focusing on technical analysis with clinical insight. Through a comparative analysis of the saliency index (SI) with relative brain age (RBA) and the examination of structural covariance networks, we uncover the saliency index's enhanced ability to pinpoint regions vital for accurate indication of clinical factors. Our results highlight the advantages of perturbation techniques in addressing the complexities of medical data, steering clinical interventions for premature neonates towards more personalized and interpretable approaches. This study not only reveals the promise of these methods in complex medical scenarios but also offers a blueprint for implementing more interpretable and clinically relevant deep learning models in healthcare settings.

SPMGAE: Self-purified masked graph autoencoders release robust expression power

To tackle the scarcity of labeled graph data, graph self-supervised learning (SSL) has branched into two paradigms: Generative methods and Contrastive methods. Inspired by MAE and BERT in computer vision (CV) and natural language processing (NLP), masked graph autoencoders (MGAEs) are gaining popularity in the generative genre. However, prevailing MGAEs are mostly designed under the assumption that the data has high homophilic score and is out of adversarial distortion. When people deliberately improve the performance on homophilic graph datasets, they ignore a critical issue that both internal heterophily and artificial attack noise are quite common in the real world. Therefore, when data itself is highly heterophilic or confronted with attacks, they merely have no defensive capability. Especially under self-supervised conditions, it is much more difficult to detect internal heterophily and resist artificial attacks. In this paper, we propose a Self-Purified Masked Graph Autoencoder (SPMGAE) to make up for the shortcomings of prevailing MGAEs in terms of robustness. SPMGAE first utilizes a self-purified module to prune raw graph data and separate perturbation information. The purified graph provides a robust graph structure for the entire pre-training process. Next, the encoding module reuses perturbation information for auxiliary training to enhance robustness, while the decoding module reconstructs the effective graph data at a finer granularity. Extensive experiments on homophilic and heterophilic datasets attacked by various attack methods demonstrate SPMGAE has a considerable robust expressive ability. Especially on small datasets with large perturbations, the improvement of defensive performance could reaches 10%–25%.

A comprehensive survey on contrastive learning

Contrastive Learning is self-supervised representation learning by training a model to differentiate between similar and dissimilar samples. It has been shown to be effective and has gained significant attention in various computer vision and natural language processing tasks. In this paper, we comprehensively and systematically sort out the main ideas, recent developments and application areas of contrastive learning. Specifically, we firstly provide an overview of the research activity of contrastive learning in recent years. Secondly, we describe the basic principles and summarize a universal framework of contrastive learning. Thirdly, we further introduce and discuss the latest advances of each functional component in detail, including data augmentation, positive/negative samples,network structure, and loss function. Finally, we summarize contrastive learning and discuss the challenges, future research trends and development directions in the area of contrastive learning.

AI-Powered Telemedicine for Automatic Scoring of Neuromuscular Examinations.

Telemedicine is now being used more frequently to evaluate patients with myasthenia gravis (MG). Assessing this condition involves clinical outcome measures, such as the standardized MG-ADL scale or the more complex MG-CE score obtained during clinical exams. However, human subjectivity limits the reliability of these examinations. We propose a set of AI-powered digital tools to improve scoring efficiency and quality using computer vision, deep learning, and natural language processing. This paper focuses on automating a standard telemedicine video by segmenting it into clips corresponding to the MG-CE assessment. This AI-powered solution offers a quantitative assessment of neurological deficits, improving upon subjective evaluations prone to examiner variability. It has the potential to enhance efficiency, patient participation in MG clinical trials, and broader applicability to various neurological diseases.

Distributed hybrid quantum-classical performance prediction for hyperparameter optimization

Hyperparameter optimization (HPO) of neural networks is a computationally expensive procedure, which requires a large number of different model configurations to be trained. To reduce such costs, this work presents a distributed, hybrid workflow, that runs the training of the neural networks on multiple graphics processing units (GPUs) on a classical supercomputer, while predicting the configurations’ performance with quantum-trained support vector regression (QT-SVR) on a quantum annealer (QA). The workflow is shown to run on up to 50 GPUs and a QA at the same time, completely automating the communication between the classical and the quantum systems. The approach is evaluated extensively on several benchmarking datasets from the computer vision (CV), high-energy physics (HEP), and natural language processing (NLP) domains. Empirical results show that resource costs for performing HPO can be reduced by up to 9% when using the hybrid workflow with performance prediction, compared to using a plain HPO algorithm without performance prediction. Additionally, the workflow obtains similar and in some cases even better accuracy of the final hyperparameter configuration, when combining multiple heuristically obtained predictions from the QA, compared to using just a single classically obtained prediction. The results highlight the potential of hybrid quantum-classical machine learning algorithms. The workflow code is made available open-source to foster adoption in the community.

Inheriting the glory of PāṆinian grammar into computer science

the integration of Pāṇinian grammar principles into computer science represents a significant advancement in computational linguistics and natural language processing. Pāṇini, an ancient Indian grammarian, formulated an elaborate system of grammar known as Aṣṭādhyāyī around the 4th century BCE, which has been revered for its systematic and concise rules governing Sanskrit language structure. In recent years, there has been a growing interest in adapting Pāṇinian principles to enhance computational frameworks for understanding human languages. This paper explores the theoretical foundations of Pāṇinian grammar and its applicability in the context of modern computational linguistics. The integration of Paninian grammar into computer science enhances the accuracy and efficiency of natural language processing system and contributes to preservation and dissemination of ancient linguistic knowledge. Through this the researchers aim to bridge gap between traditional linguistic scholarship and modern computational methodologies. By embracing the systematic and rigorous framework laid out by Pāṇini millennia ago, researchers are poised to achieve new breakthroughs in understanding and processing human language through innovative computational techniques. The journey from ancient Sanskrit grammar to innovative computer science exemplifies a harmonious blend of tradition and innovation in the quest for linguistic knowledge and technological advancement. Keywords - Pāṇini, Aṣṭādhyāyī, Computational linguistics, Machine translation, semantic analysis, syntactic rules.

VADS: Visuo-Adaptive DualStrike attack on visual question answer

Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. The adversarial vulnerability of VQA models is crucial for their reliability in real-world applications. However, current VQA attacks are mainly focused on the white-box and transfer-based settings, which require the attacker to have full or partial prior knowledge of victim VQA models. Besides that, query-based VQA attacks require a massive amount of query times, which the victim model may detect. In this paper, we propose the Visuo-Adaptive DualStrike (VADS) attack, a novel adversarial attack method combining transfer-based and query-based strategies to exploit vulnerabilities in VQA systems. Unlike current VQA attacks focusing on either approach, VADS leverages a momentum-like ensemble method to search potential attack targets and compress the perturbation. After that, our method employs a query-based strategy to dynamically adjust the weight of perturbation per surrogate model. We evaluate the effectiveness of VADS across 8 VQA models and two datasets. The results demonstrate that VADS outperforms existing adversarial techniques in both efficiency and success rate. Our code is available at: https://github.com/stevenzhang9577/VADS.

Unveiling the Potential of ChatGPT and YOLOv7 for Evaluating Children's Emotions Using Their Artistic Expressions.

Recent advancements in large language models (LLMs) have sparked considerable interest in their potential applications across various healthcare domains. One promising prospect is leveraging these generative models to accurately predict children's emotions by combining computer vision and natural language processing techniques. However, understanding children's emotional states based on their artistic expressions is equally crucial. To address this challenge, this paper presents a pipelined architecture comprising YOLOv7 and the powerful GPT-3.5 Turbo language model, where YOLOv7 is employed for object detection using art therapy imaging annotations, while GPT-3.5 interprets the sketches. After rigorously evaluating the proposed framework through a series of comprehensive experiments, we observed that our model achieved high confidence scores for both object detection and emotion interpretation. The robust performance of the proposed framework not only aids in explaining children's art but also provides valuable insights for parents and therapists. This capability enables them to better understand children's emotional states based on their artistic expressions, ultimately facilitating improved support and care.

DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation

Automatic radiology report generation is a task that combines artificial intelligence and medical information processing, and it fully relies on computer vision and natural language processing techniques. Nowadays, automatic radiology report generation is still a very challenging task because it requires semantically adequate alignment of data from two modalities: radiology images and text. Existing approaches tend to focus on coarse-grained alignment at the global level and do not take into account the disease characteristics of radiology images at fine-grained semantics, which results in the generated reports potentially omitting key disease diagnostic descriptions. In this work, we propose a new approach, disease-knowledge-enhanced fine-grained image–text alignment for automatic radiology report generation (DKA-RG). The method combines global and disease-level alignment, thus facilitating the extraction of fine-grained disease features by the model. Our approach also introduces a knowledge graph to inject medical domain expertise into the model. Our proposed DKA-RG consists of two training steps: the image–report alignment stage and the image-to-report generation stage. In the alignment stage, we use global contrastive learning to align images and texts from a high level and also augment disease contrastive learning with medical knowledge to enhance the disease detection capability. In the report generation stage, the report text generated from the images is more accurate in describing the disease information thanks to sufficient alignment. Through extensive quantitative and qualitative experiments on two widely used datasets, we validate the effectiveness of our DKA-RG on the task of radiology report generation. Our DKA-RG achieves superior performance on multiple types of metrics (natural language generation and clinical efficacy metrics) compared to existing methods, demonstrating that the method can improve the reliability and accuracy of automatic radiology report generation systems.

Attribute-Driven Filtering: A new attributes predicting approach for fine-grained image captioning

Fine-grained image captioning with attribute information has garnered significant attention in the realms of computer vision and natural language processing, demanding precise and contextually relevant descriptions of visual content. While previous attribute-driven image captioning models have shown improvements, challenges remain, such as the independence of attribute predictors and caption generators and the semantic gap between images and attributes. Another common issue is the inclusion of all attributes at every time step, despite most attributes being irrelevant to the word currently being generated. This can divert the model’s attention toward erroneous semantic details, resulting in a performance decline. To address these issues, we propose a novel Attribute-Driven Filtering (ADF) captioning network designed to provide rich and nuanced descriptions. This model incorporates a unique Attribute Predictor Module (APM) that dynamically predicts the most pertinent attributes in accordance with the textual context, utilizing different attributes at various time steps. The novelty of this approach lies in recognizing that not all attributes hold equal relevance at each time step, and the APM filters out irrelevant attributes to generate precise and contextually relevant captions. Furthermore, this model features a fusion mechanism that integrates visual information from a conventional attention module with attribute information predicted by the APM, aiming to reduce the visual semantic gap between images and attributes. Extensive experimentation demonstrates that the ADF model outperforms advanced models, achieving impressive CIDEr-D scores of 72.0 (Flickr30K) and 123.3 (MS-COCO) through reinforcement learning optimization. It consistently surpasses baseline models across diverse evaluation metrics, highlighting its effectiveness and robustness.

Participatory Visual Methods and Artificial Intelligence-Driven Analysis for Sustainable Consumption Insights

Sustainable consumption is crucial for mitigating global sustainability challenges. Understanding consumer behaviors and motivations, particularly in developing regions, is essential for designing effective interventions. This study pioneers an innovative methodology integrating participatory visual methods (photovoice) and artificial intelligence analysis to investigate food waste perceptions in an emerging economy context. Twenty-six university students participated in the study, documenting their lived experiences and perspectives on household food waste through photographs and narratives. The key results included 32% of participants expressing shock at the extent of food waste in their daily lives, while 28% showed relative indifference. AI-powered (Artificial Intelligence) computer vision and natural language processing were used to efficiently analyze the large visual and textual dataset. The mixed methods approach generated nuanced, situated insights into consumer attitudes, behaviors, and socio-cultural drivers of wastage. The key themes included low waste consciousness, aesthetic and convenience motivations, social norms, and infrastructural limitations. The participatory process proved effective for raising critical consciousness and uncovering consumption practice dynamics. AI analysis enabled rapid knowledge discovery from the qualitative data while mitigating researcher bias. This innovative integration of participatory methodologies and computational analytics advances sustainable consumption research by empowering marginalized voices and generating contextual insights from unstructured data. With further development, such human-centered AI approaches can transform the study and governance of sustainable consumption.

Genome analysis through image processing with deep learning models.

Genomic sequences are traditionally represented as strings of characters: A (adenine), C (cytosine), G (guanine), and T (thymine). However, an alternative approach involves depicting sequence-related information through image representations, such as Chaos Game Representation (CGR) and read pileup images. With rapid advancements in deep learning (DL) methods within computer vision and natural language processing, there is growing interest in applying image-based DL methods to genomic sequence analysis. These methods involve encoding genomic information as images or integrating spatial information from images into the analytical process. In this review, we summarize three typical applications that use image processing with DL models for genome analysis. We examine the utilization and advantages of these image-based approaches.

A Survey on Image Caption Generation in Various Languages

The image caption is the process of adding an explicit, coherent description to the contents of the image. This is done by using the latest deep learning techniques, which include computer vision and natural language processing, to understand the contents of the image and give it an appropriate caption. Multiple datasets suitable for many applications have been proposed. The biggest challenge for researchers with natural language processing is that the datasets are incompatible with all languages. The researchers worked on translating the most famous English data sets with Google Translate to understand the content of the images in their mother tongue. In this paper, the proposed review aims to enhance the understanding of image captioning strategies and to survey previous research related to image captioning while examining the most popular databases in different languages, mostly English, translating into other languages using the latest models for describing images, summarizing evaluation measures, and comparing them.

An Image-Retrieval Method Based on Cross-Hardware Platform Features

Artificial intelligence (AI) models have already achieved great success in fields such as computer vision and natural language processing. However, deploying AI models based on heterogeneous hardware is difficult to ensure accuracy consistency, especially for precision sensitive feature-based image retrieval. In this article, we realize an image-retrieval method based on cross-hardware platform features, aiming to prove that the features of heterogeneous hardware platforms can be mixed, in which the Huawei Atlas 300V and NVIDIA TeslaT4 are used for experiments. First, we compared the decoding differences of heterogeneous hardware, and used CPU software decoding to help hardware decoding improve the decoding success rate. Then, we compared the difference between the Atlas 300V and TeslaT4 chip architectures and tested the differences between the two platform features by calculating feature similarity. In addition, the scaling mode in the pre-processing process was also compared to further analyze the factors affecting feature consistency. Next, the consistency of capture and correlation based on video structure were verified. Finally, the experimental results reveal that the feature results from the TeslaT4 and Atlas 300V can be mixed for image retrieval based on cross-hardware platform features. Consequently, cross-platform image retrieval with low error is realized. Specifically, compared with the Atlas 300V hard and CPU soft decoding, the TeslaT4 hard decoded more than 99% of the image with a decoding pixel maximum difference of +1/−1. From the average of feature similarity, the feature similarity between the Atlas 300V and TeslaT4 exceeds 99%. The difference between the TeslaT4 and Atlas 300V in recall and mAP in feature retrieval is less than 0.1%.