General Learning Research Articles

Harnessing Large Language Models for Transformative Applications in Natural Language Processing

This review article highlights the paradigm shift brought in by LLMS in NLP, coupling their capabilities and applications with related challenges. Architectures like Transformers have given LLMs prominence. GPT, BERT, and PaLM brought tremendous improvement over tasks like text classification, machine translation, and conversational AI, demonstrating almost human-level accuracy in comprehension and generation. These models pave the way for applications in healthcare, finance, and multilingual customer service-disconnected domains made possible by their generalization and transfer learning capabilities. With the Adoption of LLMs, there is also a growing concern for high computational costs, data privacy, and inherent biases, which pose an ethical and environmental challenge. The paper concludes with a gaze into the future of NLP, emphasizing that responsible development practices should be followed to maximize the benefits of LLMs while decreasing their societal and environmental impact.

Dynamical Counterfactual Inference under Time-series Model for Waterflooding Oilfield

The performances of numerical simulation and machine learning in production forecasting are severely dependent on precise geological modeling and high-quality history matching. To address these challenges, causal inference is an effective methodology since it can provide a causality for formalizing causality in history, not statistical dependence. In this paper, to dynamically predict oil production from causality existed in waterflooding oilfield, a dynamical counterfactual inference framework is built to predict oil production. The proposed framework can forecast the oil production under non-observation of engineering factors, i.e., counterfactual, and provide the causal effect of engineering factors impacting on oil production. Meanwhile, combining with the practice exploitation in engineering factor impacting on production, a counterfactual experiment is designed to execute counterfactual prediction. Compared with general machine learning and statistical models, our results not only show better performance in oil production flooding but also guide the specific optimization in improving production, which holds more practical application significance.

Teaching design students machine learning to enhance motivation for learning computational thinking skills

The integration of computational thinking (CT) to enhance creativity in design students has often been underexplored in design education. While design thinking has traditionally been the cornerstone of university design pedagogy and remains essential, the increasing role of digital tools and artificial intelligence in modern design practices presents new opportunities for innovation. By introducing CT alongside design thinking, students can expand their creative toolkit and engage with emerging technologies more effectively. Although many design students may have limited experience with programming, incorporating accessible, no-code tools can help them confidently embrace computational methods, unlocking new pathways for creative exploration and innovation. This study proposes an alternative approach to improve the motivation of design students by introducing machine learning tools into product design processes. We developed an experimental pedagogy in which 56 industrial design university students learned how to use Waikato Environment for Knowledge Analysis (Weka), a machine learning tool, for three hours of design work a week, for a total of eight weeks. Our covariate analysis of data collected in the pretest and posttest shows that the general learning motivations in the group using Weka are significantly higher than those in the group without Weka. However, no significant differences were found between the two groups in terms of learning strategies, collaboration, or critical thinking. Students using Weka spent more time focusing on model training and tended to improve their algorithmic thinking, and the introduction of Weka appeared to enhance their motivation to learn. On the other hand, these students might have been focusing on working individually at their computers, potentially neglecting communication and collaboration. The findings suggest that teaching machine learning applications without requiring coding has the potential to boost design students' motivation to engage with CT skills, though care must be taken to maintain collaborative practices.

The Effectiveness of the Quasi-Class Action Approach: Integration of Problem-Based Learning, Flipped Classrooms, and Generative Learning Models

This study investigates the effectiveness of integrating Problem-Based Learning (PBL), flipped classrooms, and generative learning to enhance student learning outcomes in Indonesia. Employing the Literature Review method, data were gathered from a range of relevant literature evaluating these three learning models. The findings indicate that the combined approach significantly enhances students' critical thinking skills, motivation, and writing abilities compared to traditional teaching methods. Specifically, the incorporation of PBL encourages students to engage in deep learning through real-world problem-solving scenarios. Meanwhile, flipped classrooms allow for more interactive and personalized learning experiences by enabling students to review instructional content at their own pace outside of class. Generative learning, on the other hand, emphasizes the active creation of knowledge, which further strengthens students' comprehension and retention of material. This research contributes to the development of more effective and adaptive educational strategies in Indonesia, and provides a foundation for further exploration in different educational settings. Moreover, the study highlights the potential for these integrative approaches to address the diverse learning needs of students, ultimately fostering a more inclusive and supportive educational environment.

The Use of Artificial Intelligence in Digital Era Learning for Islamic Education Teachers Facing Generation Z

<strong><span lang="EN-US">Abstract:</span></strong><span lang="EN-US"> Start The purpose of this article is to provide Islamic religious education teachers with further options for technological developments by utilizing AI in learning. As well as how teachers can utilize AI to help the teaching and learning process in the classroom. Technology and progress in the digital age are unavoidable and the knowledge gap between generation Z and previous generations regarding technology and information. This article uses a descriptive qualitative method with a netnography approach. Primary sources from websites, scientific journals, literature, online media news related to the research. The data collection method is done by reading, analyzing, and recording various literature materials. The results obtained that, learning Islamic religious education can be integrated with technology effectively. As other sciences can use by integrating AI technology. Classical methods that only use lectures as a technique for delivering learning materials have shortcomings in terms of utilizing technology in today's digital era. Teachers in today's digital era are required to be fluent in using technology. For this reason, the use of AI is important to integrate and adjust to the current generation Z who live in the digital age.</span>

Optimizing data transmission in 6G software defined networks using deep reinforcement learning for next generation of virtual environments

Data transmission of Virtual Reality (VR) plays an important role in delivering a powerful VR experience. This increasing demand on both high bandwidth and low latency. 6G emerging technologies like Software Defined Network (SDN) and resource slicing are acting as promising technologies for addressing the transmission requirements of VR users. Efficient resource management becomes dominant to ensure a satisfactory user experience. The integration of Deep Reinforcement Learning (DRL) allows for dynamic network resource balancing, minimizing communication latency and maximizing data transmission rates wirelessly. Employing slicing techniques further aids in managing distributed resources across the network for different services as enhanced Mobile Broadband (eMBB) and Ultra-Reliable and Low Latency Communications (URLLC). The proposed VR-based SDN system model for 6G cellular networks facilitates centralized administration of resources, enhancing communication between VR users. This innovative solution seeks to contribute to the effective and streamlined resource management essential for VR video transmission in 6G cellular networks. The utilization of Deep Reinforcement Learning (DRL) approaches, is presented as an alternative solution, showcasing significant performance and feature distinctions through comparative results. Our results show that implementing strategies based on DRL leads to a considerable improvement in the resource management process as well as in the achievable data rate and a reduction in the necessary latency in dynamic and large scale networks.

Cerebellar non-invasive stimulation of social and emotional mentalizing: A meta-analysis

Abstract The present meta-analysis investigated the impact of non-invasive stimulation, using transcranial direct current stimulation (tDCS) and transcranial magnetic stimulation (TMS) targeting the posterior cerebellum, on social and emotional mentalizing about others. Prior research has convincingly shown that the posterior cerebellum supports social and emotional cognition. We identified 14 studies targeting the cerebellum with appropriate control conditions (i.e., sham, control site), which exclude general learning effects of the task or placebo effects. The studies included 29 task conditions where stimulation before or during a social or emotional task was applied on healthy samples. The results showed significant evidence that sustained anodal tDCS and TMS generally improved social and emotional performance after stimulation, in comparison with sham or control conditions, with a small effect size. In contrast, cathodal stimulation showed mixed facilitatory and inhibitory results. In addition, short TMS pulses, administered with the aim of interfering with ongoing social or emotional processes, induced a small but consistent inhibitory effect. Control tasks without social or emotional components also showed significant improvement after sustained anodal tDCS and TMS, suggesting that transcranial stimulation of the cerebellum may also improve other functions. This was not the case for short TMS pulses, which did not modulate non-social and non-emotional control tasks. Taken together, this meta-analysis shows that cerebellar neurostimulation confirms a causal role of the cerebellum in socio-emotional cognition, has a small but significant effect on improving socio-emotional skills, and may, therefore, have important clinical applications in pathologies where social and emotional cognition is impaired.

A survey on person and vehicle re‐identification

AbstractPerson/vehicle re‐identification aims to use technologies such as cross‐camera retrieval to associate the same person (same vehicle) in the surveillance videos at different locations, different times, and images captured by different cameras so as to achieve cross‐surveillance image matching, person retrieval and trajectory tracking. It plays an extremely important role in the fields of intelligent security, criminal investigation etc. In recent years, the rapid development of deep learning technology has significantly propelled the advancement of re‐identification (Re‐ID) technology. An increasing number of technical methods have emerged, aiming to enhance Re‐ID performance. This paper summarises four popular research areas in the current field of re‐identification, focusing on the current research hotspots. These areas include the multi‐task learning domain, the generalisation learning domain, the cross‐modality domain, and the optimisation learning domain. Specifically, the paper analyses various challenges faced within these domains and elaborates on different deep learning frameworks and networks that address these challenges. A comparative analysis of re‐identification tasks from various classification perspectives is provided, introducing mainstream research directions and current achievements. Finally, insights into future development trends are presented.

Ablation Study of Diffusion Model with Transformer Backbone for Low-count PET Denoising.

Diffusion models (DM) built from a hierarchy of denoising autoencoders have achieved remarkable progress in image generation, and are increasingly influential in the field of image restoration (IR) tasks. In the meantime, its backbone of autoencoders also evolved from UNet to vision transformer, e.g. Restormer. Therefore, it is important to disentangle the contribution of backbone networks and the additional generative learning scheme. Notably, DM shows varied performance across IR tasks, and the performance of recent advanced transformer-based DM on PET denoising is under-explored. In this study, we further raise an intuitive question, "{if we have a sufficiently powerful backbone, whether DM can be a general add-on generative learning scheme to further boost PET denoising}". Specifically, we investigate one of the best-in-class IR models, i.e., DiffIR, which is a latent DM based on the Restormer backbone. We provide a qualitative and quantitative comparison with UNet, SR3 (UNet+pixel DM), and Restormer, on the 25% low dose 18F-FDG whole-body PET denoising task, aiming to identify the best practices. We trained and tested on 93 and 12 subjects, and each subject has 644 slices. It appears that Restormer outperforms UNet in terms of PSNR and MSE. However, additional latent DM over Restormer does not contribute to better MSE, SSIM, or PSNR in our task, which is even inferior to the conventional UNet. In addition, SR3 with pixel space DM is not stable to synthesize satisfactory results. The results are consistent with the natural image super-resolution tasks, which also suffer from limited spatial information. A possible reason would be the denoising iteration at latent feature space cannot well support detailed structure and texture restoration. This issue is more crucial in the IR tasks taking inputs with limited details, e.g., SR and PET denoising.

Airborne LiDAR Point Cloud Classification Using Ensemble Learning for DEM Generation.

Airborne laser scanning (ALS) point clouds have emerged as a predominant data source for the generation of digital elevation models (DEM) in recent years. Traditionally, the generation of DEM using ALS point clouds involves the steps of point cloud classification or ground point filtering to extract ground points and labor-intensive post-processing to correct the misclassified ground points. The current deep learning techniques leverage the ability of geometric recognition for ground point classification. However, the deep learning classifiers are generally trained using 3D point clouds with simple geometric terrains, which decrease the performance of model inferencing. In this study, a point-based deep learning model with boosting ensemble learning and a set of geometric features as the model inputs is proposed. With the ensemble learning strategy, this study integrates specialized ground point classifiers designed for different terrains to boost classification robustness and accuracy. In experiments, ALS point clouds containing various terrains were used to evaluate the feasibility of the proposed method. The results demonstrated that the proposed method can improve the point cloud classification and the quality of generated DEMs. The classification accuracy and F1 score are improved from 80.9% to 92.2%, and 82.2% to 94.2%, respectively, by using the proposed methods. In addition, the DEM generation error, in terms of mean squared error (RMSE), is reduced from 0.318-1.362 m to 0.273-1.032 m by using the proposed ensemble learning.

Causal Structure Learning for Recommender System

A fundamental challenge of recommender systems (RS) is understanding the causal dynamics underlying users’ decision making. Most existing literature addresses this problem by using causal structures inferred from domain knowledge. However, there are numerous phenomenons where domain knowledge is insufficient, and the causal mechanisms must be learned from the feedback data. Discovering the causal mechanism from RS feedback data is both novel and challenging, since RS itself is a source of intervention that can influence both the users’ exposure and their willingness to interact. Also for this reason, most existing solutions become inappropriate since they require data collected free from any RS. In this article, we first formulate the underlying causal mechanism as a causal structural model and describe CSL4RS , a general causal structure learning framework for RS grounded in the real-world working mechanism. The essence of our approach is to acknowledge the unknown nature of RS intervention. We then derive the learning objective from our framework and utilize an augmented Lagrangian solver for efficient optimization. We conduct both simulation and real-world experiments to demonstrate how our approach compares favorably to existing solutions, together with the empirical analysis from sensitivity and ablation studies.

GDMol: Generative Double-Masking Self-Supervised Learning for Molecular Property Prediction.

Effective molecular feature representation is crucial for drug property prediction. Recent years have seen increased attention on graph neural networks (GNNs) that are pre-trained using self-supervised learning techniques, aiming to overcome the scarcity of labeled data in molecular property prediction. Traditional GNNs in self-supervised molecular property prediction typically perform a single masking operation on the nodes and edges of the input molecular graph, masking only local information and insufficient for thorough self-supervised training. Hence, we propose a model for molecular property prediction based on generative double-masking self-supervised learning, termed as GDMol. This integrates generative learning into the self-supervised learning framework for latent representation, and applies a second round of masking to these latent representations, enabling the model to better capture global information and semantic knowledge of the molecules for a richer, more informative representation, thereby achieving more accurate and robust molecular property prediction. Our experiments on 5 datasets demonstrated superior performance of GDMol in predicting molecular properties across different domains. Moreover, we used the masking operation to traverse through the gradient changes of each node, the magnitude and sign of which reflect the positive and negative contribution respectively of the local structure in the molecule to the prediction outcome. This in-depth interpretative analysis not only enhances the model's interpretability, but also provides more targeted insights and direction for optimizing drug molecules. In summary, this research offers novel insights on improving molecular property prediction tasks, and paves the way for further research on the application of generative learning and self-supervised learning in the field of chemistry.

Decomposed Prototype Learning for Few-Shot Scene Graph Generation

Today's scene graph generation (SGG) models typically require abundant manual annotations to learn new predicate types. Therefore, it is difficult to apply them to real-world applications with massive uncommon predicate categories whose annotations are hard to collect. In this paper, we focus on Few-Shot SGG (FSSGG) , which encourages SGG models to be able to quickly transfer previous knowledge and recognize unseen predicates well with only a few examples. However, current methods for FSSGG are hindered by the high intra-class variance of predicate categories in SGG: On one hand, each predicate category commonly has multiple semantic meanings under different contexts. On the other hand, the visual appearance of relation triplets with the same predicate differs greatly under different subject-object compositions. Such great variance of inputs makes it hard to learn generalizable representation for each predicate category with current few-shot learning (FSL) methods. However, we found that this intra-class variance of predicates is highly related to the composed subjects and objects. To model the intra-class variance of predicates with subject-object context, we propose a novel Decomposed Prototype Learning (DPL) model for FSSGG. Specifically, we first construct a decomposable prototype space to capture diverse semantics and visual patterns of subjects and objects for predicates by decomposing them into multiple prototypes. Afterwards, we integrate these prototypes with different weights to generate query-adaptive predicate representation with more reliable semantics for each query sample. We conduct extensive experiments and compare with various baseline methods to show the effectiveness of our method.

Podcasting as a Tool for Developing Oral Proficiency and Self-regulated Learning Strategies: Insights from Turkish EFL Learners

This study examines the impact of podcasts on high school students' English language learning, focusing on speaking performance and self-regulated learning strategies. The widespread adoption of digital communication technologies has transformed education, necessitating the integration of technology to enhance learning outcomes. Using a qualitative approach, the study investigates the general English learning habits of students, their perceptions of using podcasts to improve speaking skills, and how podcasts influence their ability to self-regulate learning. Findings reveal that students actively engage with English-language media, benefiting from enhanced listening comprehension, vocabulary acquisition, and pronunciation. Recording podcasts fosters confidence and fluency in speaking, while also promoting goal setting, planning, and reflective practices. The study highlights the importance of integrating podcasts into EFL instruction to create a dynamic learning environment that supports students' language proficiency goals. The results contribute to the literature on educational technology and underscore the potential of podcasts to improve language learning experiences.

Quantifying population-level neural tuning functions using Ricker wavelets and the Bayesian bootstrap

BackgroundExperience changes visuo-cortical tuning. In humans, re-tuning has been studied during aversive generalization learning, in which the similarity of generalization stimuli (GSs) with a conditioned threat cue (CS+) is used to quantify tuning functions. Previous work utilized pre-defined tuning shapes (generalization and sharpening patterns). This approach may constrain the ways in which re-tuning can be characterized since the tuning patterns may not match the prototypical functions. New methodThe present study proposes a flexible and data-driven method for precisely quantifying changes in tuning based on the Ricker wavelet function and the Bayesian bootstrap. This method was applied to EEG and psychophysics data from an aversive generalization learning paradigm. ResultsThe Ricker wavelet model fitted the steady-state visual event potentials (ssVEP), alpha-band power, and detection accuracy data well. A Morlet wavelet function was used for comparison and fit the data better in some situations, but was more challenging to interpret. The pattern of re-tuning in the EEG data, predicted by the Ricker model, resembled the shapes of the best fitting a-priori patterns. Comparison with existing methodsAlthough the re-tuning shape modeled by the Ricker function resembled the pre-defined shapes, the Ricker approach led to greater Bayes factors and more interpretable results compared to a-priori models. The Ricker approach was more easily fit and led to more interpretable results than a Morlet wavelet model. ConclusionThis work highlights the promise of the current method for capturing the precise nature of visuo-cortical tuning, unconstrained by the implementation of a-priori models.

SVQ-MAE: an efficient speech pre-training framework with constrained computational resources

Self-supervised learning for speech pre-training models has achieved remarkable success in acquiring superior speech contextual representations by learning from unlabeled audio, excelling in numerous downstream speech tasks. However, the pre-training of these models necessitates significant computational resources and training duration, presenting a high barrier to entry into the realm of pre-training learning. In our efforts, by amalgamating the resource-efficient benefits of the generative learning model, Masked Auto Encoder, with the efficacy of the vector quantization method in discriminative learning, we introduce a novel pre-training framework: Speech Vector Quantization Masked Auto Encoder (SVQ-MAE). Distinct from the majority of SSL frameworks, which require simultaneous construction of speech contextual representations and mask reconstruction within an encoder-only module, we have exclusively designed a decoupled decoder for pre-training SVQ-MAE. This allows the additional decoupled decoder to undertake the mask reconstruction task solely, reducing the learning complexity of pretext tasks and enhancing the encoder’s efficiency in extracting speech contextual representations. Owing to this innovation, by using only 4 GPUs, SVQ-NAE can achieve high performance comparable to wav2vec 2.0, which requires 64 GPUs for training. In the Speech Processing Universal Performance Benchmark, SVQ-MAE surpasses wav2vec 2.0 in both keyword spotting and emotion recognition tasks. Furthermore, in cross-lingual ASR for Mandarin, upon fine-tuning on AISHELL-1, SVQ-MAE achieves a Character Error Rate of 4.09%, outperforming all supervised ASR models.

Feature Selection as Deep Sequential Generative Learning

Feature selection aims to identify the most pattern-discriminative feature subset. In prior literature, filter (e.g., backward elimination) and embedded (e.g., LASSO) methods have hyperparameters (e.g., top- k , score thresholding) and tie to specific models, thus, hard to generalize; wrapper methods search a feature subset in a huge discrete space and is computationally costly. To transform the way of feature selection, we regard a selected feature subset as a selection decision token sequence and reformulate feature selection as a deep sequential generative learning task that distills feature knowledge and generates decision sequences. Our method includes three steps: (1) We develop a deep variational transformer model over a joint of sequential reconstruction, variational, and performance evaluator losses. Our model can distill feature selection knowledge and learn a continuous embedding space to map feature selection decision sequences into embedding vectors associated with utility scores. (2) We leverage the trained feature subset utility evaluator as a gradient provider to guide the identification of the optimal feature subset embedding; (3) We decode the optimal feature subset embedding to autoregressively generate the best feature selection decision sequence with autostop. Extensive experimental results show this generative perspective is effective and generic, without large discrete search space and expert-specific hyperparameters. The code is available at http://tinyurl.com/FSDSGL .

Generative learning for forecasting the dynamics of high-dimensional complex systems

We introduce generative models for accelerating simulations of high-dimensional systems through learning and evolving their effective dynamics. In the proposed Generative Learning of Effective Dynamics (G-LED), instances of high dimensional data are down sampled to a lower dimensional manifold that is evolved through an auto-regressive attention mechanism. In turn, Bayesian diffusion models, that map this low-dimensional manifold onto its corresponding high-dimensional space, operate on batches of physics correlated, time sequences of data and capture the statistics of the system dynamics. We demonstrate the capabilities and drawbacks of G-LED in simulations of several benchmark systems, including the Kuramoto-Sivashinsky (KS) equation, two-dimensional high Reynolds number flow over a backward-facing step, and simulations of three-dimensional turbulent channel flow. The results demonstrate that generative learning offers new frontiers for the accurate forecasting of the statistical properties of high-dimensional systems at a reduced computational cost.

Biomass hydrothermal gasification characteristics study: based on deep learning for data generation and screening strategies

Hydrothermal gasification is an import-ant way to utilize biomass resources, and accurate prediction of the biomass hydrothermal gasification process is of great significance for the formulation and optimization of process parameters and equipment. Data is a key factor in machine learning, but due to the high experimental costs associated with hydrothermal gasification processes, acquiring a large amount of experimental data for machine learning modeling is a major challenge. To address this issue, this study proposes a data generation and screening strategy based on generative adversarial networks (GAN). The data generation and screening strategy primarily rely on GAN networks trained with real data as data generators and random forest models trained with real data as data screeners. High-quality synthetic data is selected through screening criteria to augment the dataset. Four machine learning models are used to model the biomass hydrothermal gasification process based on synthetic data to validate this strategy. The results show that, compared to the original data, modeling with synthetic data leads to a significant increase in the evaluation metrics for predicting H2, CH4, and CO2, especially during the testing phase. This indicates the rationality of the proposed data generation and screening strategy.

General Machine Learning Approaches for Lithium-Ion Battery Capacity Fade Compared to Empirical Models

Today’s growing demand for lithium-ion batteries across various industrial sectors has introduced a new concern: battery aging. This issue necessitates the development of tools and models that can accurately predict battery aging. This study proposes a general framework for constructing battery aging models using machine learning techniques and compares these models with two existing empirical models, including a commercial one. To build the models, the databases produced by EVERLASTING and Bills et al. were utilized. The aim is to create universally applicable models that can address any battery-aging scenario. In this study, three types of models were developed: a vanilla neural network, a neural network inspired by extreme learning machines, and an encoder coupled with a neural network. The inputs for these models are derived from established knowledge in battery science, allowing the models to capture aging effects across different use cases. The models were trained on cells subjected to specific aging conditions and they were tested on other cells from the same database that experienced different aging conditions. The results obtained during the test for the vanilla neural network showed an RMSE of 1.3% on the Bills et al. test data and an RMSE of 2.7% on the EVERLASTING data, demonstrating similar or superior performance compared to the empirical models and proving the ability of the models to capture battery aging.