Learning Problems Research Articles

Multimodal Quanvolutional and Convolutional Neural Networks for Multi-Class Image Classification

By utilizing hybrid quantum–classical neural networks (HNNs), this research aims to enhance the efficiency of image classification tasks. HNNs allow us to utilize quantum computing to solve machine learning problems, which can be highly power-efficient and provide significant computation speedup compared to classical operations. This is particularly relevant in sustainable applications where reducing computational resources and energy consumption is crucial. This study explores the feasibility of a novel architecture by leveraging quantum devices as the first layer of the neural network, which proved to be useful for scaling HNNs’ training process. Understanding the role of quanvolutional operations and how they interact with classical neural networks can lead to optimized model architectures that are more efficient and effective for image classification tasks. This research investigates the performance of HNNs across different datasets, including CIFAR100 and Satellite Images of Hurricane Damage by evaluating the performance of HNNs on these datasets in comparison with the performance of reference classical models. By evaluating the scalability of HNNs on diverse datasets, the study provides insights into their applicability across various real-world scenarios, which is essential for building sustainable machine learning solutions that can adapt to different environments. Leveraging transfer learning techniques with pre-trained models such as ResNet, EfficientNet, and VGG16 demonstrates the potential for HNNs to benefit from existing knowledge in classical neural networks. This approach can significantly reduce the computational cost of training HNNs from scratch while still achieving competitive performance. The feasibility study conducted in this research assesses the practicality and viability of deploying HNNs for real-world image classification tasks. By comparing the performance of HNNs with classical reference models like ResNet, EfficientNet, and VGG-16, this study provides evidence of the potential advantages of HNNs in certain scenarios. Overall, the findings of this research contribute to advancing sustainable applications of machine learning by proposing novel techniques, optimizing model architectures, and demonstrating the feasibility of adopting HNNs for real-world image classification problems. These insights can inform the development of more efficient and environmentally friendly machine learning solutions.

Open-world structured sequence learning via dense target encoding

Structured sequences are popularly used to describe graph data with time-evolving node features and edges. A typical real-world scenario of structured sequences is that unknown class labels continuously arrive and thus the training and testing often across different class spaces. This scenario is also referred to as the open-world learning problem on structured sequences. In this paper, we present a new Dense Open-world Structured Sequence Learning model (DOSSL for short) to learn graph streams in the open-world learning setting. To capture both structural and temporal information, DOSSL uses a GNN-based stochastic recurrent neural network for learning node representation in graph streams, then a truncated Laplacian distribution to describe the latent distribution of graph nodes, and a sampling function is used to generate node representations. Further, DOSSL learns dense target embeddings for the known classes to improve the compactness of known class distribution and reserve enough space for open-world unknown classes. The ultimate open-world classifier is optimized to detect the samples from unknown classes under the constraints of DVAE loss, label loss, class uncertainty loss, and dense target loss. Through empirical analysis conducted on real-world datasets, it has been demonstrated that the advanced technique known as DOSSL exhibits the ability to acquire precise node classifiers by harnessing the power of graph streams.

Shuffling-type gradient method with bandwidth-based step sizes for finite-sum optimization

Shuffling-type gradient method is a popular machine learning algorithm that solves finite-sum optimization problems by randomly shuffling samples during iterations. In this paper, we explore the convergence properties of shuffling-type gradient method under mild assumptions. Specifically, we employ the bandwidth-based step size strategy that covers both monotonic and non-monotonic step sizes, thereby providing a unified convergence guarantee in terms of step size. Additionally, we replace the lower bound assumption of the objective function with that of the loss function, thereby eliminating the restrictions on the variance and the second-order moment of stochastic gradient that are difficult to verify in practice. For non-convex objectives, we recover the last iteration convergence of shuffling-type gradient algorithm with a less cumbersome proof. Meanwhile, we also establish the convergence rate for the minimum iteration of gradient norms. Under the Polyak-Łojasiewicz (PL) condition, we prove that the function value of last iteration converges to the lower bound of the objective function. By selecting appropriate boundary functions, we further improve the previous sublinear convergence rate results. Overall, this paper contributes to the understanding of shuffling-type gradient method and its convergence properties, providing insights for optimizing finite-sum problems in machine learning. Finally, numerical experiments demonstrate the efficiency of shuffling-type gradient method with bandwidth-based step size and validate our theoretical results.

Alleviating imbalanced problems of reinforcement learning when applying in real-time power network dispatching and control

Real-time power network dispatching and control (PDC) presents unique challenges that traditional methods cannot effectively address due to the consideration of temporal dynamic factors. Reinforcement learning (RL) has been introduced and proven to be effective. However, given the vast space for solutions, there is an urgent need to enhance the performance of RL further. Analyzing the characteristics of the power network to improve the performance of RL holds significant potential. This article presents a comprehensive analysis of power network characteristics, including imbalanced state distribution and the imbalanced action operation frequency, along with their impact on applying RL in real-time PDC to guide the algorithm design. Guided by the analysis, the paper proposes the Balance Deep Q-Network (DQN) algorithm to mitigate the negative impact of the imbalanced problems on algorithm performance. A K-means-based predefined curriculum learning (KPCL) module is proposed to address the state imbalance issue. It allows for differentiated access to operation scenarios based on their hardness, ensuring a balanced exploration of operation scenarios while maintaining diversity during training. An Option DQN module is proposed to tackle the action imbalance problem, utilizing a branching network structure and hierarchical action selection to reduce the coupling between actions with different usage frequencies, thereby enhancing the accuracy of action value estimation. The effectiveness of the Balance DQN algorithm is demonstrated through extensive experiments conducted on the Grid2Op platform with 14-bus and 36-bus cases. The results indicate that Balance DQN can efficiently alleviate the negative impacts of imbalanced problems on RL for real-time PDC agents and achieve good performance in both cases.

Learning feature relationships in CNN model via relational embedding convolution layer

Establishing the relationships among hierarchical visual attributes of objects in the visual world is crucial for human cognition. The classic convolution neural network (CNN) can successfully extract hierarchical features but ignore the relationships among features, resulting in shortcomings compared to humans in areas like interpretability and domain generalization. Recently, algorithms have introduced feature relationships by external prior knowledge and special auxiliary modules, which have been proven to bring multiple improvements in many computer vision tasks. However, prior knowledge is often difficult to obtain, and auxiliary modules bring additional consumption of computing and storage resources, which limits the flexibility and practicality of the algorithm. In this paper, we aim to drive the CNN model to learn the relationships among hierarchical deep features without prior knowledge and consumption increasing, while enhancing the fundamental performance of some aspects. Firstly, the task of learning the relationships among hierarchical features in CNN is defined and three key problems related to this task are pointed out, including the quantitative metric of connection intensity, the threshold of useless connections, and the updating strategy of relation graph. Secondly, Relational Embedding Convolution (RE-Conv) layer is proposed for the representation of feature relationships in convolution layer, followed by a scheme called use & disuse strategy which aims to address the three problems of feature relation learning. Finally, the improvements brought by the proposed feature relation learning scheme have been demonstrated through numerous experiments, including interpretability, domain generalization, noise robustness, and inference efficiency. In particular, the proposed scheme outperforms many state-of-the-art methods in the domain generalization community and can be seamlessly integrated with existing methods for further improvement. Meanwhile, it maintains comparable precision to the original CNN model while reducing floating point operations (FLOPs) by approximately 50%.

CARBO: Clustering and rotation based oversampling for class imbalance learning

Class imbalance of a data set is a crucial problem in machine learning where one class significantly outnumbers others. In such a data set, classification is a troublesome task for the standard classification algorithms, leading to bias towards the majority class. Different methods have been developed so far, such as oversampling, undersampling, and cost-sensitive learning, to deal with class imbalance circumstances. Among these techniques, oversampling technique does not suffer from the information loss and critical cost selection challenges. However, appropriate synthetic sample generation can be challenging and vulnerable to privacy leakage. This research proposed an oversampling technique, called CARBO, using threshold-based geometric rotation and majority class influenced clustering. Unlike the existing resampling approaches to class imbalance problem, we contribute to consider the data privacy and optimal sample generation together for effective oversampling. The performance of CARBO is evaluated using 44 benchmark imbalanced data set. The empirical analysis elucidates that CARBO can make boosting-based C4.5 ensemble classifiers perform higher for 73% of the data set than six state-of-the-art approaches. In addition, the theoretical compatibility analysis of CARBO demonstrates its robustness.

Social coevolution and Sine chaotic opposition learning Chimp Optimization Algorithm for feature selection

Feature selection is a hot problem in machine learning. Swarm intelligence algorithms play an essential role in feature selection due to their excellent optimisation ability. The Chimp Optimisation Algorithm (CHoA) is a new type of swarm intelligence algorithm. It has quickly won widespread attention in the academic community due to its fast convergence speed and easy implementation. However, CHoA has specific challenges in balancing local and global search, limiting its optimisation accuracy and leading to premature convergence, thus affecting the algorithm’s performance on feature selection tasks. This study proposes Social coevolution and Sine chaotic opposition learning Chimp Optimization Algorithm (SOSCHoA). SOSCHoA enhances inter-population interaction through social coevolution, improving local search. Additionally, it introduces sine chaotic opposition learning to increase population diversity and prevent local optima. Extensive experiments on 12 high-dimensional classification datasets demonstrate that SOSCHoA outperforms existing algorithms in classification accuracy, convergence, and stability. Although SOSCHoA shows advantages in handling high-dimensional datasets, there is room for future research and optimization, particularly concerning feature dimensionality reduction.

Improved brain effective connectivity modelling by dynamic Bayesian networks

Background:If brain effective connectivity network modelling (ECN) could be accurately achieved, early diagnosis of neurodegenerative diseases would be possible. It has been observed in the literature that Dynamic Bayesian Network (DBN) based methods are more successful than others. However, DBNs have not been applied easily and tested much due to computational complexity problems in structure learning. New method:This study introduces an advanced method for modelling brain ECNs using improved discrete DBN (Improved- dDBN) which addresses the computational challenges previously limiting DBN application, offering solutions that enable accurate and fast structure modelling. Results:The practical data and prior sizes needed for the convergence to the globally correct network structure are proved to be much smaller than the theoretical ones using simulated dDBN data. Besides, Hill Climbing is shown to converge to the true structure at a reasonable iteration step size when the appropriate data and prior sizes are used. Finally, importance of data quantization methods are analysed. Comparison with existing methods:The Improved-dDBN method performs better and robust, when compared to the existing methods for realistic scenarios such as varying graph complexity, various input conditions, noise cases and non-stationary connections. The data used in these tests is the simulated fMRI BOLD time series proposed in the literature. Conclusions:Improved-dDBN is a good candidate to be used on real datasets to accelerate developments in brain ECN modelling and neuroscience. Appropriate data and prior sizes can be identified based on the approach proposed in this study for global and fast convergence.

Decision-focused learning under decision dependent uncertainty for power systems with price-responsive demand

Contemporary power systems are experiencing a growing integration of flexible resources on the consumer side. Different from flexible demand that submits specific bids to energy markets, price-responsive demand (PRD) adjusts its power consumption without notice, simply based on the resulting electricity prices as well as internal priorities and limitations. In this paper, we demonstrate how a high penetration of PRD (the behavior of which is invisible to the operator during the day-ahead unit commitment stage) results in systematic inefficiency costs and formulate the so-termed decision-focused learning problem of learning to provide a demand forecast which, once fed as an input to the operator’s economic dispatch optimization problem, results in an efficient dispatch. Interestingly, the prescribed demand forecast affects the resulting prices, which in turn affect the actual demand realization, giving rise to a decision-dependent uncertainty. Motivated by the problem’s hard-to-evaluate objective function, we solve it using Bayesian optimization. The empirical evaluations demonstrate significant savings in the effective real-time system cost, compared to the current practice of using the default demand forecast. Moreover, the method is shown to achieve a system cost that is fairly close to the one achieved by a system that fully integrates PRD into the day-ahead process; but without requiring any change in the operator’s existing dispatch algorithm while avoiding all efforts necessary for the integration of flexible demand, which is a widely pursued field of ongoing research.

Learning With Constraint Learning: New Perspective, Solution Strategy and Various Applications.

The complexity of learning problems, such as Generative Adversarial Network (GAN) and its variants, multi-task and meta-learning, hyper-parameter learning, and a variety of real-world vision applications, demands a deeper understanding of their underlying coupling mechanisms. Existing approaches often address these problems in isolation, lacking a unified perspective that can reveal commonalities and enable effective solutions. Therefore, in this work, we proposed a new framework, named Learning with Constraint Learning (LwCL), that can holistically examine challenges and provide a unified methodology to tackle all the above-mentioned complex learning and vision problems. Specifically, LwCL is designed as a general hierarchical optimization model that captures the essence of these diverse learning and vision problems. Furthermore, we develop a gradient-response based fast solution strategy to overcome optimization challenges of the LwCL framework. Our proposed framework efficiently addresses a wide range of applications in learning and vision, encompassing three categories and nine different problem types. Extensive experiments on synthetic tasks and real-world applications verify the effectiveness of our approach. The LwCL framework offers a comprehensive solution for tackling complex machine learning and computer vision problems, bridging the gap between theory and practice.

Online Learning With Uncertain Feedback Graphs.

Online learning with expert advice is widely used in various machine learning tasks. It considers the problem where a learner chooses one from a set of experts to take advice and make a decision. In many learning problems, experts may be related, henceforth the learner can observe the losses associated with a subset of experts that are related to the chosen one. In this context, the relationship among experts can be captured by a feedback graph, which can be used to assist the learner's decision-making. However, in practice, the nominal feedback graph often entails uncertainties, which renders it impossible to reveal the actual relationship among experts. To cope with this challenge, the present work studies various cases of potential uncertainties and develops novel online learning algorithms to deal with uncertainties while making use of the uncertain feedback graph. The proposed algorithms are proved to enjoy sublinear regret under mild conditions. Experiments on real datasets are presented to demonstrate the effectiveness of the novel algorithms.

Penerapan Model Pembelajaran Role Playing terhadap Keterampilan Berbicara Peserta Didik di SMPN 1 Ampek Angkek

The aim of this research is to describe: 1). Learning problems at SMPN 1 Ampek Angkek 2). Application of the Role Playing model in PAI and Character learning at SMPN 1 Ampek Angkek 3). The impact of the Role Playing model on PAI and Character learning at SMPN 1 Ampek Angkek. The method used in this research is a qualitative library research approach and interviews with teachers at SMPN 1 Ampek Angkek. In this method, data is also collected through sources related to personal models from various literature such as Google Books, journals, repositories and various other literature that supports the discussion of this article. The results of the research show 1). Teachers still use conventional models so that students feel bored when learning takes place 2). The Role Playing model has been implemented well, which focuses more on social interaction and student cohesion 3). The role playing model makes students active in speaking and enthusiastic about learning.

Algorithm-Dependent Generalization of AUPRC Optimization: Theory and Algorithm.

Stochastic optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning. Despite extensive studies on AUPRC optimization, generalization is still an open problem. In this work, we present the first trial in the algorithm-dependent generalization of stochastic AUPRC optimization. The obstacles to our destination are three-fold. First, according to the consistency analysis, the majority of existing stochastic estimators are biased with biased sampling strategies. To address this issue, we propose a stochastic estimator with sampling-rate-invariant consistency and reduce the consistency error by estimating the full-batch scores with score memory. Second, standard techniques for algorithm-dependent generalization analysis cannot be directly applied to listwise losses. To fill this gap, we extend the model stability from instance-wise losses to listwise losses. Third, AUPRC optimization involves a compositional optimization problem, which brings complicated computations. In this work, we propose to reduce the computational complexity by matrix spectral decomposition. Based on these techniques, we derive the first algorithm-dependent generalization bound for AUPRC optimization. Motivated by theoretical results, we propose a generalization-induced learning framework, which improves the AUPRC generalization by equivalently increasing the batch size and the number of valid training examples. Practically, experiments on image retrieval and long-tailed classification speak to the effectiveness and soundness of our framework.

On the Number of Linear Regions of Convolutional Neural Networks With Piecewise Linear Activations.

One fundamental problem in deep learning is understanding the excellent performance of deep Neural Networks (NNs) in practice. An explanation for the superiority of NNs is that they can realize a large family of complicated functions, i.e., they have powerful expressivity. The expressivity of a Neural Network with Piecewise Linear activations (PLNN) can be quantified by the maximal number of linear regions it can separate its input space into. In this paper, we provide several mathematical results needed for studying the linear regions of Convolutional Neural Networks with Piecewise Linear activations (PLCNNs), and use them to derive the maximal and average numbers of linear regions for one-layer PLCNNs. Furthermore, we obtain upper and lower bounds for the number of linear regions of multi-layer PLCNNs. Our results suggest that deeper PLCNNs have more powerful expressivity than shallow PLCNNs, while PLCNNs have more expressivity than fully-connected PLNNs per parameter, in terms of the number of linear regions.

Robust Audio-Visual Contrastive Learning for Proposal-Based Self-Supervised Sound Source Localization in Videos.

By observing a scene and listening to corresponding audio cues, humans can easily recognize where the sound is. To achieve such cross-modal perception on machines, existing methods take advantage of the maps obtained by interpolation operations to localize the sound source. As semantic object-level localization is more attractive for prospective practical applications, we argue that these map-based methods only offer a coarse-grained and indirect description of the sound source. Additionally, these methods utilize a single audio-visual tuple at a time during self-supervised learning, causing the model to lose the crucial chance to reason about the data distribution of large-scale audio-visual samples. Although the introduction of Audio-Visual Contrastive Learning (AVCL) can effectively alleviate this issue, the contrastive set constructed by randomly sampling is based on the assumption that the audio and visual segments from all other videos are not semantically related. Since the resulting contrastive set contains a large number of faulty negatives, we believe that this assumption is rough. In this paper, we advocate a novel proposal-based solution that directly localizes the semantic object-level sound source, without any manual annotations. The Global Response Map (GRM) is incorporated as an unsupervised spatial constraint to filter those instances corresponding to a large number of sound-unrelated regions. As a result, our proposal-based Sound Source Localization (SSL) can be cast into a simpler Multiple Instance Learning (MIL) problem. To overcome the limitation of random sampling in AVCL, we propose a novel Active Contrastive Set Mining (ACSM) to mine the contrastive sets with informative and diverse negatives for robust AVCL. Our approaches achieve state-of-the-art (SOTA) performance when compared to several baselines on multiple SSL datasets with diverse scenarios.

Methods and algorithms of swarm intelligence for the problems of nonlinear regression analysis and optimization of complex processes, objects, and systems: review and modification of methods and algorithms

The development of high-speed methods and algorithms for global multidimensional optimization and their modifications in various fields of science, technology, and economics is an urgent problem that involves reducing computing costs, accelerating, and effectively searching for solutions to such problems. Since most serious problems involve the search for tens, hundreds, or thousands of optimal parameters of mathematical models, the search space for these parameters grows non-linearly. Currently, there are many modern methods and algorithms of swarm intelligence that solve today's scientific and applied problems, but they require modifications due to the large spaces of searching for optimal model parameters. Modern swarm intelligence has significant potential for application in the energy industry due to its ability to optimize and solve complex problems. It can be used to solve scientific and applied problems of optimizing energy consumption in buildings, industrial complexes, and urban systems, reducing energy losses, and increasing the efficiency of resource use, as well as for the construction of various elements of energy systems in general. Well-known methods and algorithms of swarm intelligence are also actively applied to forecast energy production from renewable sources, such as solar and wind energy. This allows better management of energy sources and planning of their use. The relevance of modifications of methods and algorithms is due to the issues of speeding up their work when solving machine learning problems, in particular, in nonlinear regression models, classification, and clustering problems, where the number of observed data can reach tens and hundreds of thousands or more. The work considers and modifies well-known effective methods and algorithms of swarm intelligence (particle swarm optimization algorithm, bee optimization algorithm, differential evolution method) for finding solutions to multidimensional extremal problems with and without restrictions, as well as problems of nonlinear regression analysis. The obtained modifications of the well-known classic effective methods and algorithms of swarm intelligence, which are present in the work, effectively solve complex scientific and applied tasks of designing complex objects and systems. A comparative analysis of methods and algorithms will be conducted in the next study on this topic. Keywords: optimization, swarm intelligence, mathematical modelling, nonlinear regression, complex objects and systems.

Graph-Graph Similarity Network.

Graph learning aims to predict the label for an entire graph. Recently, graph neural network (GNN)-based approaches become an essential strand to learning low-dimensional continuous embeddings of entire graphs for graph label prediction. While GNNs explicitly aggregate the neighborhood information and implicitly capture the topological structure for graph representation, they ignore the relationships among graphs. In this article, we propose a graph-graph (G2G) similarity network to tackle the graph learning problem by constructing a SuperGraph through learning the relationships among graphs. Each node in the SuperGraph represents an input graph, and the weights of edges denote the similarity between graphs. By this means, the graph learning task is then transformed into a classical node label propagation problem. Specifically, we use an adversarial autoencoder to align embeddings of all the graphs to a prior data distribution. After the alignment, we design the G2G similarity network to learn the similarity between graphs, which functions as the adjacency matrix of the SuperGraph. By running node label propagation algorithms on the SuperGraph, we can predict the labels of graphs. Experiments on five widely used classification benchmarks and four public regression benchmarks under a fair setting demonstrate the effectiveness of our method.

Subtask-Aware Representation Learning for Predicting Antibiotic Resistance Gene Properties via Gating-Controlled Mechanism.

The crisis of antibiotic resistance has become a significant global threat to human health. Understanding properties of antibiotic resistance genes (ARGs) is the first step to mitigate this issue. Although many methods have been proposed for predicting properties of ARGs, most of these methods focus only on predicting antibiotic classes, while ignoring other properties of ARGs, such as resistance mechanisms and transferability. However, acquiring all of these properties of ARGs can help researchers gain a more comprehensive understanding of the essence of antibiotic resistance, which will facilitate the development of antibiotics. In this paper, the task of predicting properties of ARGs is modeled as a multi-task learning problem, and an effective subtask-aware representation learning-based framework is proposed accordingly. More specifically, property-specific expert networks and shared expert networks are utilized respectively to learn subtask-specific features for each subtask and shared features among different subtasks. In addition, a gating-controlled mechanism is employed to dynamically allocate weights to subtask-specific semantics and shared semantics obtained respectively from property-specific expert networks and shared expert networks, thus adjusting distinctive contributions of subtask-specific features and shared features to achieve optimal performance for each subtask simultaneously. Extensive experiments are conducted on publicly available data, and experimental results demonstrate the effectiveness of the proposed framework on the task of ARGs properties prediction.

Penerapan Model Inkuiri Terbimbing sebagai Solusi Keaktifan Peserta Didik di SMP N 01 Tiumang Dharmasraya

From this research, the objectives are: 1) Describe the problems in the PAI learning process at SMP N 1 Tiumang. 2) Analyze the application of the guided inquiry model as a solution to PAI and BP learning problems at SMP N 1 Tiumang. 3) Analyze the impact of using the guided inquiry model on student activity. This research method used is a qualitative descriptive approach. Then techniques for collecting data include interviews with teachers in the PAI and BP study fields, observations at school and data obtained from various kinds of literature such as books, journals and other related sources. The results of the research show that: 1) In the learning process in the classroom the teacher is not clear about what type of learning model to use in the classroom. Learning only runs conventionally where the teacher explains the material/information in the textbook. Then the students sat down and most of them did not pay attention to the material read by the teacher. 2) The application of the guided inquiry model in the classroom is divided into 3 activities, namely the opening, core and final activities, namely the closing. In the core activity, students are divided into several groups to work on and discuss assignments that have been given by the teacher previously. 3) The impact of using this guided inquiry learning model can increase the activity of most students in class.

On adversarial robustness and the use of Wasserstein ascent-descent dynamics to enforce it

Abstract We propose iterative algorithms to solve adversarial training problems in a variety of supervised learning settings of interest. Our algorithms, which can be interpreted as suitable ascent-descent dynamics in Wasserstein spaces, take the form of a system of interacting particles. These interacting particle dynamics are shown to converge toward appropriate mean-field limit equations in certain large number of particles regimes. In turn, we prove that, under certain regularity assumptions, these mean-field equations converge, in the large time limit, toward approximate Nash equilibria of the original adversarial learning problems. We present results for non-convex non-concave settings, as well as for non-convex concave ones. Numerical experiments illustrate our results.