Low Sample Complexity Research Articles

Proximal evolutionary strategy: improving deep reinforcement learning through evolutionary policy optimization

Evolutionary Algorithms (EAs), including Evolutionary Strategies (ES) and Genetic Algorithms (GAs), have been widely accepted as competitive alternatives to Policy Gradient techniques for Deep Reinforcement Learning (DRL). However, they remain eclipsed by cutting-edge DRL algorithms in terms of time efficiency, sample complexity, and learning effectiveness. In this paper, aiming at advancing evolutionary DRL research, we develop an evolutionary policy optimization algorithm with three key technical improvements. First, we design an efficient layer-wise strategy for training DNNs through Covariance Matrix Adaptation Evolutionary Strategies (CMA-ES) in a highly scalable manner. Second, we establish a surrogate model based on proximal performance lower bound for fitness evaluations with low sample complexity. Third, we embed a gradient-based local search technique within the evolutionary policy optimization process to further improve the learning effectiveness. The three technical innovations jointly forge a new EA for DRL method named Proximal Evolutionary Strategies (PES). Our experiments on ten continuous control problems show that PES with layer-wise training can be more computationally efficient than CMA-ES; our surrogate model can remarkably reduce the sample complexity of PES in comparison to latest EAs for DRL including CMA-ES, OpenAI-ES, and Uber-GA; PES with gradient-based local search can significantly outperform several promising DRL algorithms including TRPO, AKCTR, PPO, OpenAI-ES, and Uber-GA.

What Are the Rules? Discovering Constraints from Data

Constraint programming and AI planning are powerful tools for solving assignment, optimization, and scheduling problems. They require, however, the rarely available combination of domain knowledge and mathematical modeling expertise. Learning constraints from exemplary solutions can close this gap and alleviate the effort of modeling. Existing approaches either require extensive user interaction, need exemplary invalid solutions that must be generated by experts at great expense, or show high noise-sensitivity. We aim to find constraints from potentially noisy solutions, without the need of user interaction. To this end, we formalize the problem in terms of the Minimum Description Length (MDL) principle, by which we select the model with the best lossless compression of the data. Solving the problem involves model counting, which is #P-hard to approximate. We therefore propose the greedy URPILS algorithm to find high-quality constraints in practice. Extensive experiments on constraint programming and AI planning benchmark data show URPILS not only finds more accurate and succinct constraints, but also is more robust to noise, and has lower sample complexity than the state of the art.

A modified least squares-based tomography with density matrix perturbation and linear entropy consideration along with performance analysis

Quantum state tomography identifies target quantum states by performing repetitive measurements on identical copies. In this paper, we have two key contributions aimed at improving traditional post-processing computational complexity and sample complexity of quantum tomography protocols. In the first case, we propose a new low-cost positivity constraint method based on density matrix perturbation after the least squares (LS) estimation of the density matrix. In the second case, we propose a new cost function with the maximum linear entropy and LS method to improve the sample average trace distance with reasonably low sample complexity. We call it the LS with the maximum entropy (LSME) method. Our proposed algorithm does not follow the iterative optimization technique, which is true for existing maximum likelihood and entropy-based ones. Performance analysis is conducted for our proposed methods by studying how they compare to the existing techniques for different sample complexities and dimensionalities. Extensive numerical simulations have been conducted to demonstrate the advantages of the proposed tomography algorithms.

Exploring proteoforms of the IgG2 monoclonal antibody panitumumab using microchip capillary electrophoresis-mass spectrometry

The IgG2 type monoclonal antibody panitumumab is an anti-epidermal growth factor receptor (EGFR) drug used for the treatment of EGFR-expressing, chemotherapy resistant, metastatic colorectal carcinoma. In this study, panitumumab drug product was first analysed using size exclusion chromatography coupled to mass spectrometry for rapid identity testing. The experimental data led to the identification of two panitumumab isoforms with several prominent forms remaining unidentified, despite apparently low sample complexity. Microchip capillary electrophoresis-mass spectrometry (CE-MS) was subsequently utilised for a more detailed characterization. It was observed that panitumumab is subject to partial N-terminal pyroglutamate formation. Incomplete conversion is uncharacteristic for N-terminally exposed glutamines and in case of panitumumab gives rise to forms which show successive mass offsets of 17 Da, respectively. If not separated before mass spectrometric analysis, e.g. by capillary electrophoresis, such near isobaric species coalesce into single MS peaks, which subsequently hampers or prevents their assignment. With a total of 42 panitumumab isoforms assigned by CE-MS, these observations highlight a potential pitfall of commonly applied rapid identity testing workflows and demonstrate that even low complexity biopharmaceuticals can require separation strategies which offer high separation selectivity for species close in mass.

High-accuracy model-based reinforcement learning, a survey

Deep reinforcement learning has shown remarkable success in the past few years. Highly complex sequential decision making problems from game playing and robotics have been solved with deep model-free methods. Unfortunately, the sample complexity of model-free methods is often high. Model-based reinforcement learning, in contrast, can reduce the number of environment samples, by learning an explicit internal model of the environment dynamics. However, achieving good model accuracy in high dimensional problems is challenging. In recent years, a diverse landscape of model-based methods has been introduced to improve model accuracy, using methods such as probabilistic inference, model-predictive control, latent models, and end-to-end learning and planning. Some of these methods succeed in achieving high accuracy at low sample complexity in typical benchmark applications. In this paper, we survey these methods; we explain how they work and what their strengths and weaknesses are. We conclude with a research agenda for future work to make the methods more robust and applicable to a wider range of applications.

Minimum-Variance and Low-Complexity Data-Driven Probabilistic Safe Control Design

This paper presents a direct minimum-variance (MV) data-driven safe control design approach for uncertain linear discrete-time stochastic systems. The superiority of the direct MV approach is shown by developing and comparing direct versus indirect learning approaches and MV versus certainty-equivalence (CE) approaches. First, it is shown that probabilistic safety can be guaranteed by ensuring probabilistic λ-contractivity of the safe set. Four data-based convex optimization-based algorithms are introduced to ensure the probabilistic λ-contractivity of the safe set (i.e., direct and indirect CE, direct and indirect MV). It is shown that while the CE approach results in a risk-neutral control design method with no robustness guarantees, the MV approach results in a risk-averse control design with probabilistic safety guarantees. This is because MV approach aims at learning a control gain that minimizes the variance of the state of the closed-loop system with respect to the safe set, and thus minimizes the risk of safety violation. Besides, it is shown that the direct learning approach requires weaker data richness conditions (i.e., lower sample complexity) than the indirect learning approach. Two simulation examples are provided to verify that the direct MV learning approach outperforms the other three approaches since it leads to low-complexity (i.e., low sample complexity and convex optimization) safe learning with a high probability of safety guarantees.

Applicable and Partial Learning of Graph Topology Without Sparsity Priors

This paper considers the problem of learning the underlying graph topology of Gaussian Graphical Models (GGMs) from observations. Under high-dimensional settings, to achieve low sample complexity, many existing graph topology learning algorithms assume structural constraints such as sparsity to hold. Without prior knowledge of graph sparsity, the correctness of their results is difficult to check. In this paper, we aim to do away with these assumptions by developing algorithms for learning degree-bounded GGMs and separable GGMs without any sparsity priors. The proposed algorithms, which are based only on the knowledge of conditional independence relations in the data distribution, require minimal structural assumptions while still achieving low sample complexity, and hence are ‘applicable’. Specifically, for any user defined sparsity parameter <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> , we prove that the proposed algorithms can consistently identify whether a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> -dimensional GGM is degree-bounded by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> (or strongly <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -separable) with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Omega (k \log p)$</tex-math></inline-formula> sample complexity. Besides, our algorithms also demonstrate ‘partial’ learning properties whenever the overall graph is not entirely sparse, that is, not all nodes are degree-bounded (or are strongly separable). In this case, we can still learn the sparse portions of the graph, with theoretical guarantees included. Numerical results show that existing algorithms fail even in some simple settings where sparsity assumptions do not hold, whereas our algorithms do not.

Adaptive Discretization in Online Reinforcement Learning

Adaptive Discretization in Reinforcement Learning Performance guarantees for RL algorithms are typically for worst case instances, which are pathological by design and not observed in meaningful applications. Moreover, many domains (such as computer systems and networking applications) have large state-action spaces and require algorithms to execute with low latency. This phenomenon highlights a trifecta of goals for practical RL algorithms: low sample, storage, and computational complexity. In this work, we develop an algorithmic framework for nonparametric RL with data-driven adaptive discretization. Our framework has provably better sample, storage, and computational complexity than uniform discretization or kernel regression methods. Moreover, we highlight how the performance guarantees are min-max optimal with respect to a novel instance-specific complexity measure that captures structure in facility location and newsvendor models.

Identifying the Acoustic Source via MFF-ResNet with Low Sample Complexity

Acoustic signal classification plays a central role in acoustic source identification. In practical applications, however, varieties of training data are typically inadequate, which leads to a low sample complexity. Applying classical deep learning methods to identify acoustic signals involves a large number of parameters in the classification model, which calls for great sample complexity. Therefore, low sample complexity modeling is one of the most important issues related to the performance of the acoustic signal classification. In this study, the authors propose a novel data fusion model named MFF-ResNet, in which manual design features and deep representation of log-Mel spectrogram features are fused with bi-level attention. The proposed approach involves an amount of prior human knowledge as implicit regularization, thus leading to an interpretable and low sample complexity model of the acoustic signal classification. The experimental results suggested that MFF-ResNet is capable of accurate acoustic signal classification with fewer training samples.

Quantum circuit architectures via quantum observable Markov decision process planning

Algorithms for designing quantum circuit architectures are important steps toward practical quantum computing technology. Applying agent-based artificial intelligence methods for quantum circuit design could improve the efficiency of quantum circuits. We propose a quantum observable Markov decision process planning algorithm for quantum circuit design. Our algorithm does not require state tomography, and hence has low readout sample complexity. Numerical simulations for entangled states preparation and energy minimization are demonstrated. The results show that the proposed method can be used to design quantum circuits to prepare the state and to minimize the energy.

Discovering Interpretable Data-to-Sequence Generators

We study the problem of predicting an event sequence given some meta data. In particular, we are interested in learning easily interpretable models that can accurately generate a sequence based on an attribute vector. To this end, we propose to learn a sparse event-flow graph over the training sequences, and statistically robust rules that use meta data to determine which paths to follow. We formalize the problem in terms of the Minimum Description Length (MDL) principle, by which we identify the best model as the one that compresses the data best. As the resulting optimization problem is NP-hard, we propose the efficient ConSequence algorithm to discover good event-flow graphs from data. Through an extensive set of experiments including a case study, we show that it ably discovers compact, interpretable and accurate models for the generation and prediction of event sequences from data, has a low sample complexity, and is particularly robust against noise.

Efficient Experimental Verification of Quantum Gates with Local Operations.

Verifying the correct functioning of quantum gates is a crucial step toward reliable quantum information processing, but it becomes an overwhelming challenge as the system size grows due to the dimensionality curse. Recent theoretical breakthroughs show that it is possible to verify various important quantum gates with the optimal sample complexity of O(1/ε) using local operations only, where ε is the estimation precision. In this Letter, we propose a variant of quantum gate verification (QGV) that is robust to practical gate imperfections and experimentally realize efficient QGV on a 2-qubit controlled-not gate and a 3-qubit Toffoli gate using only local state preparations and measurements. The experimental results show that, by using only 1600 and 2600 measurements on average, we can verify with 95% confidence level that the implemented controlled-not gate and Toffoli gate have fidelities of at least 99% and 97%, respectively. Demonstrating the superior low sample complexity and experimental feasibility of QGV, our work promises a solution to the dimensionality curse in verifying large quantum devices in the quantum era.

Improving Generative Moment Matching Networks with Distribution Partition

Generative moment matching networks (GMMN) present a theoretically sound approach to learning deep generative mod-els. However, such methods are typically limited by the high sample complexity, thereby impractical in generating complex data. In this paper, we present a new strategy to train GMMN with a low sample complexity while retaining the theoretical soundness. Our method introduces some auxiliary variables, whose values are provided by a pre-trained model such as an encoder network in practice. Conditioned on these variables, we partition the distribution into a set of conditional distributions, which can be effectively matched with a low sample complexity. We instantiate this strategy by presenting an amortized network called GMMN-DP with shared auxiliary variable information for the data generation task, as well as developing an efficient stochastic training algorithm.The experimental results show that GMMN-DP can generate complex samples on datasets such as CelebA and CIFAR-10, where the vanilla GMMN fails.

TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL

Transferring knowledge among various environments is important for efficiently learning multiple tasks online. Most existing methods directly use the previously learned models or previously learned optimal policies to learn new tasks. However, these methods may be inefficient when the underlying models or optimal policies are substantially different across tasks. In this paper, we propose Template Learning (TempLe), a PAC-MDP method for multi-task reinforcement learning that could be applied to tasks with varying state/action space without prior knowledge of inter-task mappings. TempLe gains sample efficiency by extracting similarities of the transition dynamics across tasks even when their underlying models or optimal policies have limited commonalities. We present two algorithms for an ``online'' and a ``finite-model'' setting respectively. We prove that our proposed TempLe algorithms achieve much lower sample complexity than single-task learners or state-of-the-art multi-task methods. We show via systematically designed experiments that our TempLe method universally outperforms the state-of-the-art multi-task methods (PAC-MDP or not) in various settings and regimes.

Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control.

Reinforcement learning (RL) and imitation learning (IL), especially equipped with deep neural networks, have been widely studied for autonomous robotic skill acquisition and control tasks. However, these methods and their extensions require extensive environmental interactions during training, which greatly prevents them from being applied to real-world robots. To alleviate this problem, we present an efficient model-free off-policy actor-critic algorithm for robotic skill acquisition and continuous control, by fusing the task reward with a task-oriented guiding reward, which is formulated by leveraging few and imperfect expert demonstrations. In this framework, the agent can explore the environment more intentionally, thus sampling efficiency can be achieved; moreover, the agent can also exploit the experience more effectively, thereby substantially improved performance can be realized simultaneously. The empirical results on robotic locomotion tasks show that the proposed scheme can lower sample complexity by 2-10 times in contrast with the state-of-the-art baseline deep RL (DRL) algorithms, while achieving performance better than that of the expert. Furthermore, the proposed algorithm achieves significant improvement in both sampling efficiency and asymptotic performance on tasks with sparse and delayed reward, wherein those baseline DRL algorithms struggle to make progress. This takes a substantial step forward to implement these methods to acquire skills autonomously for real robots.

Predictive Ultra-Reliable Communication: A Survival Analysis Perspective

Ultra-reliable communication (URC) is a key enabler for supporting immersive and mission-critical 5G applications. Meeting the strict reliability requirements of these applications is challenging due to the absence of accurate statistical models tailored to URC systems. In this letter, the wireless connectivity over dynamic channels is characterized via statistical learning methods. In particular, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model-based</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">data-driven</i> learning approaches are proposed to estimate the non-blocking connectivity statistics over a set of training samples with no knowledge on the dynamic channel statistics. Using principles of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">survival analysis</i> , the reliability of wireless connectivity is measured in terms of the probability of channel blocking events. Moreover, the maximum transmission duration for a given reliable non-blocking connectivity is predicted in conjunction with the confidence of the inferred transmission duration. Results show that the accuracy of detecting channel blocking events is higher using the model-based method for low to moderate reliability targets requiring low sample complexity. In contrast, the data-driven method yields a higher detection accuracy for higher reliability targets at the cost of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$100\times $ </tex-math></inline-formula> sample complexity.

Discriminative Training of Conditional Random Fields with Probably Submodular Constraints

Problems of segmentation, denoising, registration and 3D reconstruction are often addressed with the graph cut algorithm. However, solving an unconstrained graph cut problem is NP-hard. For tractable optimization, pairwise potentials have to fulfill the submodularity inequality. In our learning paradigm, pairwise potentials are created as the dot product of a learned vector w with positive feature vectors. In order to constrain such a model to remain tractable, previous approaches have enforced the weight vector to be positive for pairwise potentials in which the labels differ, and set pairwise potentials to zero in the case that the label remains the same. Such constraints are sufficient to guarantee that the resulting pairwise potentials satisfy the submodularity inequality. However, we show that such an approach unnecessarily restricts the capacity of the learned models. Guaranteeing submodularity for all possible inputs, no matter how improbable, reduces inference error to effectively zero, but increases model error. In contrast, we relax the requirement of guaranteed submodularity to solutions that are probably approximately submodular. We show that the conceptually simple strategy of enforcing submodularity on the training examples guarantees with low sample complexity that test images will also yield submodular pairwise potentials. Results are presented in the binary and muticlass settings, showing substantial improvement from the resulting increased model capacity.

A Survey on Deep Learning Techniques for Prognosis and Diagnosis of Cancer from Microarray Gene Expression Data

The gene expression classification and identification from DNA microarray data is efficient technique for cancer diagnosis and prognosis for specific cancer subtypes. DNA microarray technology has great potential to discover information from expression levels of thousands of gene. The collection of significant genes which can improve the accuracy can give proper direction in early diagnosis of cancer. Cancer may be of different subtypes. Cancer detection from microarray gene expression data has major challenge of low sample size, high dimensionality and complexity of the data. There is a need for fast and computationally efficient method to deal with these kind of challenges. Deep Learning has succeeded in numerous fields such as image, video, speech, and text processing. Gene expression analysis is a unique challenge to Deep Learning for various cancer detection and prediction tasks in order to set specific biomarkers for different cancer subtypes. In this paper, we briefly discuss the strengths of different Deep Learning architectures for a cancer detection and prediction of various types of cancer through gene expression analysis.

Learning Latent Events From Network Message Logs

We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a non-parametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the change-point detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of $97$ million messages collected over a period of $15$ days, from a distributed data center network which supports the operations of a large wireless service provider.

Multidimensional Sparse Fourier Transform Based on the Fourier Projection-Slice Theorem

We propose Multidimensional Random Slice-based Sparse Fourier Transform (MARS-SFT), a sparse Fourier transform for multidimensional, frequency-domain sparse signals, inspired by the idea of the Fourier projection-slice theorem. MARS-SFT identifies frequencies by operating on one-dimensional slices of the discrete-time domain data, taken along specially designed lines; these lines are parametrized by slopes that are randomly generated from a set at runtime. The discrete Fourier transforms (DFTs) of data slices represent DFT projections onto the lines along which the slices were taken. On designing the line lengths and slopes so that they allow for orthogonal and uniform projections of the sparse frequencies, frequency collisions are avoided with high probability, and the multidimensional frequencies can be recovered from their projections with low sample and computational complexity. We show analytically that the large number of degrees of freedom of frequency projections allows for the recovery of less sparse signals. Although the theoretical results are obtained for uniformly distributed frequencies, empirical evidence suggests that MARS-SFT is also effective in recovering clustered frequencies. We also propose an extension of MARS-SFT to address noisy signals that contain off-grid frequencies and demonstrate its performance in digital beamforming automotive radar signal processing. In that context, the robust MARS-SFT is used to identify range, velocity, and angular parameters of targets with low sample and computational complexity.