Unseen Samples Research Articles

Machine learning framework for determination of elastic modulus without contact model fitting

Many contact models have been proposed for determining the elastic modulus of materials based on AFM force measurement. However, contact model fitting could be a challenging task since the elastic modulus usually varies depending on the tip parameters (i.e., radius and half-opening angle) which are difficult to determine in practice. Therefore, this study proposes a supervised machine learning (SML) regression framework for determining the elastic modulus without the need to either select an appropriate contact model or perform contact model fitting. In the proposed approach, the SML regressor learns the relationship between the force-displacement (FZ) features and the elastic modulus, and then predicts the elastic modulus of unseen samples directly from their FZ features. The predictive power of the proposed framework is demonstrated using both homogeneous materials (polydimethylsiloxane, polymethyl methacrylate, and polyvinyl alcohol), and heterogeneous materials (Staphylococcus aureus bacteria and methylammonium lead iodide). It is shown that the Gaussian process regression (GPR) model achieves a higher prediction accuracy than the multiple linear regression (MLR), random forest (RF), and support vector regression (SVR) models for both groups of materials. Particularly, GPR achieves a testing coefficient of determination value of up to 91.55% for homogeneous materials, and 82.73% for heterogeneous materials. Overall, the results confirm that the proposed SML regression framework is capable of determining the elastic modulus of both homogeneous and heterogeneous materials without the need for both contact model fitting and knowledge of tip shape.

Hierarchical Semantic Risk Minimization for Large-Scale Classification

Hierarchical structures of labels usually exist in large-scale classification tasks, where labels can be organized into a tree-shaped structure. The nodes near the root stand for coarser labels, while the nodes close to leaves mean the finer labels. We label unseen samples from the root node to a leaf node, and obtain multigranularity predictions in the hierarchical classification. Sometimes, we cannot obtain a leaf decision due to uncertainty or incomplete information. In this case, we should stop at an internal node, rather than going ahead rashly. However, most existing hierarchical classification models aim at maximizing the percentage of correct predictions, and do not take the risk of misclassifications into account. Such risk is critically important in some real-world applications, and can be measured by the distance between the ground truth and the predicted classes in the class hierarchy. In this work, we utilize the semantic hierarchy to define the classification risk and design an optimization technique to reduce such risk. By defining the conservative risk and the precipitant risk as two competing risk factors, we construct the balanced conservative/precipitant semantic (BCPS) risk matrix across all nodes in the semantic hierarchy with user-defined weights to adjust the tradeoff between two kinds of risks. We then model the classification process on the semantic hierarchy as a sequential decision-making task. We design an algorithm to derive the risk-minimized predictions. There are two modules in this model: 1) multitask hierarchical learning and 2) deep reinforce multigranularity learning. The first one learns classification confidence scores of multiple levels. These scores are then fed into deep reinforced multigranularity learning for obtaining a global risk-minimized prediction with flexible granularity. Experimental results show that the proposed model outperforms state-of-the-art methods on seven large-scale classification datasets with the semantic tree.

Disturbed-entropy: A simple data quality assessment approach

From the perspective of information value, we proposed a simple and effective approach to assess data quality, called disturbed-entropy. In specific, considering image classification task, the existing samples per category are statistically represented as a pixel prototype, which is used to disturb the unseen samples. Then, the entropy of disturbed image is calculated based on predicted probability. Both the numerical and visual experiments are conducted to show the effect. In case of same data budget, the performance comparison based on selected good and bad data is significant and consistent. This work attempts to gain insight into data quality and redundancy.

Coarse-to-fine few-shot classification with deep metric learning

Few-shot classification predicts the labels of unseen samples using only a few labeled samples, and employs the samples of the classes disjoint with unseen classes for model training. It faces two primary challenges, i.e., handling sample pairs with different similarity degrees by single classifier, and learning discriminant patterns from very few labeled samples per class. To address them, this work presents a Coarse-to-Fine few-shot classification framework under the guidance of Metric-based Auxiliary learning, abbreviated as CFMA. In particular, it discriminates the image pairs with large difference by capturing global features, and models the similarity relation of the image pairs with small difference by mining the local region of interests. Moreover, CFMA adopts deep metric learning to improve the model adaptivity on the set of limited samples, and generates pseudo labels to dynamically guide the coarse learning in iteration. Empirical studies on several benchmark databases, including mini-ImageNet, tiered-ImageNet, and CUB, demonstrate that our method achieves more promising classification performance compared to many state-of-the-art alternatives.

Combination of structural and functional connectivity explains unique variation in specific domains of cognitive function

The relationship between structural and functional brain networks has been characterised as complex: the two networks mirror each other and show mutual influence but they also diverge in their organisation. This work explored whether a combination of structural and functional connectivity can improve the fit of regression models of cognitive performance. Principal Component Analysis (PCA) was first applied to cognitive data from the Human Connectome Project to identify latent cognitive components: Executive Function, Self-regulation, Language, Encoding and Sequence Processing. A Principal Component Regression approach with embedded Step-Wise Regression (SWR-PCR) was then used to fit regression models of each cognitive domain based on structural (SC), functional (FC) or combined structural-functional (CC) connectivity. Executive Function was best explained by the CC model. Self-regulation was equally well explained by SC and FC. Language was equally well explained by CC and FC models. Encoding and Sequence Processing were best explained by SC. Evaluation of out-of-sample models’ skill via cross-validation showed that SC, FC and CC produced generalisable models of Language performance. SC models performed most effectively at predicting Language performance in unseen sample. Executive Function was most effectively predicted by SC models, followed only by CC models. Self-regulation was only effectively predicted by CC models and Sequence Processing was only effectively predicted by FC models. The present study demonstrates that integrating structural and functional connectivity can help explaining cognitive performance, but that the added explanatory value (in-sample) may be domain-specific and can come at the expense of reduced generalisation performance (out-of-sample).

Failure Detection in Deep Neural Networks for Medical Imaging.

Deep neural networks (DNNs) have started to find their role in the modern healthcare system. DNNs are being developed for diagnosis, prognosis, treatment planning, and outcome prediction for various diseases. With the increasing number of applications of DNNs in modern healthcare, their trustworthiness and reliability are becoming increasingly important. An essential aspect of trustworthiness is detecting the performance degradation and failure of deployed DNNs in medical settings. The softmax output values produced by DNNs are not a calibrated measure of model confidence. Softmax probability numbers are generally higher than the actual model confidence. The model confidence-accuracy gap further increases for wrong predictions and noisy inputs. We employ recently proposed Bayesian deep neural networks (BDNNs) to learn uncertainty in the model parameters. These models simultaneously output the predictions and a measure of confidence in the predictions. By testing these models under various noisy conditions, we show that the (learned) predictive confidence is well calibrated. We use these reliable confidence values for monitoring performance degradation and failure detection in DNNs. We propose two different failure detection methods. In the first method, we define a fixed threshold value based on the behavior of the predictive confidence with changing signal-to-noise ratio (SNR) of the test dataset. The second method learns the threshold value with a neural network. The proposed failure detection mechanisms seamlessly abstain from making decisions when the confidence of the BDNN is below the defined threshold and hold the decision for manual review. Resultantly, the accuracy of the models improves on the unseen test samples. We tested our proposed approach on three medical imaging datasets: PathMNIST, DermaMNIST, and OrganAMNIST, under different levels and types of noise. An increase in the noise of the test images increases the number of abstained samples. BDNNs are inherently robust and show more than 10% accuracy improvement with the proposed failure detection methods. The increased number of abstained samples or an abrupt increase in the predictive variance indicates model performance degradation or possible failure. Our work has the potential to improve the trustworthiness of DNNs and enhance user confidence in the model predictions.

A multichannel EfficientNet deep learning-based stacking ensemble approach for lung disease detection using chest X-ray images.

This paper proposes a multichannel deep learning approach for lung disease detection using chest X-rays. The multichannel models used in this work are EfficientNetB0, EfficientNetB1, and EfficientNetB2 pretrained models. The features from EfficientNet models are fused together. Next, the fused features are passed into more than one non-linear fully connected layer. Finally, the features passed into a stacked ensemble learning classifier for lung disease detection. The stacked ensemble learning classifier contains random forest and SVM in the first stage and logistic regression in the second stage for lung disease detection. The performance of the proposed method is studied in detail for more than one lung disease such as pneumonia, Tuberculosis (TB), and COVID-19. The performances of the proposed method for lung disease detection using chest X-rays compared with similar methods with the aim to show that the method is robust and has the capability to achieve better performances. In all the experiments on lung disease, the proposed method showed better performance and outperformed similar lung disease existing methods. This indicates that the proposed method is robust and generalizable on unseen chest X-rays data samples. To ensure that the features learnt by the proposed method is optimal, t-SNE feature visualization was shown on all three lung disease models. Overall, the proposed method has shown 98% detection accuracy for pediatric pneumonia lung disease, 99% detection accuracy for TB lung disease, and 98% detection accuracy for COVID-19 lung disease. The proposed method can be used as a tool for point-of-care diagnosis by healthcare radiologists.Journal instruction requires a city for affiliations; however, this is missing in affiliation 3. Please verify if the provided city is correct and amend if necessary.correct

Dyskinesia Estimation of Imbalanced Data Using a Deep-Learning Model.

The collection of Parkinson's Disease (PD) time-series data usually results in imbalanced and incomplete datasets due to the geometric distribution of PD complications' sever-ity scores. Consequently, when training deep convolutional models on these datasets, the models suffer from overfitting and lack generalizability to unseen data. In this paper, we investigated a new framework of Conditional Generative Ad-versarial Netuwoks (cGANs) as a solution to improve the extrapolation and generalizability of the regression models in such datasets. We used a real-world PD dataset to esti-mate Dyskinesia severity in patients with PD. The developed cGAN demonstrated significantly better generalizability to unseen data samples than a traditional Convolutional Neural Network with an improvement of 34%. This solution can be applied in similar imbalanced time-series data, especially in the healthcare domain, where balanced and uniformly distributed data samples are not readily available.

Fracture R-CNN: An anchor-efficient anti-interference framework for skull fracture detection in CT images.

Skull fracture, as a common traumatic brain injury, can lead to multiple complications including bleeding, leaking of cerebrospinal fluid, infection, and seizures. Automatic skull fracture detection (SFD) is of great importance, especially in emergencymedicine. Existing algorithms for SFD, developed based on hand-crafted features, suffer from low detection accuracy due to poor generalizability to unseen samples. Deploying deep detectors designed for natural images like Faster Region-based Convolutional Neural Network (R-CNN) for SFD can be helpful but are of high redundancy and with nonnegligible false detections due to the cranial suture and skull base interference. Therefore, we, for the first time, propose an anchor-efficient anti-interference deep learning framework named Fracture R-CNN for accurate SFD with low computationalcost. The proposed Fracture R-CNN is developed by incorporating the prior knowledge utilized in clinical diagnosis into the original Faster R-CNN. Specifically, based on the distributions of skull fractures, we first propose an adaptive anchoring region proposal network (AA-RPN) to generate proposals for diverse-scale fractures with low computational complexity. Then, based on the prior knowledge that cranial sutures exist in the junctions of bones and usually contain sclerotic margins, we design an anti-interference head (A-Head) network to eliminate the cranial suture interference for better SFD detection. In addition, to further enhance the anti-interference ability of the proposed A-Head, a difficulty-balanced weighted loss function is proposed to emphasize more on distinguishing the interference areas from the skull base and the cranial sutures duringtraining. Experimental results demonstrate that the proposed Fracture R-CNN outperforms the current state-of-the-art (SOTA) deep detectors for SFD with a higher recall and fewer false detections. Compared to Faster R-CNN, the proposed Fracture R-CNN improves the average precision (AP) by 11.74% and the free-response receiver operating characteristic (FROC) score by 11.08%. Through validating on various backbones, we further demonstrate the architecture independence of Fracture R-CNN, making it extendable to other detectionapplications. As the customized deep learning-based framework for SFD, Fracture R-CNN can effectively overcome the unique challenges in SFD with less computational cost, leading to a better detection performance compared to the SOTA deep detectors. Moreover, we believe the prior knowledge explored for Fracture R-CNN would shed new light on future deep learning approaches for SFD.

Intelligent analysis of excitation-emission matrix fluorescence fingerprint to identify and quantify adulteration in camellia oil based on machine learning

Camellia oil (CAO) is a premium edible vegetable oil with medical value and biological activity, but it is susceptible to adulteration. Therefore, the demand for intelligent analysis to decipher the category and proportion of adulterated oil in CAO was the main driver of this work. Excitation-emission matrix fluorescence (EEMF) spectra of 933 vegetable oil samples were characterized by a chemometric method to obtain chemically meaningful information. Authenticity identification models were constructed using four machine learning methods to realize the discrimination of oil species adulterated in CAO mixtures. Meanwhile, quantitative models were established aiming at the fraud of CAO proportion in blended oil. Results showed that the specially constructed CNN obtained the optimal performance when evaluating unseen real-world samples, with a classification accuracy of 95.8% and 92.2%, and mean-absolute quantitative errors between 2.6 and 6.7%. Therefore, EEMF fingerprints coupled with machine learning are expected to provide intelligent and accurate analysis for authenticity detection of CAO.

Stochastic optimal well control in subsurface reservoirs using reinforcement learning

We present a case study of model-free reinforcement learning (RL) framework to solve stochastic optimal control for a predefined parameter uncertainty distribution and partially observable system. We focus on robust optimal well control problem which is a subject of intensive research activities in the field of subsurface reservoir management. For this problem, the system is partially observed since the data is only available at well locations. Furthermore, the model parameters are highly uncertain due to sparsity of available field data. In principle, RL algorithms are capable of learning optimal action policies – a map from states to actions – to maximize a numerical reward signal. In deep RL, this mapping from state to action is parameterized using a deep neural network. In the RL formulation of the robust optimal well control problem, the states are represented by saturation and pressure values at well locations while the actions represent the valve openings controlling the flow through wells. The numerical reward refers to the total sweep efficiency and the uncertain model parameter is the subsurface permeability field. The model parameter uncertainties are handled by introducing a domain randomization scheme that exploits cluster analysis on its uncertainty distribution. We present numerical results using two state-of-the-art RL algorithms, proximal policy optimization (PPO) and advantage actor–critic (A2C), on two subsurface flow test cases representing two distinct uncertainty distributions of permeability field. The results were benchmarked against optimization results obtained using differential evolution algorithm. Furthermore, we demonstrate the robustness of the proposed use of RL by evaluating the learned control policy on unseen samples drawn from the parameter uncertainty distribution that were not used during the training process.

Beyond Normal Distribution: More Factual Feature Generation Network for Generalized Zero-Shot Learning

Due to the prosperous development of generative models, research works have achieved great success on the generalized zero-shot learning (GZSL) task. In most generative methods of GZSL, researchers try to utilize attributes and normally distributed noise to generate visual features, which ignores whether the normal distribution can perfectly represent all categories. Therefore, in this article, we exploit variational auto-encoders (VAE) and visual features to generate image-level noise that can preserve class-level characteristics in more detail and propose a mechanism called more factual generative network (MFGN) to achieve more authentic generative process. In other words, it is to transfer the seen feature distribution to the unseen domains and regulate the knowledge to correct the generation of unseen samples. Extensive experiments are conducted on four popular datasets and the results demonstrate the effectiveness of the proposed work.

P-108 Real-time deep learning based multi object tracking of spermatozoa in fresh samples

Abstract Study question Can real-time deep learning model track hundreds of spermatozoa simultaneously? Summary answer The state-of-the-art deep learning detection model YOLOv5 shows possibilities of multi-sperm tracking with high sensitivity and precision. What is known already Computer-aided sperm analysis (CASA) systems can be used for the evaluation of sperm motility by applying tracking algorithms. However, the presence of cellular debris and/or cell aggregations in human semen samples makes the tracking of spermatozoa difficult, resulting in unreliable motility assessment. Thus, there is a need for an improved methodology. There are several studies on detecting spermatozoa in real-time, such as DeepSperm, which is trained and tested on bull semen samples. However, differences in characteristics between human and animal spermatozoa imply the requirement of a multi-sperm tracking system adapted to human spermatozoa. Study design, size, duration We used the open-access VISEM dataset consisting of video recordings of human semen samples from 85 participants. We selected three videos with low sperm counts to perform manual annotations (bounding boxes around sperms) and create a training dataset. Only the first 30 seconds of each video were extracted for annotations. Then, all the spermatozoa of the three videos were annotated with bounding boxes using LabelBox. The annotations will be made public in a future study. Participants/materials, setting, methods We used two different object detection models, YOLOv5 Nano and YOLOv5 XLarge to detect spermatozoa and performed transfer learning without layer freezing. Each model was trained for a maximum of 300 epochs, stopping if the validation loss did not improve for the previous 100 epochs. Leave-one-out cross-validation was performed to obtain the presented results. Precision, sensitivity, and mean average precision (mAP) are the quantitative metrics measured to evaluate our detection models. Main results and the role of chance Our models are able to detect a large number of human spermatozoa simultaneously in a given video in real-time. The model YOLOv5 Nano took around 45 min and the YOLOv5 XLarge model took around 270 min to train on one fold. In terms of training and detection, the Nano model is faster because it has a lower number of trainable parameters compared to the XLarge model, but at the expense of precision. YOLOv5 Nano has capability of predicting 200 frames per second (fps), and YOLOv5 XLarge can predict 56 fps. Both models show high prediction rates which are greater than the real-time prediction 25 fps threshold. Performance-wise, the Nano model shows an average recall value of 0.8545 which is better than the average recall value of 0.7632 of XLarge model. In contrast, the XLarge model shows a higher precision value of 0.8020 compared to the low precision of 0.7149 of the Nano model. Furthermore, the XLarge model has an average mAP (mAP_0.5) of 0.7632 which is larger than the mAP value of the Nano model of 0.7027. The current prediction errors in this stage might be a result of the small training dataset. Limitations, reasons for caution The low amount of data makes validating the models difficult, especially evaluating how well they generalize to unseen samples. Furthermore, the training data consisted of samples containing low sperm counts, making the performance on samples with high sperm concentration uncertain. Wider implications of the findings Sperm tracking is integral to achieving accurate and less subjective motility assessment. The detections can be used to analyse the individual spermatozoa, leading to better performance. Deep feature extraction, trajectory prediction, and trajectory extraction can be used for future studies like generating synthetic spermatozoa for training generalizable machine learning models. Trial registration number not applicable

Reducing Flipping Errors in Deep Neural Networks

Deep neural networks (DNNs) have been widely applied in various domains in artificial intelligence including computer vision and natural language processing. A DNN is typically trained for many epochs and then a validation dataset is used to select the DNN in an epoch (we simply call this epoch ``the last epoch") as the final model for making predictions on unseen samples, while it usually cannot achieve a perfect accuracy on unseen samples. An interesting question is ``how many test (unseen) samples that a DNN misclassifies in the last epoch were ever correctly classified by the DNN before the last epoch?". In this paper, we empirically study this question and find on several benchmark datasets that the vast majority of the misclassified samples in the last epoch were ever classified correctly before the last epoch, which means that the predictions for these samples were flipped from ``correct" to ``wrong". Motivated by this observation, we propose to restrict the behavior changes of a DNN on the correctly-classified samples so that the correct local boundaries can be maintained and the flipping error on unseen samples can be largely reduced. Extensive experiments on different benchmark datasets with different modern network architectures demonstrate that the proposed flipping error reduction (FER) approach can substantially improve the generalization, the robustness, and the transferability of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead.

Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval

Sketch-based 3D shape retrieval is a challenging task due to the large domain discrepancy between sketches and 3D shapes. Since existing methods are trained and evaluated on the same categories, they cannot effectively recognize the categories that have not been used during training. In this paper, we propose a novel domain disentangled generative adversarial network (DD-GAN) for zero-shot sketch-based 3D retrieval, which can retrieve the unseen categories that are not accessed during training. Specifically, we first generate domain-invariant features and domain-specific features by disentangling the learned features of sketches and 3D shapes, where the domain-invariant features are used to align with the corresponding word embeddings. Then, we develop a generative adversarial network that combines the domain-specific features of the seen categories with the aligned domain-invariant features to synthesize samples, where the synthesized samples of the unseen categories are generated by using the corresponding word embeddings. Finally, we use the synthesized samples of the unseen categories combined with the real samples of the seen categories to train the network for retrieval, so that the unseen categories can be recognized. In order to reduce the domain shift problem, we utilize unlabeled unseen samples to enhance the discrimination ability of the discriminator. With the discriminator distinguishing the generated samples from the unlabeled unseen samples, the generator can generate more realistic unseen samples. Extensive experiments on the SHREC'13 and SHREC'14 datasets show that our method significantly improves the retrieval performance of the unseen categories.

An Efficient Gait Abnormality Detection Method Based on Classification

In the study of human mobility, gait analysis is a well-recognized assessment methodology. Despite its widespread use, doubts exist about its clinical utility, i.e., its potential to influence the diagnostic-therapeutic practice. Gait analysis evaluates the walking pattern (normal/abnormal) based on the gait cycle. Based on the analysis obtained, various applications can be developed in the medical, security, sports, and fitness domain to improve overall outcomes. Wearable sensors provide a convenient, efficient, and low-cost approach to gather data, while machine learning methods provide high accuracy gait feature extraction for analysis. The problem is to identify gait abnormalities and if present, subsequently identify the locations of impairments that lead to the change in gait pattern of the individual. Proper physiotherapy treatment can be provided once the location/landmark of the impairment is known correctly. In this paper, classification of multiple anatomical regions and their combination on a large scale highly imbalanced dataset is carried out. We focus on identifying 27 different locations of injury and formulate it as a multi-class classification approach. The advantage of this method is the convenience and simplicity as compared to previous methods. In our work, a benchmark is set to identify the gait disorders caused by accidental impairments at multiple anatomical regions using the GaitRec dataset. In our work, machine learning models are trained and tested on the GaitRec dataset, which provides Ground Reaction Force (GRF) data, to analyze an individual’s gait and further classify the gait abnormality (if present) at the specific lower-region portion of the body. The design and implementation of machine learning models are carried out to detect and classify the gait patterns between healthy controls and gait disorders. Finally, the efficacy of the proposed approach is showcased using various qualitative accuracy metrics. The achieved test accuracy is 96% and an F1 score of 95% is obtained in classifying various gait disorders on unseen test samples. The paper concludes by stating how machine learning models can help to detect gait abnormalities along with directions of future work.

Boosting Generative Zero-Shot Learning by Synthesizing Diverse Features with Attribute Augmentation

The recent advance in deep generative models outlines a promising perspective in the realm of Zero-Shot Learning (ZSL). Most generative ZSL methods use category semantic attributes plus a Gaussian noise to generate visual features. After generating unseen samples, this family of approaches effectively transforms the ZSL problem into a supervised classification scheme. However, the existing models use a single semantic attribute, which contains the complete attribute information of the category. The generated data also carry the complete attribute information, but in reality, visual samples usually have limited attributes. Therefore, the generated data from attribute could have incomplete semantics. Based on this fact, we propose a novel framework to boost ZSL by synthesizing diverse features. This method uses augmented semantic attributes to train the generative model, so as to simulate the real distribution of visual features. We evaluate the proposed model on four benchmark datasets, observing significant performance improvement against the state-of-the-art.

HSD97 Use Machine Learning (ML) Techniques to Identify the Likelihood of Hospitalization for High-Risk Patients Diagnosed with COVID 19

HSD97 Use Machine Learning (ML) Techniques to Identify the Likelihood of Hospitalization for High-Risk Patients Diagnosed with COVID 19

Deep Learning concepts for genomics : an overview

Nowadays, Deep Learning is taking the world by a storm, known as a technology that makes use of Artificial Neural Networks to automatically extrapolate knowledge from a training data set, then uses this knowledge to give predictions for unseen samples. This data driven paradigm gained a widespread adoption in many disciplines, from handwriting recognition, driving an autonomous car to cracking the 50-year-old protein folding problem. With this review, we shed some light on the concepts of Deep Learning and provide some visualizations, skim over the different architectures such as Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and touch upon the modern architectures such as Transformers and BERT. We also provide various examples targeting the genomics field, reference utilities, libraries useful for newcomers and disseminate our feedback.

A Mathematical Framework for Learning Probability Distributions

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.