Neural Information Processing Systems Research Articles

Optimization of Hierarchical Regression Model with Application to Optimizing Multi-Response Regression K-ary Trees

THIS PAPER HAS BEEN RETRACTED The University of California, Merced campus, recently completed a formal investigation into allegations that content in this paper ("Optimization of Hierarchical Regression Model with Application to Optimizing Multi-Response Regression K-ary Trees," in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 5133– 5142) was not properly attributed to the novel work of Professor Carreira—Perpiñán, reflected in a previously published paper, "Alternating Optimization of Decision Trees, with Application to Learning Sparse Oblique Trees,” published in Advances in Neural Information Processing Systems, 2018, (vol. 31, pp. 1211–1221). The UC Merced investigation was conducted in consultation with NSF, which funded the research reflected in the original Advances in Neural Information Processing Systems paper, and in accordance with UC Merced policies and procedures on research misconduct. The allegations were substantiated by a preponderance of the evidence. The investigation committee found, in pertinent part, that “key novel ideas in the AAAI paper were taken from the NeurIPS paper without appropriate credit." The initial submission of the AAAI paper, upon which the acceptance decision was made, did not include a reference to the NeurIPS paper, and thus represented a clear case of plagiarism. A citation to the NeurIPS paper was added in the final revision, after the reviewers had already decided to accept it, making it impossible for the AAAI peer-review process to consider the contributions of the AAAI article in light of the earlier work reported in the NeurIPS paper. Furthermore, the citation to the NeurIPS paper in this manuscript does not properly credit the original contributions of the NeurIPS paper and how those ideas were reflected in the work reported in this paper. In response to a formal request made by the University of California's Chancellor's office, and after careful review by the AAAI Publications Committee, AAAI has decided to retract this paper. One of the conditions of submission of a paper for publication in AAAI conference proceedings is that authors declare explicitly that their work is original and has not appeared in a publication elsewhere. Reuse of any data should be appropriately cited. As such this paper represents a severe abuse of the scientific publishing system. The scientific community takes a very strong view on this matter and apologies are offered to readers of this proceedings that this was not detected during the submission process.

Stochastic gradient Hamiltonian Monte Carlo with variance reduction for Bayesian inference

Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high variance of noisy stochastic gradients, Dubey et al. (in: Advances in neural information processing systems, pp 1154–1162, 2016) applied the standard variance reduction technique on stochastic gradient Langevin dynamics and obtained both theoretical and experimental improvements. In this paper, we apply the variance reduction tricks on Hamiltonian Monte Carlo and achieve better theoretical convergence results compared with the variance-reduced Langevin dynamics. Moreover, we apply the symmetric splitting scheme in our variance-reduced Hamiltonian Monte Carlo algorithms to further improve the theoretical results. The experimental results are also consistent with the theoretical results. As our experiment shows, variance-reduced Hamiltonian Monte Carlo demonstrates better performance than variance-reduced Langevin dynamics in Bayesian regression and classification tasks on real-world datasets.

Depth with nonlinearity creates no bad local minima in ResNets

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations. As a result, this paper provides an affirmative answer to an open question stated in a paper in the conference on Neural Information Processing Systems 2018. This paper advances the optimization theory of deep learning only for ResNets and not for other network architectures.

Dynamic social network analysis: A novel approach using agent‐based model, author‐topic model, and pretopology

SummaryWe propose in this work a novel approach for dynamic social network analysis by combining an agent‐based model, an author‐topic model, and pretopology. We first introduce an analytical model for a dynamic social network associated with textual content using agent‐based and author‐topic models, namely, Textual‐ABM. The purpose of Textual‐ABM is to support for the concept exploitation of the “dynamics” of a social network, which contains not only network's structure transformation but also agent's interest variation over time. Agent's interest is revealed through topic probability distribution, which is estimated based on textual data using an author‐topic model. In addition to demonstrating the fluctuation of the social network related to textual content, we also exploit information propagation phenomena by proposing two expanded spreading models. The first model is an expanded model of an independent cascade model in which probability of infection is formed on homophily, namely H‐IC. We have implemented experiments on a collected dataset from the Neural Information Processing Systems Conference and have acquired satisfying results. Furthermore, we propose an extended model of pretopological cascade model from our previous work, namely, Textual‐PCM. The advantage of PCM comparison with classical cascade model is to utilize pseudoclosure function built from pretopology to define the more complex set of neighborhoods. In this work, we expand PCM to apply detail for a social network related to textual information. A toy example with some experiments and discussion is illustrated for Textual‐PCM. The work in this paper is an extended version of our paper dynamic social network analysis using author‐topic model presented in I4CS 2018 Conference.

Partially collapsed Gibbs sampling for latent Dirichlet allocation

A latent Dirichlet allocation (LDA) model is a machine learning technique to identify latent topics from text corpora within a Bayesian hierarchical framework. Current popular inferential methods to fit the LDA model are based on variational Bayesian inference, collapsed Gibbs sampling, or a combination of these. Because these methods assume a unimodal distribution over topics, however, they can suffer from large bias when text corpora consist of various clusters with different topic distributions. This paper proposes an inferential LDA method to efficiently obtain unbiased estimates under flexible modeling for heterogeneous text corpora with the method of partial collapse and the Dirichlet process mixtures. The method is illustrated using a simulation study and an application to a corpus of 1300 documents from neural information processing systems (NIPS) conference articles during the period of 2000–2002 and British Broadcasting Corporation (BBC) news articles during the period of 2004–2005.

Chebyshev polynomials, moment matching, and optimal estimation of the unseen

We consider the problem of estimating the support size of a discrete distribution whose minimum nonzero mass is at least $\frac{1}{k}$. Under the independent sampling model, we show that the sample complexity, that is, the minimal sample size to achieve an additive error of $\varepsilon k$ with probability at least 0.1 is within universal constant factors of $\frac{k}{\log k}\log^{2}\frac{1}{\varepsilon }$, which improves the state-of-the-art result of $\frac{k}{\varepsilon^{2}\log k}$ in [In Advances in Neural Information Processing Systems (2013) 2157–2165]. Similar characterization of the minimax risk is also obtained. Our procedure is a linear estimator based on the Chebyshev polynomial and its approximation-theoretic properties, which can be evaluated in $O(n+\log^{2}k)$ time and attains the sample complexity within constant factors. The superiority of the proposed estimator in terms of accuracy, computational efficiency and scalability is demonstrated in a variety of synthetic and real datasets.

Using Convolutional Neural Networks for the Detection of Spinal Transverse Processes

Introduction: Scoliosis, an excessive curvature of the spine, affects approximately 1 in 1,000 individuals. As a result, there have formerly been implementations of mandatory scoliosis screening procedures. Screening programs are no longer widely used as the harms often outweigh the benefits; it causes many adolescents to undergo frequent diagnosis X-ray procedure This makes spinal ultrasounds an ideal substitute for scoliosis screening in patients, as it does not expose them to those levels of radiation.
 Spinal curvatures can be accurately computed from the location of spinal transverse processes, by measuring the vertebral angle from a reference line [1]. However, ultrasound images are less clear than x-ray images, making it difficult to identify the spinal processes. To overcome this, we employ deep learning using a convolutional neural network, which is a powerful tool for computer vision and image classification [2].
 Method: A total of 2,752 ultrasound images were recorded from a spine phantom to train a convolutional neural network. Subsequently, we took another recording of 747 images to be used for testing. All the ultrasound images from the scans were then segmented manually, using the 3D Slicer (www.slicer.org) software. Next, the dataset was fed through a convolutional neural network. The network used was a modified version of GoogLeNet (Inception v1), with 2 linearly stacked inception models. This network was chosen because it provided a balance between accurate performance, and time efficient computations.
 Results: Deep learning classification using the Inception model achieved an accuracy of 84% for the phantom scan.
 Conclusion: The classification model performs with considerable accuracy. Better accuracy needs to be achieved, possibly with more available data and improvements in the classification model.
 Acknowledgements: G. Fichtinger is supported as a Canada Research Chair in Computer-Integrated Surgery. This work was funded, in part, by NIH/NIBIB and NIH/NIGMS (via grant 1R01EB021396-01A1 - Slicer+PLUS: Point-of-Care Ultrasound) and by CANARIE’s Research Software Program. 
 
 Figure 1: Ultrasound scan containing a transverse process (left), and ultrasound scan containing no transverse process (right). Figure 2: Accuracy of classification for training (red) and validation (blue).

Sinkhorn Algorithm for Lifted Assignment Problems

Recently, Sinkhorn's algorithm was applied for approximately solving linear programs emerging from optimal transport very effeciently [M. Cuturi, Advances in Neural Information Processing Systems, ...

Masked 3D conditional generative adversarial network for rock mesh generation

3D-GAN (Wu et al. in: Advances in Neural Information Processing Systems, pp. 82–90, 2016) has been introduced as a novel way to generate 3D models. In this paper, we propose a 3D-Masked-CGAN approach to apply in the generation of irregular 3D mesh geometry such as rocks. While there are many ways to generate 3D objects, the generation of irregular 3D models has its own peculiarity. To make a model realistic is extremely time-consuming and in high cost. In order to control the shape of generated 3D models, we extend 3D-GAN by adding conditional information into both the generator and discriminator. It is shown that that this model can generate 3D rock models with effective control over the shapes of generated models.

Uncertainty Quantification in Graph-Based Classification of High Dimensional Data

Classification of high dimensional data finds wide-ranging applications. In many of these applications equipping the resulting classification with a measure of uncertainty may be as important as the classification itself. In this paper we introduce, develop algorithms for, and investigate the properties of a variety of Bayesian models for the task of binary classification; via the posterior distribution on the classification labels, these methods automatically give measures of uncertainty. The methods are all based on the graph formulation of semisupervised learning. We provide a unified framework which brings together a variety of methods that have been introduced in different communities within the mathematical sciences. We study probit classification [C. K. Williams and C. E. Rasmussen, “Gaussian Processes for Regression,” in Advances in Neural Information Processing Systems 8, MIT Press, 1996, pp. 514--520] in the graph-based setting, generalize the level-set method for Bayesian inverse problems [M. A. Iglesias, Y. Lu, and A. M. Stuart, Interfaces Free Bound., 18 (2016), pp. 181--217] to the classification setting, and generalize the Ginzburg--Landau optimization-based classifier [A. L. Bertozzi and A. Flenner, Multiscale Model. Simul., 10 (2012), pp. 1090--1118], [Y. Van Gennip and A. L. Bertozzi, Adv. Differential Equations, 17 (2012), pp. 1115--1180] to a Bayesian setting. We also show that the probit and level-set approaches are natural relaxations of the harmonic function approach introduced in [X. Zhu et al., “Semi-supervised Learning Using Gaussian Fields and Harmonic Functions,” in ICML, Vol. 3, 2003, pp. 912--919]. We introduce efficient numerical methods, suited to large datasets, for both MCMC-based sampling and gradient-based MAP estimation. Through numerical experiments we study classification accuracy and uncertainty quantification for our models; these experiments showcase a suite of datasets commonly used to evaluate graph-based semisupervised learning algorithms.

Adversarial Attacks and Defenses Against Deep Neural Networks: A Survey

Deep learning has achieved great successes in various types of applications over recent years. On the other hand, it has been found that deep neural networks (DNNs) can be easily fooled by adversarial input samples. This vulnerability raises major concerns in security-sensitive environments. Therefore, research in attacking and defending DNNs with adversarial examples has drawn great attention. The goal of this paper is to review the types of adversarial attacks and defenses, describe the state-of-the-art methods for each group, and compare their results. In addition, we present some of the top-scored competition submissions for Neural Information Processing Systems (NIPS) in 2017, their solution models, and demonstrate their results. This adversary competition was organized by Google Brain for research scientists to come up with novel solutions that generate adversarial examples and also defend against them. Its contribution is significant on this era of machine learning and DNNs.

The neural networks used in FDM printing study

Neural networks have aroused a lively interest since 1943 when Warren McCulloch and Walter Pitts proposed a neural network model (a single layer model), that has remained fundamentally structural even today for most neural networks. Problem solving and implicit the study of a system's operating model such as 3D printing involves the association between input data, hypotheses and output data, and neural networks provide the ability to form their own model of solving. The main difference between neural networks and other information processing systems is the ability to learn from interacting with the environment and so improving performance. A correct representation of information, allowing interpretation, prediction, and response to an external stimulus, can allow the network to build a model of the considered process, in the paper case fused deposition modelling (FDM) process.

Geodesic PCA versus Log-PCA of Histograms in the Wasserstein Space

This paper is concerned with the statistical analysis of datasets whose elements are random histograms. For the purpose of learning principal modes of variation from such data, we consider the issue of computing the principal component analysis (PCA) of histograms with respect to the 2-Wasserstein distance between probability measures. To this end, we propose comparing the methods of log-PCA and geodesic PCA in the Wasserstein space as introduced in [J. Bigot et al., Ann. Inst. Henri Poincare Probab. Stat., 53 (2017), pp. 1--26; V. Seguy and M. Cuturi, Principal geodesic analysis for probability measures under the optimal transport metric, in Advances in Neural Information Processing Systems 28, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, eds., Curran Associates, Inc., Red Hook, NY, 2015, pp. 3294--3302]. Geodesic PCA involves solving a nonconvex optimization problem. To solve it approximately, we propose a novel forward-backward algorithm. This allows us to give a detailed comparison bet...

Real-time image recognition using weighted spatial pyramid networks

The latest-generation earth observation instruments on airborne and satellite platforms are currently producing an almost continuous high-dimensional data stream. This exponentially growing data poses a new challenge for real-time image processing and recognition. Making full and effective use of the spectral information and spatial structure information of high-resolution remote sensing image is the key to the processing and recognition of high-resolution remote sensing data. In this paper, the adaptive multipoint moment estimation (AMME) stochastic optimization algorithm is proposed for the first time by using the finite lower-order moments and adding the estimating points. This algorithm not only reduces the probability of local optimum in the learning process, but also improves the convergence rate of the convolutional neural network (Lee Cun et al. in Advances in neural information processing systems, 1990). Second, according to the remote sensing image with characteristics of complex background and small sensitive targets, and by automatic discovery, locating small targets, and giving high weights, we proposed a feature extraction method named weighted pooling to further improve the performance of real-time image recognition. We combine the AMME and weighted pooling with the spatial pyramid representation (Harada et al. in Comput Vis Pattern Recognit 1617–1624, 2011) algorithm to form a new, multiscale, and multilevel real-time image recognition model and name it weighted spatial pyramid networks (WspNet). At the end, we use the MNIST, ImageNet, and natural disasters under remote sensing data sets to test WspNet. Compared with other real-time image recognition models, WspNet achieve a new state of the art in terms of convergence rate and image feature extraction compared with conventional stochastic gradient descent method [like AdaGrad, AdaDelta and Adam (Zeiler in Comput Sci, 2012; Kingma and Ba in Comput Sci, 2014; Duchi et al. in J Mach Learn Res 12(7):2121–2159, 2011] and pooling method [like max-pooling, avg-pooling and stochastic-pooling (Zeiler and Fergus in stochastic-pooling for regularization of deep convolutional neural networks, 2013)].

Design and analysis of the NIPS 2016 review process

Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement. In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers. We make a number of key observations, provide suggestions that may be useful for subsequent conferences, and discuss open problems towards the goal of improving peer review.

Estimating the average need of semantic knowledge from distributional semantic models.

Continuous bag of words (CBOW) and skip-gram are two recently developed models of lexical semantics (Mikolov, Chen, Corrado, & Dean, Advances in Neural Information Processing Systems, 26, 3111-3119, 2013). Each has been demonstrated to perform markedly better at capturing human judgments about semantic relatedness than competing models (e.g., latent semantic analysis; Landauer & Dumais, Psychological Review, 104(2), 1997 211; hyperspace analogue to language; Lund & Burgess, Behavior Research Methods, Instruments, & Computers, 28(2), 203-208, 1996). The new models were largely developed to address practical problems of meaning representation in natural language processing. Consequently, very little attention has been paid to the psychological implications of the performance of these models. We describe the relationship between the learning algorithms employed by these models and Anderson's rational theory of memory (J. R. Anderson & Milson, Psychological Review, 96(4), 703, 1989) and argue that CBOW is learning word meanings according to Anderson's concept of needs probability. We also demonstrate that CBOW can account for nearly all of the variation in lexical access measures typically attributable to word frequency and contextual diversity-two measures that are conceptually related to needs probability. These results suggest two conclusions: One, CBOW is a psychologically plausible model of lexical semantics. Two, word frequency and contextual diversity do not capture learning effects but rather memory retrieval effects.

Non-asymptotic convergence analysis of inexact gradient methods for machine learning without strong convexity

Many recent applications in machine learning and data fitting call for the algorithmic solution of structured smooth convex optimization problems. Although the gradient descent method is a natural choice for this task, it requires exact gradient computations and hence can be inefficient when the problem size is large or the gradient is difficult to evaluate. Therefore, there has been much interest in inexact gradient methods (IGMs), in which an efficiently computable approximate gradient is used to perform the update in each iteration. Currently, non-asymptotic linear convergence results for IGMs are typically established under the assumption that the objective function is strongly convex, which is not satisfied in many applications of interest; while linear convergence results that do not require the strong convexity assumption are usually asymptotic in nature. In this paper, we combine the best of these two types of results by developing a framework for analysing the non-asymptotic convergence rates of IGMs when they are applied to a class of structured convex optimization problems that includes least squares regression and logistic regression. We then demonstrate the power of our framework by proving, in a unified manner, new linear convergence results for three recently proposed algorithms—the incremental gradient method with increasing sample size [R.H. Byrd, G.M. Chin, J. Nocedal, and Y. Wu, Sample size selection in optimization methods for machine learning, Math. Program. Ser. B 134 (2012), pp. 127–155; M.P. Friedlander and M. Schmidt, Hybrid deterministic–stochastic methods for data fitting, SIAM J. Sci. Comput. 34 (2012), pp. A1380–A1405], the stochastic variance-reduced gradient (SVRG) method [R. Johnson and T. Zhang, Accelerating stochastic gradient descent using predictive variance reduction, Advances in Neural Information Processing Systems 26: Proceedings of the 2013 Conference, 2013, pp. 315–323], and the incremental aggregated gradient (IAG) method [D. Blatt, A.O. Hero, and H. Gauchman, A convergent incremental gradient method with a constant step size, SIAM J. Optim. 18 (2007), pp. 29–51]. We believe that our techniques will find further applications in the non-asymptotic convergence analysis of other first-order methods.

Optimal learning with Bernstein online aggregation

We introduce a new recursive aggregation procedure called Bernstein Online Aggregation (BOA). Its exponential weights include a second order refinement. The procedure is optimal for the model selection aggregation problem in the bounded iid setting for the square loss: the excess of risk of its batch version achieves the fast rate of convergence $$\log (M)/n$$log(M)/n in deviation. The BOA procedure is the first online algorithm that satisfies this optimal fast rate. The second order refinement is required to achieve the optimality in deviation as the classical exponential weights cannot be optimal, see Audibert (Advances in neural information processing systems. MIT Press, Cambridge, MA, 2007). This refinement is settled thanks to a new stochastic conversion that estimates the cumulative predictive risk in any stochastic environment with observable second order terms. The observable second order term is shown to be sufficiently small to assert the fast rate in the iid setting when the loss is Lipschitz and strongly convex. We also introduce a multiple learning rates version of BOA. This fully adaptive BOA procedure is also optimal, up to a $$\log \log (n)$$loglog(n) factor.

A hierarchical network formation model

We present a network formation model based on a particularly interesting class of networks in social settings, where individuals' positions are determined according to a topic-based or hierarchical taxonomy. In this game-theoretic model, players are located in the leaves of a complete b-ary tree as the seed network with the objective of minimizing their collective distances to others in the network. In the grid-based model of Even-Dar and Kearns [Even-Dar, E., and M. Kearns, A small world threshold for economic network formation, Advances in Neural Information Processing Systems 19 (2007), 385–392], they demonstrate the existence of small diameter networks with the threshold of α=2 where the cost of a new link depends on the distance between the two endpoints to the power of α. We show the appearance of small diameter equilibrium networks with the threshold of α=1/4 in the hierarchical or tree-based networks. Moreover, the general set of equilibrium networks in our model are guaranteed to exist and they are pairwise Nash stable with transfers [Bloch, F., and M. O. Jackson Definitions of equilibrium in network formation games, Int J Game Theory 34 (3) (2006), 305–318].

Discriminative part model for visual recognition

The recent literature on visual recognition and image classification has been mainly focused on Deep Convolutional Neural Networks (Deep CNN) [A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105.] and their variants, which has resulted in a significant progression of the performance of these algorithms.Building on these recent advances, this paper proposes to explicitly add translation and scale invariance to Deep CNN-based local representations, by introducing a new algorithm for image recognition which is modeling image categories as a collection of automatically discovered distinctive parts. These parts are matched across images while learning their visual model and are finally pooled to provide images signatures.The appearance model of the parts is learnt from the training images to allow the distinction between the categories to be recognized. A key ingredient of the approach is a softassign-like matching algorithm that simultaneously learns the model of each part and automatically assigns image regions to the model’s parts. Once the model of the category is trained, it can be used to classify new images by finding image’s regions similar to the learned parts and encoding them in a single compact signature.The experimental validation shows that the performance of the proposed approach is better than those of the latest Deep Convolutional Neural Networks approaches, hence providing state-of-the art results on several publicly available datasets.