First-order Information Research Articles

A stochastic moving ball approximation method for smooth convex constrained minimization

AbstractIn this paper, we consider constrained optimization problems with convex objective and smooth convex functional constraints. We propose a new stochastic gradient algorithm, called the Stochastic Moving Ball Approximation (SMBA) method, to solve this class of problems, where at each iteration we first take a (sub)gradient step for the objective function and then perform a projection step onto one ball approximation of a randomly chosen constraint. The computational simplicity of SMBA, which uses first-order information and considers only one constraint at a time, makes it suitable for large-scale problems with many functional constraints. We provide a convergence analysis for the SMBA algorithm using basic assumptions on the problem, that yields new convergence rates in both optimality and feasibility criteria evaluated at some average point. Our convergence proofs are novel since we need to deal properly with infeasible iterates and with quadratic upper approximations of constraints that may yield empty balls. We derive convergence rates of order $${\mathcal {O}} (k^{-1/2})$$ O ( k - 1 / 2 ) when the objective function is convex, and $${\mathcal {O}} (k^{-1})$$ O ( k - 1 ) when the objective function is strongly convex. Preliminary numerical experiments on quadratically constrained quadratic problems demonstrate the viability and performance of our method when compared to some existing state-of-the-art optimization methods and software.

Category-Level Object Pose Estimation with Statistic Attention.

Six-dimensional object pose estimation is a fundamental problem in the field of computer vision. Recently, category-level object pose estimation methods based on 3D-GC have made significant breakthroughs due to advancements in 3D-GC. However, current methods often fail to capture long-range dependencies, which are crucial for modeling complex and occluded object shapes. Additionally, discerning detailed differences between different objects is essential. Some existing methods utilize self-attention mechanisms or Transformer encoder-decoder structures to address the lack of long-range dependencies, but they only focus on first-order information of features, failing to explore more complex information and neglecting detailed differences between objects. In this paper, we propose SAPENet, which follows the 3D-GC architecture but replaces the 3D-GC in the encoder part with HS-layer to extract features and incorporates statistical attention to compute higher-order statistical information. Additionally, three sub-modules are designed for pose regression, point cloud reconstruction, and bounding box voting. The pose regression module also integrates statistical attention to leverage higher-order statistical information for modeling geometric relationships and aiding regression. Experiments demonstrate that our method achieves outstanding performance, attaining an mAP of 49.5 on the 5°2 cm metric, which is 3.4 higher than the baseline model. Our method achieves state-of-the-art (SOTA) performance on the REAL275 dataset.

On Necessary Optimality Conditions for Sets of Points in Multiobjective Optimization

Taking inspiration from what is commonly done in single-objective optimization, most local algorithms proposed for multiobjective optimization extend the classical iterative scalar methods and produce sequences of points able to converge to single efficient points. Recently, a growing number of local algorithms that build sequences of sets has been devised, following the real nature of multiobjective optimization, where the aim is that of approximating the efficient set. This calls for a new analysis of the necessary optimality conditions for multiobjective optimization. We explore conditions for sets of points that share the same features of the necessary optimality conditions for single-objective optimization. On the one hand, from a theoretical point of view, these conditions define properties that are necessarily satisfied by the (weakly) efficient set. On the other hand, from an algorithmic point of view, any set that does not satisfy the proposed conditions can be easily improved by using first-order information on some objective functions. We analyse both the unconstrained and the constrained case, giving some examples.

The epistemic role of vividness

Abstract The vividness of mental imagery is epistemically relevant. Intuitively, vivid and intense memories are epistemically better than weak and hazy memories, and using a clear and precise mental image in the service of spatial reasoning is epistemically better than using a blurry and imprecise mental image. But how is vividness epistemically relevant? I argue that vividness is higher-order evidence about one’s epistemic state, rather than first-order evidence about the world. More specifically, the vividness of a mental image is higher-order evidence about the amount of first-order information one has about its subject matter. When vividness is sufficiently low, it can give one reason to doubt the epistemic basis of the mental image and thereby act as a defeater. This account has important implications for theorizing about the epistemic roles of memory, imagination, mental imagery and phenomenal consciousness.

Online Identification of Nonlinear Systems With Separable Structure.

Separable nonlinear models (SNLMs) are of great importance in system modeling, signal processing, and machine learning because of their flexible structure and excellent description of nonlinear behaviors. The online identification of such models is quite challenging, and previous related work usually ignores the special structure where the estimated parameters can be partitioned into a linear and a nonlinear part. In this brief, we propose an efficient first-order recursive algorithm for SNLMs by introducing the variable projection (VP) step. The proposed algorithm utilizes the recursive least-squares method to eliminate the linear parameters, resulting in a reduced function. Then, the stochastic gradient descent (SGD) algorithm is employed to update the parameters of the reduced function. By considering the tight coupling relationship between linear parameters and nonlinear parameters, the proposed first-order VP algorithm is more efficient and robust than the traditional SGD algorithm and alternating optimization algorithm. More importantly, since the proposed algorithm just uses the first-order information, it is easier to apply it to large-scale models. Numerical results on examples of different sizes confirm the effectiveness and efficiency of the proposed algorithm.

Granitoids of the western Himalaya and Karakoram as potential geothermal reservoirs – A petrological, geochemical and petrophysical study

Hot springs in various granitoid and gneissic complexes in northern Pakistan indicate elevated geothermal gradients, which warrants their evaluation for geothermal applications. Evaluation of such prospects requires extensive knowledge of rock properties and their interaction with reservoir fluids. Our contribution provides first-order information on petrophysical, geochemical, and petrographic descriptions derived from outcrop analogs. About seventy samples of (mostly) granitoids and gneisses from the Nanga Parbat Massif (NPM), the Kohistan-Ladakh Batholith (KLB), and the Karakoram Batholith (KB) were collected. Biotite, K-feldspar, and plagioclase in the granitoid and gneiss show weak to moderate alteration, which increases in shear zones, suggesting fracture-assisted fluid interaction. Geochemical results indicate that the gneisses and granites of the NPM are mostly peraluminous S-type and show enrichment in most LREEs and depletion in HREEs. In addition, depleted Sr and Ba and enriched Rb and U in granites indicate partial melting and high fractionation. On the contrary, granitoids from the KLB are of calc-alkaline I-type and characterized by the depletion of REEs and the enrichment in Ba and Sr. The granitoids of the KB, due to their different magmatic histories, are more diverse, ranging from older I-type calc-alkaline granodiorite, and younger I-type alkaline syenite and S-type calc-alkaline granite. They show an overall enrichment in HREEs along with Ba, Th, Ta, Sr, and Lu. Allanites from syenite of the KB show significant concentrations of Th and U.Petrophysical measurements reveal low matrix porosities (0.6–3.5 %), with primarily average thermal conductivities (1.48–3.37 W m−1K −1) and thermal diffusivities (0.68–1.95 ∙10–6 m2 s−1), with minor variation in specific heat capacities (744–767 J kg−1 K−1). The NPM displays higher average thermal conductivity, thermal diffusivity, and lower porosity than the KB, while the petrophysical properties of the KLB range between these two domains.The geothermal systems in the area operate due to the thickening of a granitoid-dominant crust, which is enriched with radiogenic elements. The fault zones provide channels for meteoric water to access deeply these hot and thermally conductive rocks in the subsurface. After heat exchange, the water is discharged back to the surface as hot springs. The present data set provides a better understanding of regional geothermal regimes from petrological, geochemical, and petrophysical perspectives and assists in numerical modeling for potential geothermal assessment.

Communication-Efficient Zeroth-Order Adaptive Optimization for Federated Learning

Federated learning has become a prevalent distributed training paradigm, in which local devices collaboratively train learning models without exchanging local data. One of the most dominant frameworks of federated learning (FL) is FedAvg, since it is efficient and simple to implement; here, the first-order information is generally utilized to train the parameters of learning models. In practice, however, the gradient information may be unavailable or infeasible in some applications, such as federated black-box optimization problems. To solve the issue, we propose an innovative zeroth-order adaptive federated learning algorithm without using the gradient information, referred to as ZO-AdaFL, which integrates the zeroth-order optimization algorithm into the adaptive gradient method. Moreover, we also rigorously analyze the convergence behavior of ZO-AdaFL in a non-convex setting, i.e., where ZO-AdaFL achieves convergence to a region close to a stationary point at a speed of O(1/T) (T represents the total iteration number). Finally, to verify the performance of ZO-AdaFL, simulation experiments are performed using the MNIST and FMNIST datasets. Our experimental findings demonstrate that ZO-AdaFL outperforms other state-of-the-art zeroth-order FL approaches in terms of both effectiveness and efficiency.

The sampling method for optimal precursors of El Niño–Southern Oscillation events

Abstract. The El Niño–Southern Oscillation (ENSO) is a significant climate phenomenon that appears periodically in the tropical Pacific. The intermediate coupled ocean–atmosphere Zebiak–Cane (ZC) model is the first and classical one designed to numerically forecast the ENSO events. Traditionally, the conditional nonlinear optimal perturbation (CNOP) approach has been used to capture optimal precursors in practice. In this paper, based on state-of-the-art statistical machine learning techniques1, we investigate the sampling algorithm proposed in Shi and Sun (2023) to obtain optimal precursors via the CNOP approach in the ZC model. For the ZC model, or more generally, the numerical models with a large number O(104−105) of degrees of freedom, the numerical performance, regardless of the statically spatial patterns and the dynamical nonlinear time evolution behaviors as well as the corresponding quantities and indices, shows the high efficiency of the sampling method compared to the traditional adjoint method. The sampling algorithm does not only reduce the gradient (first-order information) to the objective function value (zeroth-order information) but also avoids the use of the adjoint model, which is hard to develop in the coupled ocean–atmosphere models and the parameterization models. In addition, based on the key characteristic that the samples are independently and identically distributed, we can implement the sampling algorithm by parallel computation to shorten the computation time. Meanwhile, we also show in the numerical experiments that the important features of optimal precursors can still be captured even when the number of samples is reduced sharply.

Fair Participation via Sequential Policies

Leading approaches to algorithmic fairness and policy-induced distribution shift are often misaligned with long-term objectives in sequential settings. We aim to correct these shortcomings by ensuring that both the objective and fairness constraints account for policy-induced distribution shift. First, we motivate this problem using an example in which individuals subject to algorithmic predictions modulate their willingness to participate with the policy maker. Fairness in this example is measured by the variance of group participation rates. Next, we develop a method for solving the resulting constrained, non-linear optimization problem and prove that this method converges to a fair, locally optimal policy given first-order information. Finally, we experimentally validate our claims in a semi-synthetic setting.

Structure-exploiting interior-point solver for high-dimensional entropy-sparsified regression learning

The solution of high-dimensional nonlinear regression problems through standard machine learning approaches often relies on first-order information, due to the numerical and memory challenges arising from the computation of the Hessian matrix and of the higher-order derivatives. While this scenario seems not favorable to second-order methods, here we show that an efficient and modular structure-exploiting interior-point solver can be successfully applied to the recently introduced class of entropy-based methods for regression learning. Specifically, by exploiting the favorable structure of the problem and of the Hessian matrix, we suggest a robust solution strategy based on explicit low-rank updates combined with an iterative Symmetric Quasi-Minimal Residual (SQMR) algorithm to solve the underlying system of linear equations. The results show that the proposed structure-exploiting solver – which relies on the hybrid parallelism and distributed-memory computing paradigm – allows a significant solution time speed-up with respect to a naive solution strategy. Furthermore, through an adequate use of the Message Passing Interface (MPI) and of Open Multi-Processing (OpenMP), the proposed solver enables the solution of large-scale problems on high-performance computing architectures consisting of thousands of compute nodes. The accompanying detailed convergence and performance analyses demonstrate both numerical robustness and high-performance capabilities for increasingly high-dimensional problems.

Successive Convexification for Nonlinear Model Predictive Control with Continuous-Time Constraint Satisfaction

We propose a nonlinear model predictive control (NMPC) framework based on a direct optimal control method that ensures continuous-time constraint satisfaction and accurate evaluation of the running cost, without compromising computational efficiency. We leverage the recently proposed successive convexification framework for trajectory optimization, where: (1) the path constraints and running cost are equivalently reformulated by augmenting the system dynamics, (2) multiple shooting is used for exact discretization, and (3) a convergence-guaranteed sequential convex programming (SCP) algorithm, the prox-linear method, is used to solve the discretized receding-horizon optimal control problems. The resulting NMPC framework is computationally efficient, owing to its support for warm-starting and premature termination of SCP, and its reliance on first-order information only. We demonstrate the effectiveness of the proposed NMPC framework by means of a numerical example with reference-tracking and obstacle avoidance. The implementation is available at https://github.com/UW-ACL/nmpc-ctcs.

Equilibrated Zeroth-Order Unrolled Deep Network for Parallel MR Imaging.

In recent times, model-driven deep learning has evolved an iterative algorithm into a cascade network by replacing the regularizer's first-order information, such as the (sub)gradient or proximal operator, with a network module. This approach offers greater explainability and predictability compared to typical data-driven networks. However, in theory, there is no assurance that a functional regularizer exists whose first-order information matches the substituted network module. This implies that the unrolled network output may not align with the regularization models. Furthermore, there are few established theories that guarantee global convergence and robustness (regularity) of unrolled networks under practical assumptions. To address this gap, we propose a safeguarded methodology for network unrolling. Specifically, for parallel MR imaging, we unroll a zeroth-order algorithm, where the network module serves as a regularizer itself, allowing the network output to be covered by a regularization model. Additionally, inspired by deep equilibrium models, we conduct the unrolled network before backpropagation to converge to a fixed point and then demonstrate that it can tightly approximate the actual MR image. We also prove that the proposed network is robust against noisy interferences if the measurement data contain noise. Finally, numerical experiments indicate that the proposed network consistently outperforms state-of-the-art MRI reconstruction methods, including traditional regularization and unrolled deep learning techniques.

Ensemble clustering with low-rank optimal Laplacian matrix learning

The co-association (CA) matrix that describes connection relationship between instances is of importance for ensemble clustering. Existing ensemble clustering methods demonstrate that Laplacian matrix can help to improve the quality of CA matrix and finally produce better clustering results. However, they usually consider the first-order information only and ignore the higher-order information that exists among base clustering results. To address this issue, this paper introduces the theory of higher-order connectivity in ensemble clustering for the first time and further proposes a low-rank optimal Laplacian matrix learning (LROLML) approach for ensemble clustering. Specifically, the proposed LROLML first constructs a set of multi-order Laplacian matrices and then learns the optimal Laplacian matrix from the Laplacian matrix set with different orders. The optimal Laplacian matrix mines more higher-order link information, thereby improving the ensemble performance. We use alternating direction method of multipliers (ADMM) to solve the optimization problem and then apply the average-link hierarchical agglomerative clustering to get the final result. We conducted extensive experiments on 20 popular datasets to validate the proposed LROLML, which is compared with two types of results of basic k-means clustering and seven state-of-the-art ensemble clustering models. The results demonstrate the LROLML is superior to the compared models, showing that the LROLML is promising for ensemble clustering.

An improved information volume of mass function based on plausibility transformation method

The information volume of mass function (IVMF) is an effective tool for measuring the uncertainty of basic probability assignments in power sets. However, the current IVMF will yield counterintuitive results when applied to an inconsistent frame of discernment (FOD). To address this issue, an improved IVMF based on plausibility transformation method (PTM) is proposed in this paper. Compared to existing methods, the proposed method yields a more reasonable result in cases where the FOD is inconsistent. Additionally, the proposed IVMF can be viewed as a geometric mean of first-order information volume and higher-order information volume, which can degenerate into Shannon Entropy in a probability distribution. The efficacy and rationality of the proposed IVMF are demonstrated through a series of numerical examples and an application in threat assessment.

Trial-by-trial feedback fails to improve the consideration of acceleration in visual time-to-collision estimation.

When judging the time-to-collision (TTC) of visually presented accelerating vehicles, untrained observers do not adequately account for acceleration (second-order information). Instead, their estimations only rely on vehicle distance and velocity (first-order information). As a result, they systemically overestimate the TTC for accelerating objects, which represents a potential risk for pedestrians in traffic situations because it might trigger unsafe road-crossing behavior. Can training help reduce these estimation errors? In this study, we tested whether training with trial-by-trial feedback about the signed deviation of the estimated from the actual TTC can improve TTC estimation accuracy for accelerating vehicles. Using a prediction-motion paradigm, we measured the estimated TTCs of twenty participants for constant-velocity and accelerated vehicle approaches, from a pedestrian's perspective in a VR traffic simulation. The experiment included three blocks, of which only the second block provided trial-by-trial feedback about the TTC estimation accuracy. Participants adjusted their estimations during and after the feedback, but they failed to differentiate between accelerated and constant-velocity approaches. Thus, the feedback did not help them account for acceleration. The results suggest that a safety training program based on trial-by-trial feedback is not a promising countermeasure against pedestrians' erroneous TTC estimation for accelerating objects.

High-order paired-ASPP for deep semantic segmentation networks

Current semantic segmentation models only exploit first-order information, while rarely exploring high-order semantics. However, traditional first-order statistics are insufficient to support a solid unanimous representation. In this paper, we propose High-Order Paired-ASPP (Atrous Spatial Pyramid Pooling) Network to exploit high-order statistics from various feature levels. The network first introduces a High-Order Representation module to extract the contextual high-order information from all stages of the backbone. They can provide more semantic clues and discriminative information than the first-order ones. Besides, a Paired-ASPP module is proposed to embed high-order statistics of the early stages into the last stage. It can further preserve the boundary-related and spatial context in the low-level features for final prediction. Our experiments show that the high-order semantics significantly boost the performance on confusing objects. Our method achieves competitive performance without bells and whistles on three benchmarks, i.e., Cityscapes, ADE20K and Pascal-Context.

DiFA: Differentiable Feature Acquisition

Feature acquisition in predictive modeling is an important task in many practical applications. For example, in patient health prediction, we do not fully observe their personal features and need to dynamically select features to acquire. Our goal is to acquire a small subset of features that maximize prediction performance. Recently, some works reformulated feature acquisition as a Markov decision process and applied reinforcement learning (RL) algorithms, where the reward reflects both prediction performance and feature acquisition cost. However, RL algorithms only use zeroth-order information on the reward, which leads to slow empirical convergence, especially when there are many actions (number of features) to consider. For predictive modeling, it is possible to use first-order information on the reward, i.e., gradients, since we are often given an already collected dataset. Therefore, we propose differentiable feature acquisition (DiFA), which uses a differentiable representation of the feature selection policy to enable gradients to flow from the prediction loss to the policy parameters. We conduct extensive experiments on various real-world datasets and show that DiFA significantly outperforms existing feature acquisition methods when the number of features is large.

On-the-Fly Control of Unknown Nonlinear Systems With Sublinear Regret

We study the problem of data-driven, constrained control of unknown nonlinear dynamics from a single ongoing and finite-horizon trajectory. We consider a one-step optimal control problem with a smooth, black-box objective, typically a composition of a known cost function and the unknown dynamics. We investigate an on-the-fly control paradigm, i.e., at each time step, the evolution of the dynamics and the first-order information of the cost are provided only for the executed control action. We propose an optimization-based control algorithm that iteratively minimizes a data-driven surrogate function for the unknown objective. We prove that the proposed approach incurs sublinear cumulative regret (step-wise suboptimality with respect to an optimal one-step controller) and is worst-case optimal among a broad class of data-driven control algorithms. We also present tractable reformulations of the approach that can leverage off-the-shelf solvers for efficient implementations.

Feature-blind attentional suppression of salient distractors.

A recent paper has reported, for the first time, that people are capable of suppressing salient singleton distractors of unknown color if the search task requires them to search for the most prevalent of several shapes in the display. We identify here several potential limitations of the earlier findings. In particular, in the reported experiments, the likelihood of a salient distractor was higher than what is typically studied, the distractor object was similar in shape to the relevant objects, only two colors were studied, the distractor was consistently a fixed shape, and the distractor was always a unique shape different from the search targets. Each of these limitations leaves open some questions about the generality of the findings. We address each of the concerns here, and show, in five experiments, that the ability to suppress distractors of unknown color is a robust finding that is not compromised by the potential limitations identified. When searching for the most prevalent of several shapes in a display, people can indeed suppress capture by otherwise-salient color singleton distractors even when their color is not known in advance (i.e., in a feature-blind manner), facilitating efficient search. The experiments confirm the ability to suppress visual elements based on second-order (e.g., a unique color) or global salience information, and not merely based on first-order (e.g., a specific color) information.

Multi-Scale Feature Fusion of Covariance Pooling Networks for Fine-Grained Visual Recognition.

Multi-scale feature fusion techniques and covariance pooling have been shown to have positive implications for completing computer vision tasks, including fine-grained image classification. However, existing algorithms that use multi-scale feature fusion techniques for fine-grained classification tend to consider only the first-order information of the features, failing to capture more discriminative features. Likewise, existing fine-grained classification algorithms using covariance pooling tend to focus only on the correlation between feature channels without considering how to better capture the global and local features of the image. Therefore, this paper proposes a multi-scale covariance pooling network (MSCPN) that can capture and better fuse features at different scales to generate more representative features. Experimental results on the CUB200 and MIT indoor67 datasets achieve state-of-the-art performance (CUB200: 94.31% and MIT indoor67: 92.11%).