Hierarchical Regularization Research Articles

Estimating substance use disparities across intersectional social positions using machine learning: An application of group-lasso interaction network.

An aim of quantitative intersectional research is to model the joint impact of multiple social positions on health risk behaviors. Although moderated multiple regression is frequently used to pursue intersectional research hypotheses, such parametric approaches may produce unreliable effect estimates due to data sparsity and high dimensionality. Machine learning provides viable alternatives, offering greater flexibility in evaluating many candidate interactions amid sparse data conditions, yet remains rarely employed. This study introduces group-lasso interaction network (glinternet), a novel machine learning approach involving hierarchical regularization, to assess intersectional differences in substance use prevalence. Utilizing variable selection and parameter stabilization functionality for main and interaction effects, glinternet was employed to examine two-way interactions between three primary social positions (gender, sexual orientation, and race) predicting heavy episodic drinking, cannabis use, and cigarette use prevalence. Analyses were conducted using the All of Us Research Program (N = 283,403), a national sample with high representation from populations historically underrepresented in biomedical research. Results were replicated using holdout cross-validation and compared against logistic regression estimates. Glinternet prevalence estimates were more stable across discovery and replication samples relative to logistic regression, particularly among sparsely represented groups. Prevalence estimates for cigarette and cannabis use were elevated among sexual minority and White cisgender women compared to heterosexual and non-White women, respectively. Glinternet may improve upon traditional moderated multiple regression methods for pursuing intersectional hypotheses by improving model parsimony and parameter stability, providing novel means for quantifying health disparities among intersectional social positions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Fast convergence strategy for ambiguous inverse problems based on hierarchical regularization

Mathematical modelling allows to predict the state of a physical system based on a set of parameters. The corresponding inverse problem, i.e., when parameter values are inferred based on constraints imposed on the system state, is commonly ill-posed. Here, we consider consistent underdetermined problems, where infinitely many combinations of possible parameter values achieve a state that accurately matches all constraints. For such problems a unique regularization strategy can be used, where the ambiguity of the solution space is reduced by sequentially incorporating additional constraints. As this regularization approach always yields a consistent problem by construction, we now show that the Gauss-Newton method is the prime choice to achieve fast convergence. Moreover, using the adjoint method allows to efficiently compute the required Jacobian matrix, which makes the overall solution approach ideally suited for large test cases with many constraints and unknown parameters. We present results for several illustrative examples related to network flow, for which we visualize the solution manifolds of the regularized inverse problem. This provides an intuitive explanation of the hierarchical approach to reduce the ambiguity of the solution. Furthermore, we confirm that combining our regularization strategy with a Gauss-Newton method results in an order of magnitude lower computational cost compared to a gradient descent algorithm. This highlights the potential of our regularization strategy in combination with the Gauss-Newton method, which likely is beneficial for many comparable inverse problems, especially with large parameter spaces.

Semantic-gap-oriented feature selection in hierarchical classification learning

Semantic gap is common issues in classification tasks. Different from traditional classification tasks, the semantic gap problem can introduce serious error propagation for hierarchical classification. In this paper, we propose a hierarchical feature selection method with label enhancement (HFSLE) to solve this challenge, which estimates the label distribution and selects subsets of features for each subtask. Firstly, HFSLE combines the label enhancement and hierarchical feature selection process so that the label enhancement benefits from the feedback of the loss function. Then, the label enhancement is constrained by the consistency of the parent-child distribution to obtain a plausible label distribution. Finally, feature selection is optimized based on intra-class consistency hierarchical regularization terms, which unifies feature subsets with hierarchical semantic descriptions to narrow the semantic gap. HFSLE gives a new perspective on the hierarchical feature selection based on label distribution rather than the original logical label. To demonstrate the superiority of the method, we compared it with six well-established feature selection methods on eight hierarchical data sets, and analyzed the convergence and robustness of the method. Extensive experiments demonstrate that the proposed method can perform favorably against advanced hierarchical feature selection approaches.

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

Modern software systems are usually highly configurable, providing users with customized functionality through various configuration options. Understanding how system performance varies with different option combinations is important to determine optimal configurations that meet specific requirements. Due to the complex interactions among multiple options and the high cost of performance measurement under a huge configuration space, it is challenging to study how different configurations influence the system performance. To address these challenges, we propose HINNPerf , a novel hierarchical interaction neural network for performance prediction of configurable systems. HINNPerf employs the embedding method and hierarchic network blocks to model the complicated interplay between configuration options, which improves the prediction accuracy of the method. In addition, we devise a hierarchical regularization strategy to enhance the model robustness. Empirical results on 10 real-world configurable systems show that our method statistically significantly outperforms state-of-the-art approaches by achieving average 22.67% improvement in prediction accuracy. In addition, combined with the Integrated Gradients method, the designed hierarchical architecture provides some insights about the interaction complexity and the significance of configuration options, which might help users and developers better understand how the configurable system works and efficiently identify significant options affecting the performance.

Hierarchical regularization networks for sparsification based learning on noisy datasets

Hierarchical regularization networks for sparsification based learning on noisy datasets

Hierarchical unsupervised multi-view feature selection

Multi-view feature selection is an important research direction in the field of multi-view learning. To remove the negative effect of irrelevant and redundant information and select the important features from the multi-view data, in this paper, we propose a novel hierarchical unsupervised multi-view feature selection method, which integrates matrix decomposition and hierarchical regularization as a joint model. Specifically, to exploit the consistency information of multiple views for projection matrix, we project the multiple views into a latent basis space based on a matrix decomposition model. The hierarchical regularization containing the [Formula: see text]-norm, dependence and Frobenius-norm is imposed on projection matrix to, respectively, exploit the row-level, dependency-level and view-level feature selection. We also develop an efficient optimization algorithm to optimize our method. Extensive experimental results on six popular multi-view datasets show the effectiveness and superiority of our method by comparing with the state-of-the-art methods.

HinCTI: A Cyber Threat Intelligence Modeling and Identification System Based on Heterogeneous Information Network

Cyber attacks have become increasingly complicated, persistent, organized, and weaponized. Faces with this situation, drives a rising number of organizations across the world are showing a growing willingness to leverage the open exchange of cyber threat intelligence (CTI) for obtaining a full picture of the fast-evolving cyber threat situation and protecting themselves against cyber-attacks. However, modeling CTI is challenging due to the explicit and implicit relationships among CTI and the heterogeneity of cyber-threat infrastructure nodes involved in CTI. Owing to the limited labels of cyber threat infrastructure nodes involved in CTI, automatically identifying the threat type of infrastructure nodes for early warning is also challenging. To tackle these challenges, a practical system called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HinCTI</i> is developed for modeling cyber threat intelligence and identifying threat types. We first design a threat intelligence meta-schema to depict the semantic relatedness of infrastructure nodes. We then model cyber threat intelligence on heterogeneous information network (HIN), which can integrate various types of infrastructure nodes and rich relations among them. Following, we define a meta-path and meta-graph instances-based threat Infrastructure similarity (MIIS) measure between threat infrastructure nodes and present a MIIS measure-based heterogeneous graph convolutional network (GCN) approach to identify the threat types of infrastructure nodes involved in CTI. Moreover, through the hierarchical regularization strategy, our model can alleviate the problem of overfitting and achieve good results in the threat type identification of infrastructure nodes. To the best of our knowledge, this work is the first to model CTI on HIN for threat identification and propose a heterogeneous GCN-based approach for threat type identification of infrastructure nodes. With <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">HinCTI</i> , comprehensive experiments are conducted on real-world datasets, and experimental results demonstrate that our proposed approach can significantly improve the performance of threat type identification compared to the existing state-of-the-art baseline methods. Our work is beneficial to greatly relieve security analysts from heavy analysis work and efficiently protect organizations against cyber-attacks.

Hierarchical regularization of solution ambiguity in underdetermined inverse and optimization problems

Estimating modeling parameters based on a prescribed optimization target requires to solve an inverse problem, which is commonly ill-posed. Consequently, either infinitely many or no solutions may exist, depending on whether the system is under- or overdetermined, and whether it is consistent or inconsistent. This paper focuses on scenarios where the solution is ambiguous and infinitely many combinations of possible parameter values can accurately achieve the optimization target. Selecting the most suitable solution requires incorporating additional constraints into the model, which is achieved by regularizing the inverse problem. However, common regularization approaches require the specification of a priori unknown regularization hyperparameters that are difficult and tedious to obtain, and can have a large impact on the result.Here, a novel strategy to reduce the ambiguity of such inverse problems is presented, ensuring that the primary optimization target is always reached accurately. To further reduce the solution space, additional constraints are included, until the optimal modeling parameters are found. Importantly, the required regularization parameters have a direct physical meaning and can be derived sequentially, starting from an initial guess that can be obtained conveniently by solving the system without regularization.By considering several illustrative examples, the applicability of the method is demonstrated, and its potential for various comparable inverse problems is highlighted.

Integration of survival data from multiple studies.

We introduce a statistical procedure that integrates datasets from multiple biomedical studies to predict patients' survival, based on individual clinical and genomic profiles. The proposed procedure accounts for potential differences in the relation between predictors and outcomes across studies, due to distinct patient populations, treatments and technologies to measure outcomes and biomarkers. These differences are modeled explicitly with study-specific parameters. We use hierarchical regularization to shrink the study-specific parameters towards each other and to borrow information across studies. The estimation of the study-specific parameters utilizes a similarity matrix, which summarizes differences and similarities of the relations between covariates and outcomes across studies. We illustrate the method in a simulation study and using a collection of gene expression datasets in ovarian cancer. We show that the proposed model increases the accuracy of survival predictions compared to alternative meta-analyticmethods.

Assigning ICD-O-3 Codes to Pathology Reports using Neural Multi-Task Training with Hierarchical Regularization.

Tracking population-level cancer information is essential for researchers, clinicians, policymakers, and the public. Unfortunately, much of the information is stored as unstructured data in pathology reports. Thus, too process the information, we require either automated extraction techniques or manual curation. Moreover, many of the cancer-related concepts appear infrequently in real-world training datasets. Automated extraction is difficult because of the limited data. This study introduces a novel technique that incorporates structured expert knowledge to improve histology and topography code classification models. Using pathology reports collected from the Kentucky Cancer Registry, we introduce a novel multi-task training approach with hierarchical regularization that incorporates structured information about the International Classification of Diseases for Oncology, 3rd Edition classes to improve predictive performance. Overall, we find that our method improves both micro and macro F1. For macro F1, we achieve up to a 6% absolute improvement for topography codes and up to 4% absolute improvement for histology codes.

A physics-informed and hierarchically regularized data-driven model for predicting fluid flow through porous media

This paper presents a new deep learning data-driven model for predicting structure dependent pore-fluid velocity fields in rock. The model is based on a Convolutional Auto-Encoder (CAE) artificial neural network capable of learning from image data generated by direct numerical simulations of fluid flow through pore-structures, such as by Lattice Boltzmann or molecular dynamics methods. The main novelty of the model in comparison to previous CAE-based data-driven approaches consists of three parts. The first is a methodology for decomposing the full-domain of the porous media into sub-regions, or “sub-domains”, in order to reduce the overall size of the CAE, batch process the sub-domains in parallel, and enable the CAE to learn local and generalizable nonlinear mappings of pore-fluid velocities. The second consists of embedding the finite difference solutions of the incompressible Navier-Stokes and continuity equations into convolutional layers prior to the CAE in order to provide the CAE with knowledge of fluid dynamics physics (PhyFlow). The third main novelty is that the training of the CAE is regularized with a hierarchical loss function that encourages the learning of fluid flow patterns (in a way similar to ranked modes in principal component analysis), ranking from most to least important. This is shown to increase the stability in learning, reduce over-fitting, and promote interpretability of the CAE neural network layers (HierCAE). The comprehensive new data-driven model, which we call the PhyFlow-HierCAE model, is shown to exhibit improved accuracy and generalizability of flow field predictions over conventional CAE models, attributable to the embedded physical knowledge and the hierarchical regularization, as well as realize orders of magnitude speed-ups in computation times as a surrogate for the direct numerical simulations. Examples of training and forward predictions on unseen pore-structures are provided and evaluated for data from Lattice Boltzmann and molecular dynamics simulations of pore-fluid flow. The model is shown to be a fast and accurate emulator (or “surrogate”) for predicting effective permeability of unseen pore-structures based on learning from relatively small direct numerical simulation datasets.

An enhanced convolutional neural network for bearing fault diagnosis based on time–frequency image

Deep learning theory has been widely used for diagnosing bearing faults. However, this method still has same drawbacks. For example, single time or frequency domain analysis methods cannot effectively extract features, the ReLU function is greatly affected by the learning rate, and it is difficult to achieve satisfactory results using the same regularization for different layers. To overcome the aforementioned deficiencies: (1) short-time Fourier transform theory to obtain an input image, (2) the scaled exponential linear unit (SELU) function is introduced to avoid excessive “dead” nodes during the training process, and (3) the use of hierarchical regularization to obtain better training results. Small sample datasets were used for the test experiment in two bearing fault simulators. The experiment results showed that the proposed method has a higher fault diagnosis accuracy than existing deep learning diagnosis methods.

A Recursive Regularization Based Feature Selection Framework for Hierarchical Classification

The sizes of datasets in terms of the number of samples, features, and classes have dramatically increased in recent years. In particular, there usually exists a hierarchical structure among class labels as hundreds of classes exist in a classification task. We call these tasks hierarchical classification, and hierarchical structures are helpful for dividing a very large task into a collection of relatively small subtasks. Various algorithms have been developed to select informative features for flat classification. However, these algorithms ignore the semantic hyponymy in the directory of hierarchical classes, and select a uniform subset of the features for all classes. In this paper, we propose a new feature selection framework with recursive regularization for hierarchical classification. This framework takes the hierarchical information of the class structure into account. In contrast to flat feature selection, we select different feature subsets for each node in a hierarchical tree structure with recursive regularization. The proposed framework uses parent-child, sibling, and family relationships for hierarchical regularization. By imposing <inline-formula><tex-math notation="LaTeX">$\ell _{2,1}$</tex-math></inline-formula> -norm regularization to different parts of the hierarchical classes, we can learn a sparse matrix for the feature ranking at each node. Extensive experiments on public datasets demonstrate the effectiveness and efficiency of the proposed algorithms.

Hierarchical Regularization of Building Boundaries in Noisy Aerial Laser Scanning and Photogrammetric Point Clouds

Aerial laser scanning or photogrammetric point clouds are often noisy at building boundaries. In order to produce regularized polygons from such noisy point clouds, this study proposes a hierarchical regularization method for the boundary points. Beginning with detected planar structures from raw point clouds, two stages of regularization are employed. In the first stage, the boundary points of an individual plane are consolidated locally by shifting them along their refined normal vector to resist noise, and then grouped into piecewise smooth segments. In the second stage, global regularities among different segments from different planes are softly enforced through a labeling process, in which the same label represents parallel or orthogonal segments. This is formulated as a Markov random field and solved efficiently via graph cut. The performance of the proposed method is evaluated for extracting 2D footprints and 3D polygons of buildings in metropolitan area. The results reveal that the proposed method is superior to the state-of-art methods both qualitatively and quantitatively in compactness. The simplified polygons could fit the original boundary points with an average residuals of 0.2 m, and in the meantime reduce up to 90% complexities of the edges. The satisfactory performances of the proposed method show a promising potential for 3D reconstruction of polygonal models from noisy point clouds.

Deep hierarchical guidance and regularization learning for end-to-end depth estimation

In this work, we propose a novel deep Hierarchical Guidance and Regularization (HGR) learning framework for end-to-end monocular depth estimation, which well integrates a hierarchical depth guidance network and a hierarchical regularization learning method for fine-grained depth prediction. The two properties in our proposed HGR framework can be summarized as: (1) the hierarchical depth guidance network automatically learns hierarchical depth representations by supervision guidance and multiple side conv-operations from the basic CNN, leveraging the learned hierarchical depth representations to progressively guide the upsampling and prediction process of upper deconv-layers; (2) the hierarchical regularization learning method integrates various-level information of depth maps, optimizing the network to predict depth maps with similar structure to ground truth. Comprehensive evaluations over three public benchmark datasets (including NYU Depth V2, KITTI and Make3D datasets) well demonstrate the state-of-the-art performance of our proposed depth estimation framework.

Spatial Event Forecasting in Social Media With Geographically Hierarchical Regularization

Social media has been utilized as a significant surrogate for spatial societal event forecasting. The accuracy and discernibility of a spatial event forecasting model are two key concerns, as they determine how accurate and how detailed the model's predictions will be. Existing research focuses almost exclusively on the accuracy alone, seldom considering the accuracy and discernibility simultaneously because this would require a considerably more sophisticated model while suffering from several challenges, namely: 1) the precise formulation of the tradeoff between accuracy and discernibility; 2) the scarcity of social media data with a high spatial resolution; and 3) the characterization of spatial correlation and heterogeneity. This paper proposes a novel feature learning framework that concurrently addresses all the above challenges by formulating prediction tasks for different locations with different spatial resolutions, allowing the heterogeneous relationships among the tasks to be characterized. This characterization is then integrated into our new models based on multitask learning, with parameters optimized by our proposed algorithm based on the alternative direction method of multipliers (ADMM) and dynamic programming. Extensive experimental evaluations performed on several data sets from different domains demonstrated the effectiveness of our proposed approach.

Informative Sensor and Feature Selection via Hierarchical Nonnegative Garrote

Placing sensors in every station of a process or every element of a system to monistor its state or performance is usually too expensive or physically impossible. Therefore, a systematic method is needed to select important sensing variables. The method should not only be capable of identifying important sensors/signals among multistream signals from a distributed sensing system, but should also be able to extract a small set of interpretable features from the high-dimensional vector of a selected signal. For this purpose, we develop a new hierarchical regularization approach called hierarchical nonnegative garrote (NNG). At the first level of hierarchy, a group NNG is used to select important signals, and at the second level, the individual features within each signal are selected using a modified version of NNG that possesses good properties for the estimated coefficients. Performance of the proposed method is evaluated and compared with other existing methods through Monte Carlo simulation. A case study is conducted to demonstrate the proposed methodology that can be applied to develop a predictive model for the assessment of vehicle design comfort based on the tested drivers’ motion trajectory signals. This article has supplementary material online.

Evaluation of neighborhood center attributes on resident’s territoriality and sense of belonging a case study in Boshrooyeh, Iran

Hierarchical regularity is achieved with intelligible definition of the public and private realms. Territoriality understood as a spatial strategy includes the point where physical attributes and clear boundaries are intertwined with the people belonging to the place. For this purpose, effective domains of the territoriality have been analyzed, each in term of distinct aspects of the physical attributes of neighborhood center through survey. Surveys were conducted with 174 participants in the Miandeh Neighborhood Center of Boshrooyeh city in Iran. Analysis revealed that the arrangement of houses, indicators buildings, and short walking distance as the characteristics of space hierarchy encourages a feeling of defense. The paper concludes by establishing that the traditional neighborhood center design has a significant contribution to the sense of territoriality, therefore it should be considered in the design of urban spaces.

Hierarchically penalized additive hazards model with diverging number of parameters

In many applications, covariates can be naturally grouped. For example, for gene expression data analysis, genes belonging to the same pathway might be viewed as a group. This paper studies variable selection problem for censored survival data in the additive hazards model when covariates are grouped. A hierarchical regularization method is proposed to simultaneously estimate parameters and select important variables at both the group level and the within-group level. For the situations in which the number of parameters tends to ∞ as the sample size increases, we establish an oracle property and asymptotic normality property of the proposed estimators. Numerical results indicate that the hierarchically penalized method performs better than some existing methods such as lasso, smoothly clipped absolute deviation (SCAD) and adaptive lasso.

Techniques for regularization parameter and hyper-parameter selection in PET and SPECT imaging

Penalized maximum likelihood methods are commonly used in positron emission tomography (PET) and single photon emission computed tomography (SPECT). Due to the fact that a Poisson data-noise model is typically assumed, standard regularization parameter choice methods, such as the discrepancy principle or generalized cross validation, cannot be directly applied. In the recent work of the authors, regularization parameter choice methods for penalized negative-log Poisson likelihood problems are introduced. In this article, we apply these methods to the applications of PET and SPECT, introducing a modification that improves the performance of the methods. We then demonstrate how these techniques can be used to choose the hyper-parameters in a Bayesian hierarchical regularization approach.