Hierarchical Reduction Research Articles

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT. Our code is released at https://github.com/ZLKong/Tri-Level-ViT

Research Progress on Carbon Dioxide Reduction Coupled with the Formation of C−O Bonds to Oxygenated Compounds

AbstractIn recent years, carbon dioxide (CO2) has been released into the atmosphere resulting in rising CO2 concentration, which has a great impact on climate problems such as global warming. Carbon dioxide is a rich, cheap and easily available renewable resource. The reduction of carbon dioxide as a C1 source into synthetic fuels or high‐value chemicals has aroused great interest among chemists. Over the past few decades, great progress has been made in carbon dioxide reduction and C−C, C−N, C−S bonds construction, but there are still great challenges in the hierarchical reduction of carbon dioxide coupled with the formation of C−O bonds. In this review, we summarized the research progress of carbon dioxide reduction coupled with the formation of C−O bonds to oxygenated compounds in alcohols andprospected the opportunities and challenges in this field.

A hierarchical reduction method with selective targets for combustion modeling of large Hydrocarbons: A case study of 1,3,5-Trimethylbenzene skeletal mechanism

A novel hierarchical method of skeletal reduction for detailed hydrocarbon reaction mechanisms is proposed and applied to generate a skeletal mechanism of 1,3,5-trimethylbenzene (135TMB) which is relevant to practical fuels and their surrogates. The proposed reduction method is based on the hierarchical sensitivity analysis (HSA) implemented in a top-to-bottom manner in terms of carbon levels with selective reduction targets, such as autoignition delay time, laminar flame speed, flame extinction, speciation, etc. These reduction targets can be selected based on the desired combustion phenomena to be simulated and are only validated at their corresponding carbon levels where the respective chemistries are dominating. In contrast to the conventional global sensitivity analysis (GSA) in which species and reactions are randomly evaluated, HSA evaluates species and reactions following the oxidation sequence of fuel molecule, and therefore is able to generate skeletal mechanisms as compact as possible with significantly higher efficiency than GSA. In addition, quasi-steady-state approximation and reaction lumping are integrated in the current reduction process. Low-temperature reaction pathway is also optimized with reaction rates tuned using a particle swarm algorithm to better predict species fractions in the low-to-intermediate temperature regime. As such, based on the detailed 135TMB mechanism of Diévart et al. (2013) with 450 species and 4569 reactions, a skeletal mechanism with 46 species and 197 reactions has been successfully generated including both high- and low-temperature reaction pathways. Extensive validations of the resulting skeletal mechanism are performed against the detailed mechanism predictions and the literature experimental data over a wide range of conditions, and good agreements are observed. These results not only show the accuracy and reliability of the present skeletal 135TMB mechanism but also demonstrate the effectiveness and reduction efficiency of the proposed hierarchical reduction method.

Bronze age stone flaking at Saruq al-Hadid, Dubai, southeastern Arabia.

Excavations at Saruq al-Hadid, Dubai, UAE, discovered a stone tool technology with backed microliths dating to the Wadi Suq period and Late Bronze Age (ca. 1750–1300 BCE). The stone technology is a contemporary with metal production in the region, and the assemblage was recovered from a thick bone midden deposit at this multi-period site on the edge of the Rub’ al-Khali Desert. Small cobbles of chert were imported to the site and were reduced into flakes by hard-hammer percussion. Cores were frequently rotated during knapping and the reduction strategy was ad hoc, lacking hierarchical reduction stages. Flake tools were used as-is or modified by retouching. Some flakes were selected for backing into geometric microliths, and backing techniques often reflected high levels of stoneworking skill to produce stylised scalene shapes. A review of contemporary archaeological evidence, and the context of the Saruq al-Hadid assemblage, suggest that microliths may have been made as stone armatures for arrows despite the contemporary use of copper-based arrowheads.

A Hierarchical Reduction Method with Selective Targets for Combustion Modeling of Large Hydrocarbons a Case Study of 1,3,5-Trimethylbenzene Skeletal Mechanism

A Hierarchical Reduction Method with Selective Targets for Combustion Modeling of Large Hydrocarbons a Case Study of 1,3,5-Trimethylbenzene Skeletal Mechanism

Catalyst-free hierarchical reduction of CO2 with BH3N(C2H5)3 for selective N-methylation and N-formylation of amines

We describe herein an efficient and catalyst-free methodology for N-formylation and N-methylation of various amines using BH3N(C2H5)3 as a reductant and CO2 as C1 source. This system is compatible with a wide range of substrates, such as aromatic secondary amines, aliphatic secondary amines, heterocyclic amines, and primary amines. The desired methylamines and formamides are obtained in excellent yields by changing the reaction solvent, temperature, and the amount of BH3N(C2H5)3. The optimum yields of methylamines and formamides are up to 99 %, which are comparable to the data using different catalysts. Moreover, it has been demonstrated by mechanistic research that CO2 was activated by inserting into B–H bonds of BH3N(C2H5)3 and formamide can be regarded as an intermediate species during the N-methylation formation.

An approximate high-dimensional optimization method using hierarchical design space reduction strategy

To overcome the complicated engineering model and huge computational cost, a hierarchical design space reduction strategy based approximate high-dimensional optimization(HSRAHO) method is proposed to deal with the high-dimensional expensive black-box problems. Three classical surrogate models including polynomial response surfaces, radial basis functions and Kriging are selected as the component surrogate models. The ensemble of surrogates is constructed using the optimized weight factors selection method based on the prediction sum of squares and employed to replace the real high-dimensional black-box models. The hierarchical design space reduction strategy is used to identify the design subspaces according to the known information. And, the new promising sample points are generated in the design subspaces. Thus, the prediction accuracy of ensemble of surrogates in these interesting sub-regions can be gradually improved until the optimization convergence. Testing using several benchmark optimization functions and an airfoil design optimization problem, the newly proposed approximate high-dimensional optimization method HSRAHO shows improved capability in high-dimensional optimization efficiency and identifying the global optimum.

Abstract LB-241: Kynurenine metabolism as a biomarker and therapeutic target in hepatocellular carcinoma (HCC)

Abstract Background: Hepatocellular carcinoma (HCC) is the most prevalent cancer in the liver, the 5th more common cause of deaths by cancer in the US, and the 3th worldwide. Tumor heterogeneity, few targeted therapies, and late detection make the overall prognosis of patients unsatisfactory and demonstrate the need for better early-detection biomarkers. Liver, by its nature, is the major site of amino acid processing and highly regulated by metabolic signaling. In particular, the majority of ingested tryptophan is processed in the liver through the kynurenine pathway, the endpoint of which is de novo NAD+ biosynthesis. Cumulative evidence shows that dysregulation tryptophan-kynurenine metabolism may promote tumor reprogramming and carcinogenesis. In this work, we provide new insight on how kynurenine pathway can be used as biomarker for patient prognosis. Methods: Using a publicly available gene expression dataset from liver hepatocellular carcinoma (LIHC) samples available through The Cancer Genome Atlas (TCGA), we employ Principal Component Analysis (PCA) to show that expression of kynurenine pathway gene in cancer cells can be used to cluster patients with particular clinical profiles and predict prognosis. Hierarchical clustering and dimensionality reduction demonstrate the segregation of cluster patients based on gene-pattern expression profiles. Kaplan-Meier survival curves were used to determine patient overall survival and Wilcoxon Signed Rank test to prove statistical significance. Results: Principal Component 1 (PC1) explains 14.5% gene variance among the patients while PC2 explain 9.2% of variance. Clustering genes by expression across patients, IDO, IDO2, AADAT, KMO, CCBL1, CCBL2, KYNU and ACMSD form one cluster, HAAO and AFMID form a second, and QPRT and TDO2 fall outside of clear cluster boundaries. Hierarchical k-means clustering of the patients based on the kynurenine gene-expression profiled segregated patients into four clusters. Heatmap representation of z-score values reveal that each cluster represents tumors with elevated expression of a specific gene or pair of genes: TDO, HAAO/AFMID, QPRT, and KYNU. Other kynurenine pathway genes appear to have stable expression among the population of HCC patients. Dimensionality reduction using t-Distributed Stochastic Neighbor Embedding (t-SNE) identified the same distribution as hierarchical clustering. One of the outcomes that was clearly determined by this segregation among HCC patients was overall survival. Using the total number of patients from the dataset (n = 371), patients with high QPRT expression had poor prognosis with increased median mortality, with no effect in the maximum survival. There is a significant decrease in the survival between patients in cluster 3 (with high QPRT expression) and those in cluster 2 (with high HAAO/AFMID expression) (HR = 1.2, [95% CI 0.5-1.8] P = 0.0181, Gehan-Breslow-Wilcoxon Test). Furthermore, patients with high vs low QPRT expression have significantly different survival rates (HR = 1.4, [95% CI 0.9-2.2] P = 0.0344, Gehan-Breslow-Wilcoxon Test). Conclusions: Kynurenine metabolism can be used to determine patient prognosis among HCC patients. In ongoing work, we are testing the hypothesis that inhibition of QPRT with phthalic acid will sensitize HepG2 to cisplatin-induced apoptosis, to validate our prediction model. Citation Format: Raul Castro-Portuguez, Samuel Freitas, George L. Sutphin. Kynurenine metabolism as a biomarker and therapeutic target in hepatocellular carcinoma (HCC) [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr LB-241.

Expressive Attribute-Based Encryption with Constant-Size Ciphertexts from the Decisional Linear Assumption

We propose a key-policy attribute-based encryption (KP-ABE) scheme with constant-size ciphertexts, whose selective security is proven under the decisional linear (DLIN) assumption in the standard model. The proposed scheme also has semi-adaptively security, which is a recently proposed notion of security. The access structure is expressive, that is given by non-monotone span programs. It also has fast decryption, i.e., a decryption includes only a constant number of pairing operations. As an application of our KP-ABE construction, we also propose a fully secure attribute-based signatures with constant-size secret (signing) keys from the DLIN. For achieving the above results, we employ a hierarchical reduction technique on dual pairing vector spaces and a modified form of pairwise independence lemma specific to our proposed schemes.

Fully Secure Functional Encryption with a Large Class of Relations from the Decisional Linear Assumption

This paper presents a fully secure (adaptively secure) practical functional encryption scheme for a large class of relations, that are specified by non-monotone access structures combined with inner-product relations. The security is proven under a standard assumption, the decisional linear assumption, in the standard model. Our scheme is constructed on the concept of dual pairing vector spaces and a hierarchical reduction technique on this concept is employed for the security proof. The proposed functional encryption scheme covers, as special cases, (1) key-policy, ciphertext-policy and unified-policy attribute-based encryption with non-monotone access structures, (2) (hierarchical) attribute-hiding functional encryption with inner-product relations and functional encryption with nonzero inner-product relations and (3) spatial encryption and a more general class of encryption than spatial encryption.

Zwitterionic Covalent Organic Frameworks as Catalysts for Hierarchical Reduction of CO2 with Amine and Hydrosilane.

Controllable hierarchical reduction of carbon dioxide (CO2) to selectively afford versatile chemicals with specific carbon oxidation state is important but still remains a huge challenge to be realized. Here, we report new zwitterionic covalent organic frameworks ([BE] X%-TD-COFs), prepared by introducing betaine groups (BE) onto the channel walls of presynthesized frameworks via pore surface engineering methodology, as the heterogeneous organocatalysts for CO2 reduction. The adjustable density of immobilized BE groups as well as good preservation of crystallinity and porosity inherited from their parent COFs endow [BE] X%-TD-COFs with highly ordered catalytic site distribution and one-dimensional mass transport pathway in favor of catalysis. By controlling the reaction temperature and amount of CO2, [BE] X%-TD-COFs present high activity in catalyzing reduction of CO2 with amine and phenylsilane (PhSiH3) to produce formamides, aminals, and methylamines, respectively, with high yield and selectivity. Furthermore, high stability and insolubility bring excellent reusability to [BE]X%-TD-COFs with well-maintained catalytic performance after four cycles of use. Notably, this is a novel example that COFs are developed as heterogeneous catalysts for hierarchical two-, four-, and six-electron reduction of CO2 with amines and PhSiH3 to form C-N bonds as well as afford C+II, C0, and C-II species efficiently and selectively.

Identification of Power Transformer Winding Fault Types by a Hierarchical Dimension Reduction Classifier

Frequency response analysis (FRA) demonstrates significant advantages in the diagnosis of transformer winding faults. The instrument market desires intelligent diagnostic functions to ensure that the FRA technique is more practically useful. In this paper, a hierarchical dimension reduction (HDR) classifier is proposed to identify types of typical incipient winding faults. The classifier procedure is hierarchical. First, measured frequency response (FR) curves are preprocessed using binarization and binary erosion to normalize FR data. Second, the pre-processed data are divided into groups according to the definition of dynamic frequency sub-bands. Then, hybrid algorithms comprised of two conventional and two novel quantitative indices are used to reduce the dimension of the FR data and extract the features for identifying typical types of transformer winding faults. The classifier provides an integration of a priori expertise and quantitative analysis in the furtherance of the automatic identification of FR data. Twenty-six sets of FR data from different types of power transformers with multiple types of winding faults were collected from an experimental simulation, literature, and real tests performed by a grid company. Finally, real case studies were conducted to verify the performance of the HDR classifier in the automatic identification of transformer winding faults.

Sensor selection scheme in activity recognition based on hierarchical feature reduction

To better understand the activity state of human, we might need multiple sensors on different parts of the body. According to different types of activities, the number and slot of required sensors would also be different. Therefore, how to determine the number and slot of necessary sensors regarding to wearers’ experience and processing efficiency is a meaningful study in actual practice. In this work, we propose a novel sensor selection scheme that is based on the improvement of the feature reduction process of the recognition. This scheme applies a hierarchical feature reduction method based on mutual information with max relevance and low-dimensional embedding strategy. It divides the process of feature reduction into two stages: first, redundant sensors are removed with one-order sequential forward selection based on mutual information; second, feature selection strategy that maximizing class-relevance is integrated with low-dimensional mapping so that the set of features will be further compressed. To verify the feasibility and superiority of the scheme, we design a complete solution for real practice of human activity recognition. According to the results of the experiments, we are able to recognize human activities accurately and efficiently with as few sensors as possible.

Transition-Metal-Free Catalysis for the Reductive ­Functionalization of CO2 with Amines

Reductive functionalization of CO2 with amines and a reductant, which combines both reduction of CO2 and C–N bond formation in one pot to produce versatile chemicals and energy-storage materials such as formamides, aminals, and methylamines that are usually derived from petroleum feedstock, would be appealing and promising. Herein, we give a brief review on recent developments in the titled CO2 chemistry by employing transition-metal-free catalysis, which can be catalogued as below according to the diversified energy content of the products, that is formamides, aminals, and methylamines being consistent with 2-, 4-, and 6-electron reduction of CO2, respectively. Notably, hierarchical reduction of CO2 with amines to afford at least two products, for example, formamides and methylamines, could be realized with the same catalyst through tuning the hydrosilane type, reaction temperature, or CO2 pressure. Finally, the opportunities and challenges of the reductive functionalization of CO2 with amines are also highlighted.1 Introduction2 2-Electron Reduction of CO2 to Formamide3 6-Electron Reduction of CO2 to Methylamine4 4-Electron Reduction of CO2 to Aminal5 Hierarchical Reduction of CO2 with Amines6 Conclusion

Betaine Catalysis for Hierarchical Reduction of CO2 with Amines and Hydrosilane To Form Formamides, Aminals, and Methylamines

AbstractAn efficient, sustainable organocatalyst, glycine betaine, was developed for the reductive functionalization of CO2 with amines and diphenylsilane. Methylamines and formamides were obtained in high yield by tuning the CO2 pressure and reaction temperature. Based on identification of the key intermediate, that is, the aminal, an alternative mechanism for methylation involving the C0 silyl acetal and aminal is proposed. Furthermore, reducing the CO2 amount afforded aminals with high yield and selectivity. Therefore, betaine catalysis affords products with a diversified energy content that is, formamides, aminals and methylamines, by hierarchical two‐, four‐ and six‐electron reduction, respectively, of CO2 coupled with C−N bond formation.

Betaine Catalysis for Hierarchical Reduction of CO2 with Amines and Hydrosilane To Form Formamides, Aminals, and Methylamines.

An efficient, sustainable organocatalyst, glycine betaine, was developed for the reductive functionalization of CO2 with amines and diphenylsilane. Methylamines and formamides were obtained in high yield by tuning the CO2 pressure and reaction temperature. Based on identification of the key intermediate, that is, the aminal, an alternative mechanism for methylation involving the C0 silyl acetal and aminal is proposed. Furthermore, reducing the CO2 amount afforded aminals with high yield and selectivity. Therefore, betaine catalysis affords products with a diversified energy content that is, formamides, aminals and methylamines, by hierarchical two-, four- and six-electron reduction, respectively, of CO2 coupled with C-N bond formation.

S-Caffe

Availability of large data sets like ImageNet and massively parallel computation support in modern HPC devices like NVIDIA GPUs have fueled a renewed interest in Deep Learning (DL) algorithms. This has triggered the development of DL frameworks like Caffe, Torch, TensorFlow, and CNTK. However, most DL frameworks have been limited to a single node. In order to scale out DL frameworks and bring HPC capabilities to the DL arena, we propose, S-Caffe; a scalable and distributed Caffe adaptation for modern multi-GPU clusters. With an in-depth analysis of new requirements brought forward by the DL frameworks and limitations of current communication runtimes, we present a co-design of the Caffe framework and the MVAPICH2-GDR MPI runtime. Using the co-design methodology, we modify Caffe's workflow to maximize the overlap of computation and communication with multi-stage data propagation and gradient aggregation schemes. We bring DL-Awareness to the MPI runtime by proposing a hierarchical reduction design that benefits from CUDA-Aware features and provides up to a massive 133x speedup over OpenMPI and 2.6x speedup over MVAPICH2 for 160 GPUs. S-Caffe successfully scales up to 160 K-80 GPUs for GoogLeNet (ImageNet) with a speedup of 2.5x over 32 GPUs. To the best of our knowledge, this is the first framework that scales up to 160 GPUs. Furthermore, even for single node training, S-Caffe shows an improvement of 14\% and 9\% over Nvidia's optimized Caffe for 8 and 16 GPUs, respectively. In addition, S-Caffe achieves up to 1395 samples per second for the AlexNet model, which is comparable to the performance of Microsoft CNTK.

Approximation of skewed interfaces with tensor-based model reduction procedures: Application to the reduced basis hierarchical model reduction approach

In this article we introduce a procedure, which allows to recover the potentially very good approximation properties of tensor-based model reduction procedures for the solution of partial differential equations in the presence of interfaces or strong gradients in the solution which are skewed with respect to the coordinate axes. The two key ideas are the location of the interface either by solving a lower-dimensional partial differential equation or by using data functions and the subsequent removal of the interface of the solution by choosing the determined interface as the lifting function of the Dirichlet boundary conditions. We demonstrate in numerical experiments for linear elliptic equations and the reduced basis-hierarchical model reduction approach that the proposed procedure locates the interface well and yields a significantly improved convergence behavior even in the case when we only consider an approximation of the interface.

Stack-through silicon via dynamic power consumption optimization in three-dimensional integrated circuit

Stack-through silicon via (TSV) used in three-dimensional integrated circuit has good temperature and heat transfer characteristics. A novel model for optimizing the dynamic power consumption based on stacked-TSV is proposed in this paper, in which delay, area and minimum aperture are comprehensively considered. After extracting single TSV parasitic electrical parameters, we analyze the influences of TSV size on multilayer TSV power consumption and delay performance, thereby building the hierarchical reduction TSV structure step by step. Moreover, the influences of TSV height and thickness of oxide layer are discussed. Results show that the model can significantly improve the dynamic power consumption at the expense of little delay. The power consumption optimization reduction is up to 19.52% with 5% delay penalty.

An expanded double-quantitative model regarding probabilities and grades and its hierarchical double-quantitative attribute reduction

Probabilities and grades serve as relative and absolute measures, respectively. They are used to establish the decision-theoretic rough set (DTRS) and graded rough set (GRS) – two basic quantitative models. The double-quantification of probabilities and grades exhibits systematicness and completeness in view of the two-dimensional feature of the approximate space; however, double-quantitative construction becomes a problem, and double-quantitative reduction is rarely reported. Thus, this paper mainly constructs an expanded double-quantitative model by logically integrating probabilities and grades; it further studies relevant double-quantitative reduction by hierarchically preserving specific regions. (1) First, a novel model is established via the logic integration and expansion requirement, and its regional system and granular hierarchy are studied via granular computing. Thus, regional semantics is extracted via basic semantics granules. Regional calculation is realized by two algorithms, and the algorithm regarding calculation granules exhibits optimization according to algorithm analyses. (2) Second, three types of model-regional preservation reducts and their hierarchy are discussed in the two-category case. Thus, SRP-Reduct, CRP-Reduct, and APP-Reduct are studied by exploring four-region preservation properties, constructing two-region classification regions, and preserving four original approximations, respectively. Furthermore, a relevant reduction hierarchy is thoroughly achieved. (3) Moreover, the model and its reduction are illustrated by two examples of decision tables. The constructional model conducts double-quantification regarding probabilities and grades; thus, it exhibits double-quantitative semantics and benignly expands DTRS-Model, GRS-Model, and Pawlak-Model. Furthermore, its hierarchical reduction reflects some double-quantitative reduction essence; thus, its reduction expands qualitative Pawlak-Reduction while guides quantitative DTRS-Reduction and GRS-Reduction.