Model Size Reduction Research Articles

Accurate and Efficient Plate and Rod Microfinite Element Models for Whole Bone Segments Based on High-Resolution Peripheral Computed Tomography

The high-resolution peripheral quantitative computed tomography (HR-pQCT) provides unprecedented visualization of bone microstructure and the basis for constructing patient-specific microfinite element (μFE) models. Based on HR-pQCT images, we have developed a plate-and-rod μFE (PR μFE) method for whole bone segments using individual trabecula segmentation (ITS) and an adaptive cortical meshing technique. In contrast to the conventional voxel approach, the complex microarchitecture of the trabecular compartment is simplified into shell and beam elements based on the trabecular plate-and-rod configuration. In comparison to voxel-based μFE models of μCT and measurements from mechanical testing, the computational and experimental gold standards, nonlinear analyses of stiffness and yield strength using the HR-pQCT-based PR μFE models demonstrated high correlation and accuracy. These results indicated that the combination of segmented trabecular plate-rod morphology and adjusted cortical mesh adequately captures mechanics of the whole bone segment. Meanwhile, the PR μFE modeling approach reduced model size by nearly 300-fold and shortened computation time for nonlinear analysis from days to within hours, permitting broader clinical application of HR-pQCT-based nonlinear μFE modeling. Furthermore, the presented approach was tested using a subset of radius and tibia HR-pQCT scans of patients with prior vertebral fracture in a previously published study. Results indicated that yield strength for radius and tibia whole bone segments predicted by the PR μFE model was effective in discriminating vertebral fracture subjects from nonfractured controls. In conclusion, the PR μFE model of HR-pQCT images accurately predicted mechanics for whole bone segments and can serve as a valuable clinical tool to evaluate musculoskeletal diseases.

Design and Implementation of a Convolutional Neural Network on an Edge Computing Smartphone for Human Activity Recognition

Edge computing aims to integrate computing into everyday settings, enabling the system to be context-aware and private to the user. With the increasing success and popularity of deep learning methods, there is an increased demand to leverage these techniques in mobile and wearable computing scenarios. In this paper, we present an assessment of a deep human activity recognition system's memory and execution time requirements, when implemented on a mid-range smartphone class hardware and the memory implications for embedded hardware. This paper presents the design of a convolutional neural network (CNN) in the context of human activity recognition scenario. Here, layers of CNN automate the feature learning and the influence of various hyper-parameters such as the number of filters and filter size on the performance of CNN. The proposed CNN showed increased robustness with better capability of detecting activities with temporal dependence compared to models using statistical machine learning techniques. The model obtained an accuracy of 96.4% in a five-class static and dynamic activity recognition scenario. We calculated the proposed model memory consumption and execution time requirements needed for using it on a mid-range smartphone. Per-channel quantization of weights and per-layer quantization of activation to 8-bits of precision post-training produces classification accuracy within 2% of floating-point networks for dense, convolutional neural network architecture. Almost all the size and execution time reduction in the optimized model was achieved due to weight quantization. We achieved more than four times reduction in model size when optimized to 8-bit, which ensured a feasible model capable of fast on-device inference.

Efficient Ultrasound Image Analysis Models with Sonographer Gaze Assisted Distillation.

Recent automated medical image analysis methods have attained state-of-the-art performance but have relied on memory and compute-intensive deep learning models. Reducing model size without significant loss in performance metrics is crucial for time and memory-efficient automated image-based decision-making. Traditional deep learning based image analysis only uses expert knowledge in the form of manual annotations. Recently, there has been interest in introducing other forms of expert knowledge into deep learning architecture design. This is the approach considered in the paper where we propose to combine ultrasound video with point-of-gaze tracked for expert sonographers as they scan to train memory-efficient ultrasound image analysis models. Specifically we develop teacher-student knowledge transfer models for the exemplar task of frame classification for the fetal abdomen, head, and femur. The best performing memory-efficient models attain performance within 5% of conventional models that are 1000× larger in size.

Performance Characterization of Deep Learning Models for Breathing-based Authentication on Resource-Constrained Devices

Providing secure access to smart devices such as smartphones, wearables and various other IoT devices is becoming increasingly important, especially as these devices store a range of sensitive personal information. Breathing acoustics-based authentication offers a highly usable and possibly a secondary authentication mechanism for secure access. Executing sophisticated machine learning pipelines for such authentication on such devices remains an open problem, given their resource limitations in terms of storage, memory and computational power. To investigate this challenge, we compare the performance of an end-to-end system for both user identification and user verification tasks based on breathing acoustics on three type of smart devices: smartphone, smartwatch and Raspberry Pi using both shallow classifiers (i.e., SVM, GMM, Logistic Regression) and deep learning based classifiers (e.g., LSTM, MLP). Via detailed analysis, we conclude that LSTM models for acoustic classification are the smallest in size, have the lowest inference time and are more accurate than all other compared classifiers. An uncompressed LSTM model provides an average f-score of 80%-94% while requiring only 50--180 KB of storage (depending on the breathing gesture). The resulting inference can be done on smartphones and smartwatches within approximately 7--10 ms and 18--66 ms respectively, thereby making them suitable for resource-constrained devices. Further memory and computational savings can be achieved using model compression methods such as weight quantization and fully connected layer factorization: in particular, a combination of quantization and factorization achieves 25%--55% reduction in LSTM model size, with almost no loss in performance. We also compare the performance on GPUs and show that the use of GPU can reduce the inference time of LSTM models by a factor of 300%. These results provide a practical way to deploy breathing based biometrics, and more broadly LSTM-based classifiers, in future ubiquitous computing applications.

New lower bounds for solving a scheduling problem with resource collaboration

The collaboration of robots or other renewable resources for the efficient performance of complex operations is an essential part of modern industrial systems. Here we considered a key feature of the scheduling problem, namely, alternative modes by which to execute an operation. Every operation mode uses a specific set of collaborating resources to perform the operation. This structure allows the scheduling engineer to incorporate in each operation a set of activities using few resources, where each resource performs some of the activities during a portion of the operation's duration. As such, this setup allows the reuse of previously designed solution segments in a manner that conserves engineering efforts and reduces model size while retaining the flexibility conferred by the use of alternative modes. In this paper, we propose new lower bounds for improving the solution process efficiency and we demonstrate a model-driven decision support system for formulating and solving such scheduling problems. Two types of bounds are derived and analyzed, LP relaxation and parameters-based calculation. Both types consider partial solutions at the customized branch and bound solution tree. The dominance relations among the bounds are determined analytically, and the performances of the dominant bounds are compared empirically. The results demonstrate a 70% runtime reduction compared to previously published methods and bounds.

The impact of short-term variability and uncertainty on long-term power planning

Traditionally, long-term investment planning models have been the apparent tool to analyse future developments in the energy sector. With the increasing penetration of renewable energy sources, however, the modelling of short-term operational issues becomes increasingly important in two respects: first, in relation to variability and second, with respect to uncertainty. A model that includes both may easily become intractable, while the negligence of variability and uncertainty may result in sub-optimal and/or unrealistic decision-making. This paper investigates methods for aggregating data and reducing model size to obtain tractable yet close-to-optimal investment planning decisions. The aim is to investigate whether short-term variability or uncertainty is more important and under which circumstances. In particular, we consider a generation expansion problem and compare various representations of short-term variability and uncertainty of demand and renewable supply. The main results are derived from a case study on the Danish power system. Our analysis shows that the inclusion of representative days is crucial for the feasibility and quality of long-term power planning decisions. In fact, we observe that short-term uncertainty can be ignored if a sufficient number of representative days is included.

Implementation of Timoshenko element local deflection for vertical track modelling

ABSTRACTA vertical track model suitable for the study of the dynamic response and the interaction between wheel and rail in the time domain is developed by using Timoshenko beam elements, and its performance is optimised by accounting for the local deflection of these type of elements. Implementation of the local system enables one to obtain an accurate description of the contact force in a more computationally efficient way than other numerical methods, and it leads to an almost total elimination of the discontinuities caused by the local displacement underestimation and the shear incompatibilities introduced by the conventional formulation with Timoshenko beam elements. The work presented here describes both static and dynamic approaches of the local system directly obtained through the resolution of the beam governing equations. The dynamic approach guarantees the shear rotation continuity and describes accurately the track frequency content. The method is tested for parametric excitation, in which it ensures the smoothness of the response. Secondly, it is tested for short-pitch corrugated rails, where overestimation of the contact force is avoided with a reduced model size when compared with conventional Timoshenko element formulation. The results are validated by comparison with those from previous studies.

GMCS: fast computation of genetic minimal cut sets in large networks.

The identification of minimal gene knockout strategies to engineer metabolic systems constitutes one of the most relevant applications of the COnstraint-Based Reconstruction and Analysis (COBRA) framework. In the last years, the minimal cut sets (MCSs) approach has emerged as a promising tool to carry out this task. However, MCSs define reaction knockout strategies, which are not necessarily transformed into feasible strategies at the gene level. We present a more general, easy-to-use and efficient computational implementation of a previously published algorithm to calculate MCSs to the gene level (gMCSs). Our tool was compared with existing methods in order to calculate essential genes and synthetic lethals in metabolic networks of different complexity, showing a significant reduction in model size and computation time. gMCS is publicly and freely available under GNU license in the COBRA toolbox (https://github.com/opencobra/cobratoolbox/tree/master/src/analysis/gMCS). Supplementary data are available at Bioinformatics online.

Structured deep Fisher pruning for efficient facial trait classification

High efficiency is desirable for many interactive biometrics tasks, including facial trait recognition. Although deep convolutional nets are effective for a multitude of classification tasks, their high space and time demands make them impractical for PCs and mobile devices without a powerful GPU. In this paper, we propose a structured filter-level pruning approach based on Fisher LDA [1], which boosts efficiency while maintaining accuracy for facial trait classification. It starts from the last convolutional layer where we find filter activations are less correlated. Through Fisher's LDA, we show that this decorrelation makes it safe to discard directly filters with high within-class variance and low between-class variance. The pruning goes on by tracing deconvolution based dependency over layers. Combined with light classifiers, the reduced CNNs can achieve comparable accuracies on example facial traits from the LFWA(+) and CelebA datasets, but with large reductions in model size (96%–98% for VGG-16, 81% for GoogLeNet) and computation (as high as 80% for VGG-16, 61% for GoogLeNet).

Model-Based Dynamic Optimization of Monoclonal Antibodies Production in Semibatch Operation—Use of Reformulation Techniques

Monoclonal antibodies (mAbs) constitute one of the leading products of the biopharmaceutical market with significant therapeutic and diagnostic applications. This has drawn increased attention to the intensification of their production processes, where model-based approaches can be utilized for successful optimization and control purposes. In this manuscript, dynamic optimization of mAb production in mammalian cell cultures in semibatch operation is performed. To develop a model suitable for optimization, reformulation steps consisting of function smoothening, reducing model size, and scaling are applied to a predictive energy-based model for mAb production presented in Quiroga et al. (2016). Optimization of the reformulated model leads to the derivation of an optimal feeding strategy, accounting for indirect quality measures, dilution effects, and the energy requirements of the cells. The results highlight the increased production outcome by using the reformulated model, and thus indicate the strong depe...

To aggregate or to eliminate? Optimal model simplification for improved process performance prediction

Operational process models such as generalised stochastic Petri nets (GSPNs) are useful when answering performance questions about business processes (e.g. ‘how long will it take for a case to finish?’). Recently, methods for process mining have been developed to discover and enrich operational models based on a log of recorded executions of processes, which enables evidence-based process analysis. To avoid a bias due to infrequent execution paths, discovery algorithms strive for a balance between over-fitting and under-fitting regarding the originating log. However, state-of-the-art discovery algorithms address this balance solely for the control-flow dimension, neglecting the impact of their design choices in terms of performance measures. In this work, we thus offer a technique for controlled performance-driven model reduction of GSPNs, using structural simplification rules, namely foldings. We propose a set of foldings that aggregate or eliminate performance information. We further prove the soundness of these foldings in terms of stability preservation and provide bounds on the error that they introduce with respect to the original model. Furthermore, we show how to find an optimal sequence of simplification rules, such that their application yields a minimal model under a given error budget for performance estimation. We evaluate the approach with two real-world datasets from the healthcare and telecommunication domains, showing that model simplification indeed enables a controlled reduction of model size, while preserving performance metrics with respect to the original model. Moreover, we show that aggregation dominates elimination when abstracting performance models by preventing under-fitting due to information loss.

Proper-Orthogonal-Decomposition-Based Buckling Analysis and Optimization of Hybrid Fiber Composite Shells

Composite cylindrical shells have been widely used in the fuel tanks of launch vehicles. To reduce the material cost of carbon-fiber-reinforced polymer shells, hybrid fiber composite shells composed of carbon fibers, glass fibers, and an epoxy matrix would be a more promising choice. Along with the development of heavy lift launch vehicles, the diameter of a required shell structure could become large. The buckling analysis of composite shells based on detailed finite element models was too time consuming. From the point of view of model size reduction, accurate reduced-order models for buckling analysis along with optimization methods were needed to improve the efficiency of the design and analysis cycle of composite shells. A combined proper orthogonal decomposition and optimization method is proposed in this paper for an efficient buckling analysis of composite shells. In the offline steps of proper orthogonal decomposition, one single finite-element-model buckling analysis is performed based on a full-order model of the initial design with the certain volume fractions of hybrid fibers. Furthermore, snapshots generated from the eigenvector values of the finite-element-model buckling analysis result from the initial design, and a proper orthogonal decomposition basis is determined. By using the obtained proper orthogonal decomposition basis from the initial design, the online steps of proper orthogonal decomposition can be carried out for the buckling analysis of any composite shell with varying volume fractions of hybrid fibers. To further improve the prediction accuracy for a large design space, the mixed proper orthogonal decomposition snapshot method is proposed by adding additional proper orthogonal decomposition snapshots into the original ones. The proposed method contributes to improving the prediction accuracy among the entire design space significantly. Finally, a low-cost optimization framework is established for hybrid fiber composite shells, during which the proper orthogonal decomposition method is used for accelerating the buckling analysis. In particular, a convergence criterion and an updated proper orthogonal decomposition basis strategy are proposed for the optimization framework, aiming at guaranteeing the credibility of the optimal proper orthogonal decomposition buckling result. By means of an illustrative example, the effectiveness and efficiency of the developed framework are demonstrated.

Adaptive floating node method for modelling cohesive fracture of composite materials

The cohesive element has been widely employed to model delamination and matrix cracking in composite materials. However, an extremely fine mesh along the potential crack path is required to achieve accurate predictions of stresses within the cohesive zone. A sufficient number of cohesive elements must be present within the cohesive zone ahead of the crack tip, resulting in very high computational cost and time for application to practical composite structures. To mitigate this problem, an adaptive floating node method (A-FNM) with potential to reduce model size and computational effort is proposed. An element with adaptive partitioning capabilities is designed such that it can be formulated as a master element, a refined element and a coarsened element, depending on the damage state in the progressive damage process. A relatively coarse overall mesh may be used initially, and by transforming the element configurations adaptively, the local refinement and coarsening schemes are applied in the analysis. The localized stress gradient ahead of the crack front within the refinement zone is captured by the refined elements. The refinement and coarsening operations are performed at the elemental level with fixed nodal connectivity, so that global successive remeshing in adaptive mesh refinement (AMR) techniques is avoided; this is the key difference between AMR and A-FNM. It is demonstrated that, without loss of accuracy, the present method simplifies the modelling procedure and reduces computational cost.

Electromagnetic Modeling of Wire-Ground Structures by using a MTL Based Approach

A multi-conductor transmission lines-based approach is developed to model electromagnetic coupling of wires above metallic structures. With this approach, global impedances and local distributions are accessible. According the frequency range and the geometry of the studied structures, it is possible to only take into account well-chosen parts. In that case, the shading concept provides an efficient way to determine which part is sufficient, thereby allowing reduction in model size. Global quantities results are validated against measurements, and local ones by theoretical knowledge. This work is usefulness in both the nomadic applications (automotive and aeronautic wiring ...) and the usual electrical applications (home and industrial wiring).

Sparsing Deep Neural Network Using Semi-Discrete Matrix Decomposition

Deep learning has gained a lot of successes in various areas, including computer vision, natural language process, and robot control. Convolution neural network (CNN) is the most commonly used model in deep neural networks. Despite their effectiveness on feature abstraction, CNNs need powerful computation even in the inference stage, which becomes a major obstacle in their deployment in embedded and mobile devices. In order to solve this problem, we 1) propose to make decomposition on convolution layers and full connected layers in CNNs with naive semi-discrete matrix decomposition (SDD), which achieves the low-rank decomposition and parameters sparse at the same time; and 2) we propose a layer-merging scheme which merges two out of all the three result matrices, which can avoid the explode of the intermediate data come with the naive semi-discrete matrix decomposition; 3) we propose a progressive training strategy to speed up the converging. We implement this optimized method in image classification and object detection networks. Under the loss of network accuracy by 1%, we achieve significant running time and model size reduction. The full-connected layer of the LeNet network achieves $7\times $ speedup in the inference stage. In the Faster-Rcnn, the weight parameters are reduced by the factor of $5.85\times $ , and it can have a speedup by the factor of $1.75\times $ .

Continuous-Time State-Space Unsteady Aerodynamic Modeling for Efficient Loads Analysis

The present paper proposes a continuous-time state-space formulation of the unsteady vortex lattice method, which is derived through a discretization of the governing advection equation for transport of vorticity in the wake. A continuous-time system is obtained by only discretizing the advection equation in space, while retaining the derivative with respect to time. The discretization in space is based on the discontinuous Galerkin method. The present method can be applied to any arbitrary nonuniform wake discretization and can be extended to higher-order panel methods or a nonflat wake shape. The method is extended to compressible flows by applying the Prandtl–Glauert transformation. The time-dependent terms in the small disturbance potential equation are neglected. Thus, incompressible flow solution procedures are applied with minimum modifications to unsteady compressible problems. The benefits are demonstrated by applying the model to the gust analysis of a general aircraft wing, varying the time step, and introducing a nonuniform wake discretization, resulting in a reduced model size for a given accuracy. The resulting continuous-time state-space model can be used for efficient loads analysis of general aircraft wings including the effects of compressibility and allows for easy integration with structural or flight dynamic models for efficient aero(servo)elastic analyses.

Foot Depth Map Point Cloud Completion using Deep Learning with Residual Blocks

Fit is extremely important in footwear as fit largely determines performanceand comfort. Current footwear fit estimation mainly usesonly shoe size, which is extremely limited in characterizing theshape of a foot or the shape of a shoe. 3D scanning presents asolution to this, where a foot shape can be captured and virtuallyfit with shoe models. Traditional 3D scanning techniques have theirown complications however, stemming from their need to collectviews covering all aspects of an object. In this work we explore adeep learning technique to compete a foot scan point cloud frominformation contained in a single depth map view. We examine thebenefits of implementing residual blocks in architectures for this application,and find that they can improve accuracies while reducingmodel size and training time.

Inductance Extraction for PCB Prelayout Power Integrity Using PMSR Method

Proper power integrity (PI) analysis is required for printed circuit board (PCB) power distribution network (PDN) design. Top-layer interconnect inductance for PI has always been a vital concern for high-speed industry. Developing a simple physics-based equivalent circuit model for critical structures is essential for understanding the physics of the system and for intelligent designs. In this paper, a physics-based model size reduction (PMSR) method is applied to get the equivalent circuit model for the above-ground geometries. The extracted physics-based models are also based on the partial element equivalent circuit (PEEC) method, and can be used in analyzing the structure in its parts. By applying PMSR method, a physics-based equivalent circuit model can be proposed and this circuit model is related to the geometric features of the design. In this way, PMSR method can provide an intuitive guideline in designing PCB and reducing above inductances, therefore, a low-ripple dc voltage can be delivered through PDN. Taking advantage of PEEC and PMSR methods, the top-layer inductances of three different geometries are calculated and the physics-based circuit models are obtained, respectively.

A Reduced Order Modeling Methodology for Steam Turbine Clearance Control Design

Finite element models (FEMs) are extensively used in the design optimization of utility scale steam turbines. As an example, by simulating multiple startup scenarios of steam power plants, engineers can obtain turbine designs that minimize material utilization, and at the same time, avoid the damaging effects of large thermal stresses or rubs between rotating and stationary parts. Unfortunately, FEMs are computationally expensive and only a limited amount of simulations can be afforded to get the final design. For this reason, numerous model reduction techniques have been developed to reduce the size of the original model without a significant loss of accuracy. When the models are nonlinear, as is the case for steam turbine FEMs, model reduction techniques are relatively scarce and their effectiveness becomes application dependent. Although there is an abundant literature on model reduction for nonlinear systems, many of these techniques become impractical when applied to a realistic industrial problem. This paper focuses on a class of nonlinear FEM characteristic of thermo-elastic problems with large temperature excursions. A brief overview of popular model reduction techniques is presented along with a detailed description of the computational challenges faced when applying them to a realistic problem. The main contribution of this work is a set of modifications to existing methods to increase their computational efficiency. The methodology is demonstrated on a steam turbine model, achieving a model size reduction by four orders of magnitude with only 4% loss of accuracy with respect to the full order FEMs.

A Multi-way Multi-task Learning Approach for Multinomial Logistic Regression*. An Application in Joint Prediction of Appointment Miss-opportunities across Multiple Clinics.

Whether they have been engineered for it or not, most healthcare systems experience a variety of unexpected events such as appointment miss-opportunities that can have significant impact on their revenue, cost and resource utilization. In this paper, a multi-way multi-task learning model based on multinomial logistic regression is proposed to jointly predict the occurrence of different types of miss-opportunities at multiple clinics. An extension of L1 / L2 regularization is proposed to enable transfer of information among various types of miss-opportunities as well as different clinics. A proximal algorithm is developed to transform the convex but non-smooth likelihood function of the multi-way multi-task learning model into a convex and smooth optimization problem solvable using gradient descent algorithm. A dataset of real attendance records of patients at four different clinics of a VA medical center is used to verify the performance of the proposed multi-task learning approach. Additionally, a simulation study, investigating more general data situations is provided to highlight the specific aspects of the proposed approach. Various individual and integrated multinomial logistic regression models with/without LASSO penalty along with a number of other common classification algorithms are fitted and compared against the proposed multi-way multi-task learning approach. Fivefold cross validationis used to estimate comparing models parameters and their predictive accuracy. The multi-way multi-task learning framework enables the proposed approach to achieve aconsiderable rate of parameter shrinkage and superior prediction accuracy across various types of miss-opportunities and clinics. The proposed approach provides an integrated structure to effectively transfer knowledge among different miss-opportunities and clinics to reduce model size, increase estimation efficacy, and more importantly improve predictions results. The proposed framework can be effectively applied to medical centers with multiple clinics, especially those suffering from information scarcity on some type of disruptions and/or clinics.