Step Size Research Articles

An analysis of second-order sav-filtered time-stepping finite element method for unsteady natural convection problems

This paper presents an unconditionally stable time-filtering algorithm for natural convection equations. The algorithm is based on the scalar auxiliary variables in the exponential function and adopts a completely discrete Back-Euler combining time filter scheme. The proposed scheme requires minimal invasive modification of the existing program to improve the time accuracy from first-order to second-order without increasing the computational complexity, and we demonstrate the unconditional stability of the proposed algorithm and analyze its second-order convergence. In addition, due to the increasing demand for low-memory solvers, the application of a time-adaptive algorithm can improve the accuracy and efficiency of the proposed algorithm, so we extend the method to variable step sizes and construct an adaptive algorithm. Finally, the effectiveness of the proposed method and the accuracy of the theoretical results are verified by numerical experiments.

W‐schemes in spectral methods for reaction‐diffusion equations

AbstractWe consider reaction‐diffusion equations, which are parabolic systems of partial differential equations. Assuming periodic boundary conditions in space, a multidimensional Fourier transform can be used. A spectral method yields an initial value problem of a stiff system of ordinary differential equations for time‐dependent Fourier coefficients. We investigate the application of Rosenbrock‐Wanner W‐schemes in the time integration of the ordinary differential equations including step size selection. Results of numerical computations are presented using a Turing system in two space dimensions.

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

This paper comprehensively analyzes distributed high-performance computing methods for accelerating deep learning training. We explore the evolution of distributed computing architectures, including data parallelism, model parallelism, and pipeline parallelism, and their hybrid implementations. The study delves into optimization techniques crucial for large-scale training, such as distributed optimization algorithms, gradient compression, and adaptive learning rate methods. We investigate communication-efficient algorithms, including Ring All Reduce variants and decentralized training approaches, which address the scalability challenges in distributed systems. The research examines hardware acceleration and specialized systems, focusing on GPU clusters, custom AI accelerators, high-performance interconnects, and distributed storage systems optimized for deep learning workloads. Finally, we discuss this field's challenges and future directions, including scalability-efficiency trade-offs, fault tolerance, energy efficiency in large-scale training, and emerging trends like federated learning and neuromorphic computing. Our findings highlight the synergy between advanced algorithms, specialized hardware, and optimized system designs in pushing the boundaries of large-scale deep learning, paving the way for future breakthroughs in artificial intelligence.

Multi-Dimensional Adaptive Learning Rate Gradient Descent Optimization Algorithm for Network Training in Magneto-Optical Defect Detection

Multi-Dimensional Adaptive Learning Rate Gradient Descent Optimization Algorithm for Network Training in Magneto-Optical Defect Detection

Leg compliance is required to explain the ground reaction force patterns and speed ranges in different gaits.

Two simple models, vaulting over stiff legs and rebounding over compliant legs, are employed to describe the mechanics of legged locomotion. It is agreed that compliant legs are necessary for describing running and that legs are compliant while walking. Despite this agreement, stiff legs continue to be employed to model walking. Here, we show that leg compliance is necessary to model walking and, in the process, identify the principles that underpin two important features of legged locomotion: First, at the same speed, step length, and stance duration, multiple gaits that differ in the number of leg contraction cycles are possible. Among them, humans and other animals choose a gait with M-shaped vertical ground reaction forces because it is energetically favored. Second, the transition from walking to running occurs because of the inability to redirect the vertical component of the velocity during the double stance phase. Additionally, we also examine the limits of double spring-loaded pendulum (DSLIP) as a quantitative model for locomotion, and conclude that DSLIP is limited as a model for walking. However, insights gleaned from the analytical treatment of DSLIP are general and will inform the construction of more accurate models of walking.

Analysis of regularized federated learning

Federated learning is an efficient machine learning tool for dealing with heterogeneous big data and privacy protection. Federated learning methods with regularization can control the level of communications between the central and local machines. Stochastic gradient descent is often used for implementing such methods on heterogeneous big data, to reduce the communication costs. In this paper, we consider such an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level. We improve the method by allowing flexible step sizes and carry out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting. In the non-convex setting, we derive rates of convergence when the smooth objective function satisfies a Polyak-Łojasiewicz condition. When the objective function is strongly convex, a sufficient and necessary condition for the convergence in expectation is presented.

Lossy Image Compression with Stochastic Quantization

Introduction. Lossy image compression algorithms play a crucial role in various domains, including graphics, and image processing. As image information density increases, so do the resources required for processing and transmission. One of the most prominent approaches to address this challenge is color quantization, proposed by Orchard et al. (1991). This technique optimally maps each pixel of an image to a color from a limited palette, maintaining image resolution while significantly reducing information content. Color quantization can be interpreted as a clustering problem (Krishna et al. (1997), Wan (2019)), where image pixels are represented in a three-dimensional space, with each axis corresponding to the intensity of an RGB channel. The purpose of the paper. Scaling of traditional algorithms like K-Means can be challenging for large data, such as modern images with millions of colors. This paper reframes color quantization as a three-dimensional stochastic transportation problem between the set of image pixels and an optimal color palette, where the number of colors is a predefined hyperparameter. We employ Stochastic Quantization (SQ) with a seeding technique proposed by Arthur et al. (2007) to enhance the scalability of color quantization. This method introduces a probabilistic element to the quantization process, potentially improving efficiency and adaptability to diverse image characteristics. Results. To demonstrate the efficiency of our approach, we present experimental results using images from the ImageNet dataset. These experiments illustrate the performance of our Stochastic Quantization method in terms of compression quality, computational efficiency, and scalability compared to traditional color quantization techniques. Conclusions. This study introduces a scalable algorithm for solving the color quantization problem without memory constraints, demonstrating its efficiency on a subset of images from the ImageNet dataset. The convergence speed of the algorithm can be further enhanced by modifying the update rule with alternative methods to Stochastic Gradient Descent (SGD) that incorporate adaptive learning rates. Moreover, the stochastic nature of the proposed solution enables the utilization of parallelization techniques to simultaneously update the positions of multiple quants, potentially leading to significant performance improvements. This aspect of parallelization and its impact on algorithm efficiency presents a topic for future research. The proposed method not only addresses the limitations of existing color quantization techniques but also opens up new possibilities for optimizing image compression algorithms in resource-constrained environments. Keywords: non-convex optimization, stochastic optimization, stochastic quantization, color quantization, lossy compression.

Statistical Analysis of nnU-Net Models for Lung Nodule Segmentation.

This paper aims to conduct a statistical analysis of different components of nnU-Net models to build an optimal pipeline for lung nodule segmentation in computed tomography images (CT scan). This study focuses on semantic segmentation of lung nodules, using the UniToChest dataset. Our approach is based on the nnU-Net framework and is designed to configure a whole segmentation pipeline, thereby avoiding many complex design choices, such as data properties and architecture configuration. Although these framework results provide a good starting point, many configurations in this problem can be optimized. In this study, we tested two U-Net-based architectures, using different preprocessing techniques, and we modified the existing hyperparameters provided by nnU-Net. To study the impact of different settings on model segmentation accuracy, we conducted an analysis of variance (ANOVA) statistical analysis. The factors studied included the datasets according to nodule diameter size, model, preprocessing, polynomial learning rate scheduler, and number of epochs. The results of the ANOVA analysis revealed significant differences in the datasets, models, and preprocessing.

Stability of step size control based on a posteriori error estimates

A posteriori error estimates based on residuals can be used for reliable error control of numerical methods. Here, we consider them in the context of ordinary differential equations and Runge-Kutta methods. In particular, we take the approach of Dedner & Giesselmann (2016) and investigate it when used to select the time step size. We focus on step size control stability when combined with explicit Runge-Kutta methods and demonstrate that a standard I controller is unstable while more advanced PI and PID controllers can be designed to be stable. We compare the stability properties of residual-based estimators and classical error estimators based on an embedded Runge-Kutta method both analytically and in numerical experiments.

Endogenous innovation scale and patent policy in a monetary schumpeterian growth model

Abstract This paper develops a monetary R&D-driven endogenous growth model featuring endogenous innovation scales and the price-marginal cost markup. To endogenize the step size of quality improvement, we propose a tradeoff mechanism between the risk of innovation failure and the benefit of innovation success in R&D firms. Several findings emerge from the analysis. First, a rise in the nominal interest rate decreases economic growth; however, its relationship with social welfare is ambiguous. Second, either strengthening patent protection or raising the professional knowledge of R&D firms leads to an ambiguous effect on economic growth. Third, the Friedman rule of a zero nominal interest rate fails to be optimal in view of the social welfare maximum. Finally, our numerical analysis indicates that the extent of patent protection and the level of an R&D firm’s professional knowledge play a crucial role in determining the optimal interest rate.

Generalized Lipschitz Tracing of Implicit Surfaces

AbstractWe present a versatile and robust framework to render implicit surfaces defined by black‐box functions that only provide function value queries. We assume that the input function is locally Lipschitz continuous; however, we presume no prior knowledge of its Lipschitz constants. Our pre‐processing step generates a discrete acceleration structure, a Lipschitz field, that provides data to infer local and directional Lipschitz upper bounds. These bounds are used to compute safe step sizes along rays during rendering. The Lipschitz field is constructed by generating local polynomial approximations to the input function, then bounding the derivatives of the approximating polynomials. The accuracy of the approximation is controlled by the polynomial degree and the granularity of the spatial resolution used during fitting, which is independent from the resolution of the Lipschitz field. We demonstrate that our process can be implemented in a massively parallel way, enabling straightforward integration into interactive and real‐time modelling workflows. Since the construction only requires function value evaluations, the input surface may be represented either procedurally or as an arbitrarily filtered grid of function samples. We query the original implicit representation upon ray trace, as such, we preserve the geometric and topological details of the input as long as the Lipschitz field supplies conservative estimates. We demonstrate our method on both procedural and discrete implicit surfaces and compare its exact and approximate variants.

A new variational integrator for constrained mechanical system dynamics

A new variational integrator is proposed to solve constrained mechanical systems. The main distinguishing feature of the present integrator comes from the distinct discretization of Lagrangians based on the Hamilton's principle in its most general form. Specifically, Hermite interpolation is used for discrete positions, which provides at least C1 continuity for generalized coordinates. The velocities and momentums are interpolated using quadratic polynomials for the consistency, such that the kinematic relation between velocities and positions can be exactly satisfied. Meanwhile, the Gauss-Legendre quadrature rule is employed to guarantee the accuracy of discrete Lagrange equations. To tackle constrained mechanical systems, a coordinate partition approach is used to eliminate the constraint equations. The local incremental rotation vector is exploited to get rid of rotation singularities in spatial problems. Moreover, an adaptive stepsize strategy is implemented to improve the efficiency. The strengths of the new integrator lie in the accessible large step sizes in the simulation and its global second-order accuracy for positions as well as velocities. Several examples are performed and analyzed to validate its accuracy and capabilities.

Ablation and molten layer flow simulation for plate model of SiO2f/SiO2 composite material using particle method

In this paper, the moving particle semi-implicit method (MPS) is extended from calculating free mobility to simulating the extremely viscous and temperature-dependent molten layer flow of SiO2f/SiO2 composite material under aerodynamic heating conditions, which includes strong heating and shear of incoming flow. A method for applying heat flux and airflow shear, based on the conceptual particle approach, has been established. Heat transfer, melting, solidification, and evaporation behaviors are considered, with temperature-dependent viscosity variations also accounted for. The ablative regression of the plate model is verified using experimental results of the SiO2f/SiO2 composite material, and results from convergence analysis demonstrate the accuracy of the space step size selection. Surface morphology analysis through three-dimensional computation indicates that the extended particle method also accurately describes the surface morphology of SiO2f/SiO2 composite material under aerodynamic heating conditions. Thus, the extended particle method accurately simulates both the ablation process and the surface morphology of the SiO2f/SiO2 composite material. The influences of acceleration and surface tension are discussed. Ablative recession, when subject to acceleration, is smaller than that observed in its absence. When exposed to surface tension, the liquid layer tends to form a spherical shape, and the particles behave as a cohesive unit, resulting in smaller ablative recession than in the absence of surface tension.

Deep neural network based distribution system state estimation using hyperparameter optimization

In the past decade, distribution system state estimation has become a crucial topic in power system research due to the increasing importance of distribution networks amidst the decline of centralized energy production. This paper addresses a gap in the literature regarding the application of modern hyperparameter optimization techniques in low-voltage distribution system state estimation using deep neural networks. In particular, it demonstrates the use of the Tree-structured Parzen Estimator algorithm, which is a Bayesian hyperparameter optimization method, for distribution system state estimation on real Hungarian low-voltage networks. The study uses data from four real-life low-voltage supply areas in Hungary, which were modeled to address the challenges in obtaining network information. Compared to traditional methods like the weighted least squares method, the Tree-structured Parzen Estimator algorithm significantly improves the accuracy of the voltage amplitude and angle estimations, reducing the relative error by 14–73%. Additionally, it is shown that TPE outperforms simpler methods like Random Search in hyperparameter optimization. The results also reveal connections between the distribution system size and optimal hyperparameters, such as batch size, learning rate, and hidden layer configuration. The proposed non-iterative algorithm, combined with the parallel computation capabilities of deep neural networks utilizing GPU, resulted in four orders of magnitude improvement in runtime. These advancements make the proposed approach a valuable tool for renewable energy integration planning and real-time monitoring, highlighting its potential for practical applications in the power industry.

Inverse Estimation of Swirl Cooling Heat Transfer Coefficient

Swirl cooling is one of the most widely studied techniques in the field of heat transfer enhancement. This paper presents a novel, non-intrusive, and rather inexpensive inverse method to estimate the local heat transfer coefficient along the swirl chamber. First, the direct problem of two-dimensional heat conduction inside the chamber’s wall is solved numerically. The optimal number of sensors, their axial and radial positions, and other parameters concerning the inverse algorithm are determined using simulated experiments. The experimental setup is built based on these parameters. Transient temperature readings from several thermocouples embedded inside the chamber wall are fed to an inverse algorithm based on the conjugate gradient method. This algorithm takes advantage of an adjoint differential equation and the sensitivity differential equation to calculate the gradients and optimal step sizes, respectively. The local transient heat transfer coefficient is estimated for three Reynolds numbers along four straight lines parallel to the axis of the chamber.

Enhanced multimodal biometric access control system using chicken swarm optimization and self-organizing feature maps: a study on ear and iris recognition

Access control systems are crucial for securing sensitive data and system components by allowing authorized access while blocking unauthorized entities. Traditional unimodal biometric systems, make use of a single physiological or behavioral trait, have limitations such as susceptibility to spoofing and environmental constraints, leading to reduced reliability. This study explores an enhanced multimodal biometric access control system that combines ear and iris traits using an enhanced Self-Organizing Feature Map (SOFM) algorithm improved with Chicken Swarm Optimization (CSO). The system's performance is evaluated against traditional SOFM, with a focus on recognition accuracy and processing time. The data used to train the classifier for this study were collected from 190 individuals, encompassing a total of 2,280 images of iris, and ear traits. Preprocessing involved cropping, resizing, and grayscale conversion using histogram equalization. Feature extraction utilized Local Binary Patterns (LBP), followed by feature fusion at the feature level to create an integrated feature set. The enhanced SOFM algorithm was then applied for classification, with the CSO technique optimizing the learning rate and weight parameters for improved performance. At different thresholds, the CSO-SOFM classifier outperformed the standard SOFM classifier using metrics such as Sensitivity, Specificity, Precision, Accuracy and Recognition time.

Enhanced hybrid LSTM and SLAR modeling for in-depth analysis of temporal and spatial patterns in compositional data for environmental monitoring

A novel methodology for analyzing compositional data (CoDa) integrates long short-term memory (LSTM) networks with spatial lag autoregressive (SLAR) models to simultaneously capture temporal and spatial patterns. The proposed approach leverages LSTM’s strengths in handling temporal relationships and SLAR’s ability to capture spatial correlations, employing the centered log-ratio (CLR) transformation to manage CoDa’s characteristics, making it suitable for advanced analysis. The Adam optimizer ensures efficient and stable model training, enhancing convergence speed and precision. The combined approach involves preprocessing CoDa with CLR, initializing LSTM layers, performing forward passes through LSTM, integrating the SLAR model, generating outputs, post-processing data, calculating loss, and performing backpropagation. The methodology’s effectiveness is demonstrated through a case study on pollutant emissions data from a waste incineration plant, focusing on key pollutants such as Dioxins and Furans, Heavy Metals (Hg, Pb, Cd), Nitrogen Oxides (NO), Sulfur Dioxide (SO), Particulate Matter (PM), and Carbon Monoxide (CO). The LSTM network, configured with two hidden layers, dynamically adjusts learning rates for faster convergence. Integrating SLAR improves spatial pattern accounting, significantly enhancing predictive accuracy. Comparative analysis shows substantial performance metric improvements over standard LSTM models, with the LSTM-SLAR model reducing errors and explaining a higher proportion of data variability. In process safety and risk engineering, the proposed methodology enhances pollutant emission monitoring and control, enables proactive risk management, ensures regulatory compliance, and improves risk assessment accuracy, making it invaluable for dynamic risk assessment in environmental monitoring and is broadly applicable to fields requiring detailed temporal and spatial analysis, demonstrating robustness and versatility in handling complex CoDa.

Hitting the High-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models

Abstract We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for the stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD), which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations, which give an excellent match to the theory.

Aero-optical effects of a hypersonic vehicle based on the refractive index step ray tracing method

The aero-optical effect caused by the high-speed flight of aircraft can affect the accuracy of astronomical navigation, so studying the laser transmission characteristics in high-speed flow fields is essential. This study adopts a 3D spherical blunt cone model as the physical aircraft model, incorporates the latest reanalysis data from the National Centers for Environmental Prediction as atmospheric parameters, and uses Fluent software to establish a hypersonic aircraft external flow field model. The ray tracing error is calculated using a method based on the refractive index step size. Results indicate that, under the conditions of flying altitude of 5 km and incident angle of 45°, the proposed ray tracing method has better accuracy than the traditional method. Compared with the Runge–Kutta method and the method based on geometric step size, the proposed ray tracing method reduces the error by 46.4% and 12.5%, respectively, at the flight Mach number of 5.

Optimizing dose parameters for enhanced maskless lithography in MoS2-based devices

Maskless lithography simplifies the fabrication process and reduces costs compared to electron beam (E-beam) lithography, making it a more efficient choice for patterning nano-devices. Maskless lithography presents a promising avenue for expediting device fabrication by eliminating the need for masks. This technique can streamline the production of basic electronic devices, offering an efficient and low-cost alternative to traditional lithographic methods, like E-beam lithography. This study utilized a 405 nm photodiode to achieve pattern-writing with a minimum linewidth of 1 μm. Exploring optimal parameters includes adjustments in beam intensity, scan speed, and step size. Maskless lithography was applied to 2D transition metal dichalcogenides (TMDCs) material, MoS2, to investigate their electrical transport characteristics. The fabricated device exhibits an ON/OFF ratio of ∼1.7 × 106 and a mobility of ∼0.833 cm2/V·s, indicating a high switching efficiency. The results demonstrate optimized maskless lithography's potential for swift and cost-effective fabrication, offering intermediate-resolution patterning capabilities.