Gradient Optimization Research Articles

On Some Works of Boris Teodorovich Polyak on the Convergence of Gradient Methods and Their Development

The paper presents a review of the current state of subgradient and accelerated convex optimization methods, including the cases with the presence of noise and access to various information about the objective function (function value, gradient, stochastic gradient, higher derivatives). For nonconvex problems, the Polyak–Lojasiewicz condition is considered and a review of the main results is given. The behavior of numerical methods in the presence of a sharp minimum is considered. The aim of this review is to show the influence of the works of B.T. Polyak (1935–2023) on gradient optimization methods and their surroundings on the modern development of numerical optimization methods.

Impact of graded CIGS absorber layer on Cd-free CIGS thin-film solar cell performance: numerical optimization

Abstract This paper investigates replacing the CdS buffer layer with (Zn, Mg)O and Zn(S, O) double buffer layers. To increase the efficiency of the structure, our strategy involves numerically studying and comparing three types of CIGS bandgap absorber layers: FB (Flat Band), Double Gradient Bandgap (DG) with a high bandgap near the Mo contact (back side), a low bandgap near the buffer layer (front side), and a minimum bandgap between them, and Simple Gradient Bandgap (SG) with a high bandgap near the Mo contact (back side) and a low bandgap near the buffer layer (front side). Each type features different linear graded profiles in the thickness direction to determine the optimal bandgap gradient of the CIGS layer. The performance of the three types is numerically analyzed using the SCAPS-1D simulator. Simulation enables us to test multiple structural combinations efficiently, saving both time and money, while providing results that guide experimental research. The proposed device structure is composed of the following layers: ZnO:Al/n-Zn1-xMgxO/n- ZnSxO1-x/p- Graded-CuIn1-xGaxSe2 /Mo. To study the effects of linear bandgap gradient distribution of Ga⁄((Ga+In) ) ratio according to the thickness direction on CIGS solar cell performance, the electrical properties of the proposed CIGS model were initially matched with experimental data to confirm the structure's accuracy. It was found that the Double Gradient Bandgap (DG), with maxima of 1.44 eV at the back side (near Mo contact), 1.238 eV at the front side (near n-type buffer layer interface), and a minimum of 1.154 eV at a position of 1.7 µm from the back side, has a higher efficiency than the other structures with 26.22% (Voc = 0.8134 V, Jsc = 37.82 mA/cm², FF = 85.21%). Optimizing the bandgap gradient of the CIGS absorber layer helps improve the efficiency of CIGS solar cells. 

Numerical investigation of PEMFC performance enhancement through combined design of anodic fishbone ridge groove channel and optimal gas diffusion layer porosity gradient

The flow field plate is one of the core components of proton exchange membrane fuel cells (PEMFC), and its structure significantly influences the thermo-electrochemical coupling transport characteristics of PEMFCs. Optimizing the parameters of the flow channels of bipolar plates is crucial for enhancing the mass transfer of reacting gas and improving water removal capacity. In this study, an anodal fishbone ridge groove combined flow channel (F-SFC) is designed to enhance the diffusion uniformity of anode hydrogen gas, and a 3D multi-phase model of PEMFC is established using the multiphysics simulation software COMSOL Multiphysics. The F-SFC was compared with traditional bipolar plates featuring symmetric anode and cathode flow channels in terms of overall output performance, and the results are presented. The findings indicate that the F-SFC design achieves an average increase of 28.78% in current density and 21.07% in power density compared to conventional fishbone flow channels. When compared to traditional serpentine flow channels, the F-SFC design shows a 13.5% increase in current density and a 10.42% increase in power density. Furthermore, adding multilayer porosity gradients to gas diffusion layers (GDLs) greatly improves internal mass transfer. A gradient porosity of 0.7, 0.5, and 0.3 along the Z-axis produces a more equal distribution of current density and water concentration on the membrane. This work sheds fresh light on optimizing PEMFC design for increased power density, providing useful theoretical advice for future advances.

Various optimized artificial neural network simulations of advection-diffusion processes

The aim of this research is to describe an artificial neural network (ANN) based method to approximate the solutions of the natural advection-diffusion equations. Although the solutions of these equations can be obtained by various effective numerical methods, feed forward neural network (FFNN) techniques combined with different optimization techniques offer a more practicable and flexible alternative than the traditional approaches to solve those equations. However, the ability of FFNN techniques to solve partial differential equations is a questionable issue and has not yet been fully concluded in the existing literature. The reliability and accuracy of computational results can be advanced by the choice of optimization techniques. Therefore, this study aims to take an effective step towards presenting the ability to solve the advection-diffusion equations by leveraging the inherent benefits of ANN methods while avoiding some of the limitations of traditional approaches. In this technique, the solution process requires minimizing the error generated by using a differential equation whose solution is considered as a trial solution. More specifically, this study uses a FFNN and backpropagation technique, one of the variants of the ANN method, to minimize the error and the adjustment of parameters. In the solution process, the loss function (error) needs to be minimized; this is accomplished by fitting the trial function into the differential equation using appropriate optimization techniques and obtaining the network output. Therefore, in this study, the commonly used techniques in the literature, namely gradient descent (GD), particle swarm optimization (PSO) and artificial bee colony (ABC), are selected to compare the effectiveness of gradient and gradient-free optimization techniques in solving the advection-diffusion equation. The calculations with all three optimization techniques for linear and nonlinear advection-diffusion equations have been run several times to obtain the optimum accuracy of the results. The computed results are seen to be very promising and in good agreement with the effective numerical methods and the physics-informed neural network (PINN) method in the literature. It is also concluded that the PSO-based algorithm outperforms other methods in terms of accuracy.

Ensemble LVQ Model for Photovoltaic Line-to-Line Fault Diagnosis Using K-Means Clustering and AdaGrad

Line-to-line (LL) faults are one of the most frequent short-circuit conditions in photovoltaic (PV) arrays which are conventionally detected and cleared by overcurrent protection devices (OCPDs). However, OCPDs are shown to face challenges when detecting LL faults under critical detection conditions, i.e., low mismatch levels and/or high fault impedance values. This occurs due to insufficient fault current, thus leaving the LL faults undetected and leading to power losses and even catastrophic fire hazards. To compensate for OCPD deficiencies, recent studies have proposed modern artificial intelligence (AI)-based methods. However, various limitations can still be witnessed even in AI-based methods, such as (i) most of the models requiring a massive training dataset, (ii) critical fault detection conditions not being taken into consideration, (iii) models not being accurate enough when dealing with critical conditions, etc. To this end, the present paper proposes a learning vector quantization (LVQ)-based ensemble learning model in which three LVQs are individually trained to detect and classify LL faults in PV arrays. The initial LVQ vectors are determined using the k-means clustering method, and the learning rate is optimized by the adaptive gradient (AdaGrad) optimizer. The training and testing datasets are collected according to the PV array’s current–voltage (I–V) characteristic curve, and several features are extracted based on the Canberra and chi-squared distance techniques. The model utilizes a small training dataset, considers various critical detection conditions for LL faults—such as different mismatch levels and fault impedance values—and the final experimental results show that the model achieves an impressive average accuracy of 99.26%.

A short-term load demand forecasting: Levenberg–Marquardt (LM), Bayesian regularization (BR), and scaled conjugate gradient (SCG) optimization algorithm analysis

A short-term load demand forecasting: Levenberg–Marquardt (LM), Bayesian regularization (BR), and scaled conjugate gradient (SCG) optimization algorithm analysis

Diamond Surfaces with Lateral Gradients for Systematic Optimization of Surface Chemistry for Relaxometry - a Low-Pressure Plasma-Based Approach.

Diamond is increasingly popular because of its unique material properties. Diamond defects called nitrogen vacancy (NV) centers allow for measurements with unprecedented sensitivity. However, to achieve ideal sensing performance, NV centers need to be within nanometers from the surface and are thus strongly dependent on the local surface chemistry. Several attempts have been made to compare diamond surfaces. However, due to the high price of diamond crystals with shallow NV centers, a limited number of chemical modifications have been studied. Here, we developed a systematic method to investigate the continuity of different local environments with varying densities and natures of surface groups in a single experiment on a single diamond plate. To achieve this goal, we used diamonds with a shallow ensemble of NV centers and introduced a chemical gradient across the surface. More specifically, we used air and hydrogen plasma. The gradients were formed by a low-pressure plasma treatment after masking with a right-angled triangular prism shield. As a result, the surface contained gradually more oxygen/hydrogen toward the open end of the shield. We then performed wide-field relaxometry to determine the effect of surface chemistry on the sensing performance. As expected, relaxation times and thus sensing performance indeed vary along the gradient.

Metaheuristic optimization of extreme gradient boosting machine for enhanced prediction of lateral strength of reinforced concrete columns under cyclic loadings

The estimation of lateral strength in reinforced concrete (RC) columns subjected to cyclic loads is crucial in structural design. The failure of RC columns under lateral forces can lead to catastrophic structural collapses. This fact emphasizes the need for accurate assessments of their dynamic behavior. This paper proposes a data-driven model for estimating the lateral strength of RC columns. A historical dataset comprising 12 predictor variables and 247 samples has been compiled to train and validate of the proposed approach. The extreme gradient boosting machine (XGBoost) is employed to establish a predictive relationship between the lateral strength of RC columns and their influencing factors. Since model selection is critical for constructing a reliable prediction mode, this study relies on utilizing metaheuristic approaches, including Genetic Algorithm, Particle Swarm Optimization, Artificial Bee Colony, and Ant Colony Optimization, to optimize the performance of the XGBoost model. Experimental results show that the integration of Ant Colony Optimization and XGBoost can help attain outstanding prediction accuracy with a correlation of determination (R2) of 0.95. Additionally, an asymmetric squared error loss function is utilized to reduce overestimations by 12 %. The newly developed method can be utilized in practical applications where reliable predictions of the lateral strength of RC columns under cyclic loads are required.

SA-SGA: Simulated Annealing Optimization and Stochastic Gradient Ascent Reinforcement Learning for Feature Selection

SA-SGA: Simulated Annealing Optimization and Stochastic Gradient Ascent Reinforcement Learning for Feature Selection

Multi-parameter optimization of polarization gradient cooling for ⁸⁷Rb atoms based on reinforcement learning

Multi-parameter optimization of polarization gradient cooling for ⁸⁷Rb atoms based on reinforcement learning

Information fusion in order-2 fuzzy environments: A matrix transformation perspective

Order-2 information granules, as a representation of multi-layer structured information, have recently rekindled discussions. It is usually generated by abstracting order-1 information granules. When the dependence relationship and corresponding membership function of the order-1 information granules (reference information granules) are known, Pedrycz et al. used a method based on gradient optimization to complete the aggregation of the order-2 information granules. In this paper, we discuss a more specific scenario: not only the dependencies between reference information granules are captured, but also the order-1 information granules can be expressed as specific fuzzy information distributions. For this, we propose a fusion scheme of order-2 fuzzy sets based on matrix transformation called CQRP. To our knowledge, it is the first method that completely integrates the structural information of the order-2 environment into the fusion of order-2 fuzzy sets. In the process, we also creatively proposed the attraction and exclusion of structural information, deepening the understanding of structural information. Through sufficient comparison and analysis, we prove that it makes fuller use of information in the order-2 environment and is more reasonable and effective in tasks such as classification and identification.

Grid Optimization of Free-Form Spatial Structures Considering the Mechanical Properties

In recent years, the application of free-form surface spatial grid structures in large public buildings has become increasingly common. The layouts of grids are important factors that affect both the mechanical performance and aesthetic appeal of such structures. To achieve a triangular grid with good mechanical performance and uniformity on free-form surfaces, this study proposes a new method called the “strain energy gradient optimization method”. The grid topology is optimized to maximize the overall stiffness, by analyzing the sensitivity of nodal coordinates to the overall strain energy. The results indicate that the overall strain energy of the optimized grid has decreased, indicating an improvement in the structural stiffness. Specifically, compared to the initial grid, the optimized grid has a 30% decrease in strain energy and a 43.3% decrease in maximum nodal displacement. To optimize the smoothness of the grid, the study further applies the Laplacian grid smoothing method. Compared to the mechanically adjusted grid, the structural mechanical performance does not significantly change after smoothing, while the geometric indicators are noticeably improved, with smoother lines and regular shapes. On the other hand, compared to the initial grid, the smoothed grid has a 21.4% decrease in strain energy and a 28.3% decrease in maximum nodal displacement.

Deep bidirectional hierarchical matrix factorization model for hyperspectral unmixing

Existing deep nonnegative matrix factorization-based approaches treat shallow and deep layers equally or with similar strategies, neglecting the heterogeneous physical structures between the shallow and progressively deeper layers, thus failing to explore the latent different sub-manifold structures in the hyperspectral image. In this paper, we propose a deep nonnegative matrix factorization model with bidirectional constraints to achieve hyperspectral unmixing. The sub-manifold structures in hyperspectral image are fully exploited by filtering and penalizing the shallow abundance layer with a denoised regularizer and a manifold regularizer. In contrast to the shallow abundance layer, the remaining layers are constrained by an extremely common regularizer to avoid over-denoising and maintain fidelity. In this way, the fine cues between different substances are exploited to a large extent. Additionally, the overall reconstruction error can be well controlled because the performance of the designed feedback mechanism can be fine-tuned by the inverse hierarchical constraints. Finally, we employ Nesterov's optimal gradient method to solve the optimization problem effectively. Experiment results are conducted on both synthetic datasets and real datasets, and all results show that the proposed method is superior to recent canonical unmixing methods.

A-316 A quantitative gas chromatography tandem mass spectrometry method for metabolic profiling of urine organic acids and acylglycines

Abstract Background The analysis of urine organic acids (UOA) and acylglycines (UAG) are an essential investigation for the diagnosis inborn errors of metabolism (IEMs). The vast majority of methods for these analytes use gas chromatography mass spectrometry (GC-MS) and are only qualitative. Gas chromatography tandem mass spectrometry (GC-MS/MS) offers greater sensitivity and selectivity and enables the combination of qualitative screening with quantitative analysis. We describe the development and validation results of a quantitative GC-MS/MS method. Methods Urine creatinine concentrations were measured (CobasPro, Roche Diagnostics) and normalized to 1.25 mmol/L. After internal standard addition, each urine specimen, calibration standard and QC were incubated with a hydroxylamine solution followed by a liquid-liquid extraction. Further sample concentration permitted the isolated UOA and UAG to be derivatized using N,O-bis(trimethylsilyl)trifluoroacetamine (BSTFA) with 1% chlorotrimethylsilane (TMCS) prior to injection. GC-MS/MS analysis was performed using a GC-MS-TQ8050 triple quadrupole system (Shimadzu) with a 30 m DB-5MS (Agilent J&W) capillary column (0.25 mm, ID of 0.25 µm). Split injection mode was deployed with an initial column temperature of 70°C. The mass spectrometer was operated in electron ionization (EI) and multiple reaction monitoring (MRM) modes. Results The optimal temperature gradient for high-resolution UOA and UAG separations was derived and their respective retention times determined. MS/MS detection parameters were optimized to permit the identification of precursor and production ions for all (N=36) derivatized analytes. Total analysis time was 33.8 min per injection. This method permits the measurement of 11 samples in one batch and is highly linear for all analytes (R^2 >0.99 and average slope 1.04). The limit of quantification (LOQ) was established and ranged from <0.05 to 2.1 µmol/mmol creatinine. Intra- and inter-day imprecision (CV%) was determined using matrix-matched abnormal and normal QC. Intra-day imprecision (N=10) ranged from 1.3% (2-ketoglutaric acid) to 9.7% (3-hydroxybutyric acid). Inter-day imprecision (N=30 over 15 days) ranged from 2.2% (ethylmalonic acid and methylmalonic acid) to 20.5% (3-hydroxy-3-methylglutaric acid). The method had an average accuracy of 113±14% for analytes included in the ERNDiM quantitative organic acids (urine) external quality assurance program. For analytes not included in this proficiency testing, the method showed good agreement when compared to a national reference laboratory, with average Deming correlation coefficients and regression slopes being 0.9944 and 1.1, respectively. No interferences were detected when monitoring the analyte quantifier and qualifier ion ratios in patient urine. Conclusions The devised GC-MS/MS protocol permits the unbiased identification and quantification of UOA and UAG used to investigate IEMs. The method has acceptable sensitivity, specificity, precision and accuracy for clinical routine clinical testing.

Research on Surface Defect Detection Method of Wire Rope Outer Covering Layer Based on Deep Learning

Abstract Aiming at the requirement of a certain enterprise for the visual inspection accuracy of the surface defects of the outer sheath of the wire rope to reach 95 %, this paper adopts the existing Faster R-CNN algorithm to detect the surface defects of the outer sheath of the wire rope, and its accuracy mAP 0.5 is 8 8.7%. In order to improve the detection accuracy, the Faster R-CNN algorithm is improved. Among them, ResNeXt-101 is obtained on the basis of ResNeXt-50. It is used as the backbone network, and the attention mechanism feature pyramid network ACFPN is introduced to extract multi-scale features to enhance the network’s defect detection ability. The defect detection accuracy mAP 0.5 is improved to 93.2% In view of the diversity and large size differences of the defects of the outer sheath of the wire rope, a new loss function is constructed for the network. At the same time, the BO algorithm is used to optimize the hyperparameters of the stochastic gradient optimizer SGD of the improved network. The final designed ResNeXt101-ACFPN-Faster R-CNN algorithm is used to detect the surface defects of the outer sheath of the wire rope, and the detection accuracy mAP 0.5 reaches 95.8%. The detection goal of the enterprise is achieved. The research results show that the network optimization method proposed in this paper has a good effect on improving the detection accuracy of the surface defects of the outer sheath of the wire rope, and also verifies the effectiveness and feasibility of the algorithm improvement.

Dual Attention Adversarial Attacks With Limited Perturbations.

The construction of undetectable adversarial examples with few perturbances remains a difficult problem in adversarial attacks. At present, most solutions use the standard gradient optimization algorithm to build adversarial examples by applying global perturbations to benign samples and then launch attacks on the targets (e.g., face recognition systems). However, when the perturbance size is limited, the performance of these approaches suffers substantially. The content of crucial places in an image, on the other hand, will impact the final prediction; if these areas can be investigated and limited perturbances introduced, an acceptable adversarial example will be constructed. Based on the foregoing research, this article offers a dual attention adversarial network (DAAN) to produce adversarial examples with limited perturbations. DAAN initially searches for effective areas in an input image using the spatial attention network and channel attention network, and then creates space and channel weights. Following that, these weights direct an encoder and a decoder to generate effective perturbation, which is then combined with the input to produce an adversarial example. Finally, the discriminator determines if the created adversarial examples are true or false, and the attacked model is utilized to determine whether the generated samples fit the attack targets. Extensive studies on various datasets show that DAAN not only delivers the best attack performance across all comparison algorithms with few perturbations, but it can also significantly improve the defensiveness of the attacked models.

Leveraging visible-near-infrared spectroscopy and machine learning to detect nickel contamination in soil: Addressing class imbalances for environmental management

Excessive concentrations of Ni in soil have many severe effects, negatively affecting human health and leading to disease, while also posing a threat to animals and plants. Although the dangers of high Ni concentrations have been widely recognized, rapid and large-scale tools for the identification of Ni contamination are still lacking. Visible-near-infrared (Vis-NIR) spectroscopy has been employed to rapidly identify Ni contamination; however, previous studies suffer from issues inherent to small datasets and the tendency to negate data imbalances. To address these issues, a large dataset comprising 18,675 soil samples was used to predict soil Ni contamination by combining Vis-NIR data with machine learning (ML). The data imbalance inherent to previous studies was addressed using two data sampling methods. To build a robust classification model for Ni contamination, four spectral preprocessing methods and four ML algorithms were compared. The optimal extreme gradient boosting model achieved recall, accuracy, area under the curve, and geometric mean scores of 0.8203, 0.8806, 0.9268, and 0.8508, respectively. Model predictions across the United States identified specific regions with high possibility of Ni contamination. Overall, the model developed in this study offers an improved accuracy in predicting soil Ni contamination at the continental scale, and can be used to prioritize further testing and guide policymaking.

Knowledge distillation via adaptive meta-learning for graph neural network

With the ever-increasing scale of graph-structured data, the computational resource requirements of large-scale graph neural networks (GNNs) can impede their deployment on devices with limited resources. Through knowledge distillation (KD), the expertise of an expansive, previously trained model (the teacher) can be transferred to a smaller architecture (the student) while maintaining comparable performance. However, existing KD approaches for GNNs often fix the teacher model during student learning. Without considering the students' learning feedback, there is a performance reduction after KD is achieved. With this objective, this research proposed an expertise transfer method via adaptive meta-learning for GNNs. The teacher can continuously update its parameters according to the student's optimal gradient direction in each KD step. Thus, the teacher learns to teach appropriate knowledge to the student. To maintain the structural features of each node and further avoid over-smoothing, we also introduced local structure preservation loss. Comprehensive experiments across four benchmarks demonstrate the effectiveness of our methodology.

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

This paper comprehensively analyzes distributed high-performance computing methods for accelerating deep learning training. We explore the evolution of distributed computing architectures, including data parallelism, model parallelism, and pipeline parallelism, and their hybrid implementations. The study delves into optimization techniques crucial for large-scale training, such as distributed optimization algorithms, gradient compression, and adaptive learning rate methods. We investigate communication-efficient algorithms, including Ring All Reduce variants and decentralized training approaches, which address the scalability challenges in distributed systems. The research examines hardware acceleration and specialized systems, focusing on GPU clusters, custom AI accelerators, high-performance interconnects, and distributed storage systems optimized for deep learning workloads. Finally, we discuss this field's challenges and future directions, including scalability-efficiency trade-offs, fault tolerance, energy efficiency in large-scale training, and emerging trends like federated learning and neuromorphic computing. Our findings highlight the synergy between advanced algorithms, specialized hardware, and optimized system designs in pushing the boundaries of large-scale deep learning, paving the way for future breakthroughs in artificial intelligence.

Exploring Sampling Strategies and Genetic Diversity Analysis of Red Beet Germplasm Resources Using SSR Markers

By using 14 SSR primer pairs, we here analyzed and compared the amplification results of 534 DNA samples from six red sugar beet germplasm resources under three treatments. These data were used to explore the sampling strategy for the aforementioned resources. Based on the sampling strategy results, 21 SSR primer pairs were used to evaluate the genetic diversity of 47 red sugar beet germplasm resources. The six population genetic parameters used for exploring the sampling strategy unveiled that individual plants within the population had a relatively large genetic distance. The genetic parameters Ne, I, and Nei’s of the randomly mixed sampling samples increased rapidly between 10 and 30 plants before decreasing. Therefore, when SSR technology was used to analyze the genetic diversity of the red sugar beet germplasm resources, the optimal sampling gradient for each population was the adoption of a random single-plant mixed sampling sample of no less than 10 plants and no more than 30 plants. The 21 SSR primer pairs were used to detect genetic diversity in 30 random mixed samples of 47 resources. The average polymorphic information content (PIC) was 0.5738, the average number of observed alleles (Na) was 4.1905, the average number of effective alleles (Ne) was 2.8962, the average Shannon’s information index (I) was 1.1299, the average expected heterozygosity (Nei’s) was 0.6127, and the average expected heterozygosity (He) was 0.6127. The genetic distance of the 47 germplasm resources ranged from 0.0225 to 0.551 (average: 0.316). According to the population structure analysis, the most suitable K value was six, which indicated the presence of six populations. Based on the clustering analysis results, the 47 germplasm resources were segregated into six groups, with obvious clustering and some germplasm resources noted for having groups with close genetic relationships. We here established a more accurate and scientific sampling strategy for analyzing the genetic diversity of red sugar beet germplasm resources by using SSR molecular markers. The findings provide a reference for collecting and preserving red sugar beet germplasms and protecting their genetic diversity.