L2 Regularization Methods Research Articles

Geodetic data inversion to estimate a strain-rate field by introducing sparse modeling

Many studies have estimated crustal deformation from observed geodetic data. So far, because most studies have applied a smoothness constraint, which includes the assumption of local uniformity of a strain-rate field, localized strain rates near fault zones have tended to be underestimated when we invert spatially sporadic GNSS data. To overcome this difficulty, we introduce sparse modeling into the estimation of a strain-rate field. Specifically, we impose a sparsity constraint as well as the smoothness constraint on strain rates as prior information, which are expressed by the L1-norm and the L2-norm of the second-order derivative of the velocity field, respectively. To investigate the validity and limitation of the proposed method, we conduct synthetic tests, in which we consider an anti-plane strain problem due to a steady slip on a buried strike-slip fault. As a result, we find: (1) regardless of the locking depth of the fault, the proposed method reproduces localized strain rates near the fault with almost equal or better accuracy than the L2 regularization method (i.e., only the smoothness constraint); (2) the advantage of the proposed method over the L2 regularization method is clearer when data coverage is worse (i.e., when fewer observation points are available); and (3) the proposed method can be applied when observation errors are small. Next, we apply the proposed method to the GNSS data across the Arima-Takatsuki fault zone, which is one of the most active strike-slip faults in Japan. As a result, the proposed method estimates about 1.0×10-8/yr faster strain rates near the fault zone than the L2 regularization method, which corresponds to a 20–30% greater strain-rate concentration. The faster strain rates result in the estimation of a shallower locking depth: 11 km by the proposed method, compared to 17 km by the L2 regularization method. The former is closer to the depth of D90, 12–14 km, above which 90% of earthquakes occur.Graphical

A hybrid model based on the photovoltaic conversion model and artificial neural network model for short-term photovoltaic power forecasting

Photovoltaic (PV) power is greatly uncertain due to the random meteorological parameters. Therefore, accurate PV power forecasting results are significant for the dispatching of power and improving of system stability. This paper proposes a hybrid forecasting model for one-day-ahead PV power forecasting under different cloud amount conditions. The proposed model consists of an improved artificial neural network (ANN) algorithm and a PV power conversion model. First, the ANN model is designed to forecast the plane of array (POA) irradiance and ambient temperature. Backpropagation, gradient descent, and L2 regularization methods are applied in the structure of the ANN model to achieve the best weights, improve the prediction accuracy, and alleviate the effect of overfitting. Second, the PV power conversion model employs the forecasted results of POA irradiance and ambient temperature to determine the PV power produced by a PV module. In addition to the basic temperature factor, environmental efficiency and a reflection efficiency are incorporated into the conversion model to account for real PV module losses. The performance of the proposed model is validated with real weather and PV power data from Alice Springs and Climate Data Store. Results indicate that the model improves the forecast accuracy compared to four benchmark models. Specifically, it reduces root mean square error (RMSE) and normalized RMSE (nRMSE) by up to 25% under cloudy conditions and offers a 3% shorter training time compared to extreme gradient boosting.

A ship motion forecasting approach based on Fourier transform, regularized Bi-LSTM and chaotic quantum adaptive WOA

Ship motion forecasting is of great significance to assist ship navigation and ensure ship operation. However, due to the coupling influence of wind, waves, currents and other factors, the ship motion has a strong time variation, which brings great challenges to the improvement of the ship motion forecasting accuracy. This paper proposes a novel approach for ship motion forecasting. Firstly, Fourier Transform (FT) is used for data pre-processing to deal with the data noise. Secondly, the Bi-LSTM is applied to simulate the nonlinear dynamical system of ship motion, and the Dropout mechanism and l2 regularization method are used to improve the generalization performance of the network, and a novel ship motion prediction model (FRB model) is constructed. Thirdly, in view of the inherent defects of Whale Optimization Algorithm (WOA), based on chaotic mapping and quantum computing, a new Chaotic Quantum Adaptive Whale Optimization Algorithm (CQAWOA) is proposed to construct a new ship motion forecasting approach, namely FRB&CQAWOA. Finally, based on the measured pitch and roll data of ship, the feasibility and superiority of the established FRB model and the proposed CQAWOA are tested. The test results indicate that the proposed model have better prediction accuracy than other algorithms selected in this paper.

Wavelet denoising based on comprehensive index optimization and improved L2 regularization for load identification

Load identification, as an inverse problem, is still one of the challenges in the field of vibration engineering. In this paper, a comprehensive load identification method combining entropy weight Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) noise reduction evaluation algorithm and improved L2 regularization method is proposed to improve the load identification accuracy from the following two perspectives. Firstly, the entropy weight TOPSIS method is used to evaluate the wavelet denoising scheme comprehensively, and the optimal denoising scheme is found to reduce the noise interference in the process of inverse problem solving. Secondly, we introduce a new regularization filter function, and search for the regularization parameters by Generalized Cross-Validation (GCV) criterion and one-dimensional optimization algorithm to improve the ill-posedness of the inverse problem. Finally, the simulation and experimental verification of single-source load identification are carried out with the scaled crankcase model, while the accuracy is compared with the Tikhonov regularization method. The results show that the integrated algorithm proposed in this paper can improve the accuracy of load identification while overcoming the inadequacy of inverse problem. Unlike previous studies, the comprehensive evaluation method of denoising effect introduced in this paper provides a way to judge the denoising effect in practical engineering, and the experimental model is closer to practical engineering structures, which shows potential engineering application value.

Integrating L1 and weighted L2 regularization for moving force identification from combined response measurements

Existing regularization methods have been proven effective in the ill-posed problem of moving force identification (MFI). However, both the constant and time-varying components of moving forces cannot be accurately identified at the same time, especially under higher measurement noises. To address this issue, a new MFI method is proposed by integrating the L1 and weighted L2 regularization methods in this study, where the presented weighted L2 regularization strategy achieves a more nuanced balance of moving force components through the incorporation of weight coefficients. Furthermore, the inherent solving characteristics of L1 and weighted L2 regularization methods are fully utilized to solve different components in moving forces separately. Numerical simulations and experimental verifications are carried out to assess the effectiveness of the proposed method. The results show that the proposed method can accurately identify moving forces even under 25% higher noise levels and it can effectively evaluate the axle loads and gross vehicle weight of the experimental vehicle. Besides, comparative studies further show that the proposed method outperforms the existing L1 and L2 regularization methods in the identification of both constant and time-varying forces, which provides a potential MFI tool for a higher accuracy and noise robustness to fix the MFI problem in practice.

Prediction of friction coefficient of polymer surface using variational mode decomposition and machine learning algorithm based on noise features

Establishing a correlation between the friction coefficient and friction noise of metal-friction polymer interfaces is a challenging task in various environments. To address this issue, our study utilizes machine learning algorithms to construct a friction data-based model, elucidating the relationship between noise and friction coefficient. We propose the variational mode decomposition (VMD) along with five machine learning algorithms, each capturing unique data characteristics. Algorithm optimization is achieved through the implementation of L1 and L2 regularization methods. By comparing the regression results, we develop a tribological model that offers a novel approach for constructing a nonlinear model, enabling precise monitoring of the friction coefficient by leveraging noise signals.

A Feature Fusion Method for Driving Fatigue of Shield Machine Drivers Based on Multiple Physiological Signals and Auto-Encoder

The driving fatigue state of shield machine drivers directly affects the safe operation and tunneling efficiency of shield machines during metro construction. To cope with the problem that it is challenging to simulate the working conditions and operation process of shield machine drivers using driving simulation platforms and that the existing fatigue feature fusion methods usually show low recognition accuracy, shield machine drivers at Shenyang metro line 4 in China were taken as the research subjects, and a multi-modal physiological feature fusion method based on an L2-regularized stacked auto-encoder was designed. First, the ErgoLAB cloud platform was used to extract the combined energy feature (E), the reaction time, the HRV (heart rate variability) time-domain SDNN (standard deviation of normal-to-normal intervals) index, the HRV frequency-domain LF/HF (energy ratio of low frequency to high frequency) index and the pupil diameter index from EEG (electroencephalogram) signals, skin signals, pulse signals and eye movement data, respectively. Second, the physiological signal characteristics were extracted based on the WPT (wavelet packet transform) method and time–frequency analysis. Then, a method for driving fatigue feature fusion based on an auto-encoder was designed aiming at the characteristics of the L2-regularization method to solve the over-fitting problem of small sample data sets in the process of model training. The optimal hyper-parameters of the model were verified with the experimental method of the control variable, which reduces the loss of multi-modal feature data in compression fusion and the information loss rate of the fused index. The results show that the method proposed outperforms its competitors in recognition accuracy and can effectively reduce the loss rate of deep features in existing decision-making-level fusion.

Evolutionary polynomial regression improved by regularization methods.

Evolutionary polynomial regression (EPR) is a data mining tool that has been widely used in solving various geotechnical engineering problems. The fitness function is the core of EPR. However, overfitting may still occur in EPR, and this issue may cause the testing dataset not to perform effectively. Improvement of the EPR fitness function through L1 and L2 regularization methods is critical to avoid overfitting and enhance good generalization. First, the appropriate values of the regularization parameter λ of the L1 regularization method (L1RM) and L2 regularization method (L2RM) are determined by comparing the test data sets. Then, the EPR with a classical fitness function is compared with that of L1 or L2 regularization methods to evaluate their abilities in developing regression and producing accurate predictions. The results show that the fitness function combined with the regularization method could improve the EPR. However, L1RM performs better in prediction than L2RM. Improvement of EPR using L1RM could solve problems associated with construction constitutive models or could be used for prediction in geotechnical engineering.

Evaluation of the Elastic Modulus and Plateau Stress of a 2D Porous Aluminum Alloy Based on a Convolutional Neural Network

Porous metals are a new ultra-light material with high specific stiffness, specific strength, and good energy absorption properties. The elastic modulus and plateau stress of porous metals are essential parameters. There have been many studies on the effects of the matrix material, porosity, and pore size on the elastic modulus and plateau stress of porous metals, but few studies can be found on the impact of pore arrangement. The pore arrangement of porous metals cannot be quantitatively described, and the design space of a porous metal structure under the same porosity is vast. With the powerful learning and prediction ability of neural networks, the influence of pore arrangement can be better understood. In this paper, a convolutional neural network was used to explore the impact of pore arrangement on both the elastic modulus and plateau stress of a porous aluminum alloy. Firstly, a finite element method was used to simulate the compression of a porous aluminum alloy to obtain a training sample library. Secondly, a convolutional neural network was built to positively predict the elastic modulus and plateau stress of the porous aluminum alloy. Partial samples were used to select the best training model from five convolutional neural network candidates. Dropout, Batch Normalization, and L2 regularization methods were used to alleviate the over-fitting phenomenon in training. All data in the database were then trained and predicted, and the predicted goodness of fit of the elastic modulus and plateau stress were 0.8785 and 0.5922, respectively. A search method based on the convolutional neural network was then used to iteratively search the database. Under the condition of using a small amount of data, the pore structure with the best elastic modulus and plateau stress in the database could be determined, and the inverse design of a structure with high elastic modulus and plateau stress could be realized.

A locally weighted linear ridge regression framework for spatial interpolation of monthly precipitation over an orographically complex area

AbstractThis article aimed to investigate the potential of using three large‐scale precipitation products (PPs), including TRMM, ERA5‐Land, and MSWEP, and two land surface characteristics, including NDVI and Land Surface Temperature (LST), to improve the accuracy of an elevation‐based spatial non‐stationary, Locally Weighted Linear Regression (LWLR) method. A two‐step approach consisting of downscaling and merging was proposed and assessed across an orographically complex region in Iran to construct monthly precipitation maps for the 2001–2015 period. In the first step, three large‐scale PPs were downscaled to 1 km spatial resolution via the Area To Point Kriging (ATPK) method to address the spatial resolution discrepancy between the coarse pixels of PPs and pointwise gauge data. In the second step, five regression models with different combinations of predictors were developed and compared with the benchmark precipitation‐elevation regression model. The L2 regularization method was adopted to overcome the potential multicollinearity issue. The proposed framework could successfully diagnose the multicollinearity issues and dynamically estimate the regularization parameter in space and time. We utilized the holdout method to validate and compare the developed models, in which 50% of gauges were used for fitting the regression functions, and the remaining gauges were assigned to the validation dataset. The validation results showed that the overall monthly accuracy of the benchmark univariate elevation‐based model was improved where the MAE metric was decreased by 16.5%, and CC and KGE were increased by 3.4 and 5.8%, respectively. Also, the rBias was enhanced from −5.72 to 1.24%. Mean monthly precipitation maps of the 2001–2015 period generated by the model with the best combination of predictors and the benchmark model displayed the same spatial pattern consistent with the topography. Further analysis under different validation scenarios revealed that the contribution of PPs and auxiliary variables is the greatest when the training gauge network is sparse. Also, the developed multivariate model has a higher extrapolation ability and is more robust than the benchmark model.

Research on the identification method of safety wearing of electric power workers based on deep learning

Aiming at the difficulties of manual monitoring and compliance with the current wear identification of electric power workers, the detection and identification of safety helmets, work clothes, and insulating gloves are used to carry out normative identification and warning, and a deep learning-based power worker safety wear identification method is proposed in this paper. The AlexNet and Inception are introduced to increase the width and depth of the artificial neural network. At the same time, the ReLU activation function with better performance is used to reduce the amount of network computation, and the Global Average Pooling layer is used to replace the fully connected layer with more parameters. The improved convolution neural network model has a total of 13 layers. In order to prevent the network from overfitting, the Early-stopping mechanism and the L2 regularization method are used to improve the performance of the network model. The experimental results show that the algorithm can achieve a good recognition effect on the staff who do not wear safety according to the regulations in the video, and the feasibility and effectiveness of the algorithm in practical application are verified.

A Novel Method for Staggered SAR Imaging in an Elevation Multichannel System

Synthetic aperture radar (SAR) is an advanced remote sensing technique, capable of observing Earth’s surface independent of weather conditions and sunlight illumination. Restricted by the minimum antenna area, however, conventional spaceborne SAR systems cannot achieve high azimuth resolution in a wide swath. In addition, blind ranges are present as the constant pulse repetition interval (PRI) is used. To solve these problems, a PRI-staggered elevation multichannel SAR (EMC-SAR) system is employed in this article. By transmitting the continuously PRI-varied sequence, the blind ranges are located at different regions in different receive instants, effectively avoiding the loss of coverage in elevation. In this system, three issues are required to be addressed: 1) recovering the missed data located at blind ranges; 2) suppressing range ambiguous components; and 3) restoring the PRI-varied signal into a regular grid. To deal with these problems, we propose a novel SAR imaging method for a PRI-staggered EMC-SAR system. To be applied on-ground, assume downlinking of the individual elevation channels. First, the modified <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> -insensitive loss tube regression with the L2 regularization method is applied to recover the missed data. Then, the range ambiguous components are suppressed by performing digital beamforming (DBF) based on the elevation multichannel technique, where the covariance matrix is constructed by using an iterative adaptive algorithm. After that, a generalized scaling transform is employed to restore the PRI-varied signal into a uniform sampled grid. Finally, a well-focused SAR image can be obtained by performing the conventional SAR imaging techniques. The effectiveness of the proposed method is validated by both simulated and real SAR data processing results.

Research on Web GUI Image Recognition System Based on Improved Convolutional Neural Network

In this paper, a MobileNet V2 convolutional neural network depending on L2 regularization method, a amended particle swarm model and Dropout method are constructed in a bid for enhancing the accuracy and speed of Web GUI image recognition for establishing Web Web GUI image recognition system of Web GUI test. Firstly, an improved MobileNet V2 convolutional neural network is constructed. Secondly, the basic models of L2 regularization method, improved particle swarm method and Dropout method are studied, the improved MobileNet V2 convolution neural network optimization algorithm is proposed, and the basic model of image processing is designed. Finally, simulation analysis is carried out on Web GUI image recognition through using the amended convolutional neural network on basis of MINIST data set as the research object. The simulation results illustrate that the amended convolutional neural network proposed in this study is more accurate and efficient in Web GUI image recognition.

A new image reconstruction strategy for TMR-EMT: combining regularization theory with guided image filtering method

Electromagnetic tomography based on tunneling magnetoresistance (TMR-EMT) can be used to obtain the solid phase (magnetic catalyst) distribution in a gas–liquid–solid three-phase fluidized bed based on changes in permeability. However, the TMR-EMT system has a higher sensitivity near the TMR sensor and lower sensitivity in other positions, which makes the ill-conditioned property of image reconstruction more serious compared with the traditional coil measurement EMT system. As a result, the reconstructed image quality of the solid phase distribution is low. Aiming to address this problem, a new image reconstruction strategy, based on guided image filtering (GIF) and regularization theory, is proposed for TMR-EMT to improve the reconstruction quality of the solid phase distribution in a gas–liquid–solid three-phase fluidized bed. First, the L2 regularization method and L1 regularization method are used to reconstruct the image of the permeability distribution in the region of interest (ROI). On this basis, the reconstructed images of the L2 regularization and L1 regularization are used as the input image and guided image of GIF respectively for filtering the output. Finally, the image of solid phase distribution in the ROI is obtained according to the reconstructed image of the permeability. Simulation and experimental results indicate that the proposed strategy can take into account the sparsity of L1 regularization and the smoothness of L2 regularization simultaneously, and obtain a higher image reconstruction quality.

DBFU-Net: Double branch fusion U-Net with hard example weighting train strategy to segment retinal vessel.

BackgroundMany fundus imaging modalities measure ocular changes. Automatic retinal vessel segmentation (RVS) is a significant fundus image-based method for the diagnosis of ophthalmologic diseases. However, precise vessel segmentation is a challenging task when detecting micro-changes in fundus images, e.g., tiny vessels, vessel edges, vessel lesions and optic disc edges.MethodsIn this paper, we will introduce a novel double branch fusion U-Net model that allows one of the branches to be trained by a weighting scheme that emphasizes harder examples to improve the overall segmentation performance. A new mask, we call a hard example mask, is needed for those examples that include a weighting strategy that is different from other methods. The method we propose extracts the hard example mask by morphology, meaning that the hard example mask does not need any rough segmentation model. To alleviate overfitting, we propose a random channel attention mechanism that is better than the drop-out method or the L2-regularization method in RVS.ResultsWe have verified the proposed approach on the DRIVE, STARE and CHASE datasets to quantify the performance metrics. Compared to other existing approaches, using those dataset platforms, the proposed approach has competitive performance metrics. (DRIVE: F1-Score = 0.8289, G-Mean = 0.8995, AUC = 0.9811; STARE: F1-Score = 0.8501, G-Mean = 0.9198, AUC = 0.9892; CHASE: F1-Score = 0.8375, G-Mean = 0.9138, AUC = 0.9879).DiscussionThe segmentation results showed that DBFU-Net with RCA achieves competitive performance in three RVS datasets. Additionally, the proposed morphological-based extraction method for hard examples can reduce the computational cost. Finally, the random channel attention mechanism proposed in this paper has proven to be more effective than other regularization methods in the RVS task.

Protein Post-translational Modification Site Prediction using Deep Learning

The study and identification of protein post-translational modification(PTM) sites in rice seeds plays an important role in breeding and yield increasing in intelligent agriculture. Deep learning, especially deep convolutional neural networks(CNN) become more and more popular in intelligent agriculture because of its excellent image processing and data analysis capabilities. In this paper, the Denoised-Oversampling is adopted to denoise the samples and balance the data. The protein sequences are transformed into corresponding numerical feature vectors. Then, the processed feature subset is input to the deep convolution network. In fully connected layers, we used the L2 regularization method to prevent the overfitting problem. The phosphorylation sites are predicted by the sigmoid classifier. Finally, in the experimental part, in order to truly reflect the predictive power of the model, we divided the dataset into training set, validation set and independent test set. The prediction accuracy of the independent test set is 83.20%. The test result shows that the proposed algorithm has better prediction performance than the existing phosphorylation prediction algorithms.

Effect on model performance of regularization methods

Artificial Neural Networks with numerous parameters are tremendously powerful machine learning systems. Nonetheless, overfitting is a crucial problem in such networks. Maximizing the model accuracy and minimizing the amount of loss is significant in reducing in-class differences and maintaining sensitivity to these differences. In this study, the effects of overfitting for different model architectures with the Wine dataset were investigated by Dropout, AlfaDropout, GausianDropout, Batch normalization, Layer normalization, Activity normalization, L1 and L2 regularization methods and the change in loss function the combination with these methods. Combinations that performed well were examined on different datasets using the same model. The binary cross-entropy loss function was used as a performance measurement metric. According to the results, the Layer and Activity regularization combination showed better training and testing performance compared to other combinations.

An outdoor fire recognition algorithm for small unbalanced samples

Deep learning (DL) is a hot topic in the machine vision. A large number of data sets are necessary for efficient image recognition. Otherwise the overfitting will easily occur. However, most actual samples are limited and unbalanced. To diminish the negative impact of small unbalanced samples on image recognition, the Deep Convolutional Generative Adversarial Network (DC-GAN) was improved to simulate data distribution, and relied on the improved network to generate a highly diverse dataset of balanced fire images in the work. Then, the number of output layer nodes was finetuned for the training of the target dataset by the layer freezing method. The training of small unbalanced samples was realized, using exponentially decaying learning rate, L2 regularization method, and Adam optimization algorithm. Simulation results showed that the proposed algorithm converged faster by fixing the convolutional layer parameters of the pre-trained model and finetuning the fully connected layer through transfer learning. Besides, 99% of fire images were correctly recognized, without inducing the problem of small sample overfitting. The proposed algorithm provides a desirable tool for outdoor fire recognition.

A novel transfer diagnosis method under unbalanced sample based on discrete-peak joint attention enhancement mechanism

Deep learning has been widely used in intelligent fault diagnosis field in the era of industrial internet due to its excellent big data analysis ability. However, it lacks the specific attention enhancement mechanism in the training process, and is difficult to be applied to the speed transfer fault diagnosis under unbalanced training sample condition. Considering these challenges, a neural network called discrete-peak joint attention enhancement (DPJAE) convolutional model is proposed. First, the proposed discrete channel attention enhancement method is applied to enhance the characteristic discreteness of convolutional neural network channels. Then, the peak channel attention enhancement method follows, which is employed to enhance feature peaks in convolutional neural network channels. Finally, the discrete-peak joint attention enhancement method is placed in the middle of a well-designed robust CNN architecture to enhance the discreteness and peak value for channel characteristics. In order to avoid the over-fitting phenomenon of high variance in the training process of the deep network model, L2 regularization method is introduced to regularize the weights for the front layer network. The diagnosis results on the open test dataset show that the DPJAE model not only gets high accuracy in the unbalanced training samples, but also achieves high performance in the rotating speed transfer testing samples. The proposed method is also verified on the private dataset. Compared with other diagnostic methods, the method has better diagnostic results.

Methodology of Data Fusion Using Deep Learning for Semantic Segmentation of Land Types in the Amazon

This study proposes a methodology using deep learning and a multi-resolution segmentation algorithm to perform the semantic segmentation of remote sensing images. Initially the image is segmented using a CNN, and then an image with homogeneous regions is generated using a multi-resolution segmentation algorithm. Finally, a data fusion process is performed with these two images, generating the final classified image. The field of study was the Brazilian Amazon region. The proposed methodology classifies images in the following classes: forest, pasture and agriculture. The input data used were LANDSAT-8/OLI images. The reference data were extracted from the results of the TerraClass project in 2014. Two datasets were evaluated: the first with six bands and the second with three bands. Three CNN architectures were evaluated together with three optimization methods: SGDM, ADAM, and RMSProp and the dropout and L2 regularization methods as methods for generalization improvement. The best model, CNN + optimization method + technique for generalization improvement, evaluated in the validation set, was submitted to a 5-fold cross validation methodology, and the results were compared with pre-trained networks using the learning transfer methodology; in this case the networks used for comparison were ResNet50, InceptionResnetv2, MobileNetv2 and Xception. The proposed methodology was evaluated through image segmentation of some regions of the Amazon. Finally, the proposed methodology was evaluated in regions used by other authors. The accuracy values obtained for the images evaluated were over 99%.