Kernel Feature Research Articles

Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks*

Abstract We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Starting from a dynamical mean field theory description of infinite width deep neural network kernel and prediction dynamics, we provide a characterization of the O ( 1 / width ) fluctuations of the dynamical mean field theory order parameters over random initializations of the network weights. Our results, while perturbative in width, unlike prior analyses, are non-perturbative in the strength of feature learning. We find that once the mean field/µP parameterization is adopted, the leading finite size effect on the dynamics is to introduce initialization variance in the predictions and feature kernels of the networks. In the lazy limit of network training, all kernels are random but static in time and the prediction variance has a universal form. However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently. In two layer networks, we show how feature learning can dynamically reduce the variance of the final tangent kernel and final network predictions. We also show how initialization variance can slow down online learning in wide but finite networks. In deeper networks, kernel variance can dramatically accumulate through subsequent layers at large feature learning strengths, but feature learning continues to improve the signal-to-noise ratio of the feature kernels. In discrete time, we demonstrate that large learning rate phenomena such as edge of stability effects can be well captured by infinite width dynamics and that initialization variance can decrease dynamically. For convolutional neural networks trained on CIFAR-10, we empirically find significant corrections to both the bias and variance of network dynamics due to finite width.

Automatic identification of MS lesions based on local steering kernel features and sparse dictionary training

Automatic identification of MS lesions based on local steering kernel features and sparse dictionary training

A Ship’s Maritime Critical Target Identification Method Based on Lightweight and Triple Attention Mechanisms

The ability to classify and recognize maritime targets based on visual images plays an important role in advancing ship intelligence and digitalization. The current target recognition algorithms for common maritime targets, such as buoys, reefs, other ships, and bridges of different colors, face challenges such as incomplete classification, low recognition accuracy, and a large number of model parameters. To address these issues, this paper proposes a novel maritime target recognition method called DTI-YOL (DualConv Triple Attention InnerEIOU-You Only Look Once). This method is based on a triple attention mechanism designed to enhance the model’s ability to classify and recognize buoys of different colors in the channel while also making the feature extraction network more lightweight. First, the lightweight double convolution kernel feature extraction layer is constructed using group convolution technology to replace the Conv structure of YOLOv9 (You Only Look Once Version 9), effectively reducing the number of parameters in the original model. Second, an improved three-branch structure is designed to capture cross-dimensional interactions of input image features. This structure forms a triple attention mechanism that accounts for the mutual dependencies between input channels and spatial positions, allowing for the calculation of attention weights for targets such as bridges, buoys, and other ships. Finally, InnerEIoU is used to replace CIoU to improve the loss function, thereby optimizing loss regression for targets with large scale differences. To verify the effectiveness of these algorithmic improvements, the DTI-YOLO algorithm was tested on a self-made dataset of 2300 ship navigation images. The experimental results show that the average accuracy of this method in identifying seven types of targets—including buoys, bridges, islands and reefs, container ships, bulk carriers, passenger ships, and other ships—reached 92.1%, with a 12% reduction in the number of parameters. This enhancement improves the model’s ability to recognize and distinguish different targets and buoy colors.

Expressibility-induced concentration of quantum neural tangent kernels

Quantum tangent kernel methods provide an efficient approach to analyzing the performance of quantum machine learning models in the infinite-width limit, which is of crucial importance in designing appropriate circuit architectures for certain learning tasks. Recently, they have been adapted to describe the convergence rate of training errors in quantum neural networks in an analytical manner. Here, we study the connections between the expressibility and value concentration of quantum tangent kernel models. In particular, for global loss functions, we rigorously prove that high expressibility of both the global and local quantum encodings can lead to exponential concentration of quantum tangent kernel values to zero. Whereas for local loss functions, such issue of exponential concentration persists owing to the high expressibility, but can be partially mitigated. We further carry out extensive numerical simulations to support our analytical theories. Our discoveries unveil a fundamental feature of quantum neural tangent kernels, indicating that the issue of their concentration cannot be bypassed merely by transitioning to a local encoding scheme while maintaining high expressibility. This offers valuable insights for the design of wide quantum variational circuit models in practical applications.

DeepKernel: 2D-kernels clustering based mutant reduction for cost-effective deep learning model testing

Mutation testing is a practical approach for evaluating the quality of deep learning (DL) testing datasets. However, the enormous mutants during testing lead to significant testing overhead. Feature clustering is a conventional method that reduces the number of mutants while preserving the mutants’ distribution diversity. This distribution diversity is considered crucial for maintaining the effectiveness of testing assessment ability. DL model relies on convolutional kernels to extract data features and construct logic. Thus, using kernels to measure the differences among DL mutants is a feasible approach. This paper proposes DeepKernel, a convolutional kernel features clustering based reduction method. Specifically, it considers 2D-Kernel sparsity and 2D-Kernel entropy as kernel features. The features are clustered to construct a subset with equivalent testing assessment capability to the original set. Empirical studies on four classical DL models demonstrate that: (1) there is a significant correlation between the distribution diversity of the mutants and their testing assessment ability, as indicated by a Spearman Correlation Coefficient of 0.9689. (2) the reduced set maintains a similar distribution diversity and testing effectiveness as the original set. (3) when preserving the effectiveness of the mutation testing, our method reduces 63.47% of mutants and outperforms random selection.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Investigation of the impact of local properties of 3D data on the accuracy of block matching-based speckle tracking echocardiography

Abstract Introduction: Due to its simplicity, block-matching is a popular motion-tracking method used in speckle-tracking echocardiography. Improvement of its robustness and accuracy is thus of prime interest. Although it seems plausible that the quality of block matching-based tracking depends on the local properties of image data, and thus, it should be possible to assess in advance how well certain portions of the image data are suited for displacement estimation, the potential relationship has not been studied extensively. Material and methods: This study aimed to search for a relationship between selected features of echocardiographic data and the quality of local displacement estimation. The study used a 3D synthetic B-mode imaging sequence data with known ground truth. Frame-to-frame displacements were estimated for 9856 points in five different frame pairs with mean inter-frame displacements of 0.15, 0.87, 2, 3.02, and 3.84 mm. In each case, tracking errors were evaluated against thirteen grayscale image features, the displacement’s magnitudes, and the normalized cross-correlation (NCX) values. Additionally, a multi-variable regression model was applied to test the combined ability of the proposed features to predict tracking quality. Results: Median tracking error magnitudes were 0.06, 0.13, 0.28, 0.74, and 1.5 mm for each image pair. Weak correlation between errors and individual data features was found only in the case of 3 features: NCX (Pearson’s correlation coefficients in the range of −0.366 to −0.223), number of speckles within the kernel (−0.283, −0.282, and −0.214 for three lowest deformations) and mean of the 3D gradient (−0.252, −0.237 and −0.25). The regression model, however, provided significant prediction improvement with R2 exceeding 0.5. Conclusions: In conclusion, only a weak relationship between the individual investigated kernel features and tracking accuracy has been established, but their combined strength can be assessed as at least moderate.

Sentiment analysis on twitter about the death penalty using the support vector machine method

It is estimated that 175 million people in Indonesia utilize the Internet, according to the most recent We Are Social survey. 160 million of them are internet users who utilize social media, according to this data. It is estimated that 19.5 million Indonesians use Twitter. This is consistent with the numerous tweets that users have posted on Twitter about a variety of topics, including politics, music, health, and education. The death penalty is still one of the most popular subjects that is addressed on Twitter. When a judge rules that someone will be executed as retribution for a crime they have committed, this is referred to as the death penalty. As a result, sentiment analysis utilizing the Support Vector Machine technique with linear kernel features and Python programming was used to study public opinions on the death sentence. To improve the accuracy of the results obtained, data labeling on 848 data that were received through the scraping process was done manually in this study. Positive data is categorized as belonging to the class that supports the death sentence, while negative data is categorized as belonging to the class that opposes it. The study that was done shows an 8:2 difference between the training and test data. After preprocessing a dataset containing 758 data points, of which 606 will be utilized for training and 152 for testing, we obtain 91% accuracy, 91% precision, 100% recall, and 95% f1-score

YOLO-ADual: A Lightweight Traffic Sign Detection Model for a Mobile Driving System

Traffic sign detection plays a pivotal role in autonomous driving systems. The intricacy of the detection model necessitates high-performance hardware. Real-world traffic environments exhibit considerable variability and diversity, posing challenges for effective feature extraction by the model. Therefore, it is imperative to develop a detection model that is not only highly accurate but also lightweight. In this paper, we proposed YOLO-ADual, a novel lightweight model. Our method leverages the C3Dual and Adown lightweight modules as replacements for CPS and CBL modules in YOLOv5. The Adown module effectively mitigates feature loss during downsampling while reducing computational costs. Meanwhile, C3Dual optimizes the processing power for kernel feature extraction, enhancing computation efficiency while preserving network depth and feature extraction capability. Furthermore, the inclusion of the CBAM module enables the network to focus on salient information within the image, thus augmenting its feature representation capability. Our proposed algorithm achieves a mAP@0.5 of 70.1% while significantly reducing the number of parameters and computational requirements to 51.83% and 64.73% of the original model, respectively. Compared to various lightweight models, our approach demonstrates competitive performance in terms of both computational efficiency and accuracy.

Chemical composition prediction in goji (Lycium barbarum) using hyperspectral imaging and multi-task 1DCNN with attention mechanism

The bioactive components of goji berries (Lycium barbarum) are crucial determinants of their nutritional and commercial value. In this study, we combined hyperspectral imaging technology (HSI) with a one-dimensional convolutional neural network (1DCNN) to predict the content of chemical compositions in goji. To enhance the model's ability to focus on relevant information, we introduced the channel attention module (CAM), spectral attention module (SAM) and fused them together, which can focus on output features of the convolution kernels differently and adaptively emphasize more effective spectral bands, respectively. Moreover, considering the limitations of traditional single-task prediction methods and the inherent correlations among different constituents, we employed a multi-task CNN for the simultaneous prediction of various goji constituents. The results indicate that the attention-enhanced 1DCNN model outperforms both the partial least squares regression (PLSR) model and the vanilla 1DCNN. With multi-task learning, the model achieves optimal performance, achieving an average R2 of 0.9435 for the prediction of the three components. Our research develops an efficient and accurate method for predicting the constituents of goji berries, providing a valuable, convenient, and effective tool for assessing and detecting the quality of fruit products in the food industry.

Power Quality Transient Disturbance Diagnosis Based on Dynamic Large Convolution Kernel and Multi-Level Feature Fusion Network

Power quality is an important metric for the normal operation of a power system, and the accurate identification of transient signals is of great significance for the improvement of power quality. The diverse types of power system transient signals and strong characteristic coupling brings new challenges to the analysis and identification of power system transient signals. In order to enhance the identification accuracy of transient signals, one method of power system transient signal identification is proposed based on a dynamic large convolution kernel and multilevel feature fusion network. First, the more fine-grained and more informative features of the transient signals are extracted by the dynamic large convolution kernel feature extraction module. Then, the multi-scale local features are adaptively fused by the multilevel feature fusion module. Finally, the fused features are reduced in dimension by the fully connected layer in the classification module and fed into the SoftMax layer for transient signal type detection. The proposed method can effectively improve the small receptive field problem of convolutional neural networks and the lack of ability of Transformer network in extracting local context information. Compared with five other power quality transient disturbance identification models, the experimental results show that the proposed method has better diagnostic accuracy and anti-noise capability.

Real-Time Detection Technology of Corn Kernel Breakage and Mildew Based on Improved YOLOv5s

In order to solve low recognition of corn kernel breakage degree and corn kernel mildew degree during corn kernel harvesting, this paper proposes a real-time detection method for corn kernel breakage and mildew based on improved YOlOv5s, which is referred to as the CST-YOLOv5s model algorithm in this paper. The method continuously obtains images through the discrete uniform sampling device of corn kernels and generates whole corn kernels, breakage corn kernels, and mildew corn kernel dataset samples. We aimed at the problems of high similarity of some corn kernel features in the acquired images and the low precision of corn kernel breakage and mildew recognition. Firstly, the CBAM attention mechanism is added to the backbone network of YOLOv5s to finely allocate and process the feature information, highlighting the features of corn breakage and mildew. Secondly, the pyramid pooling structure SPPCPSC, which integrates cross-stage local networks, is adopted to replace the SPPF in YOLOv5s. SPP and CPSC technologies are used to extract and fuse features of different scales, improving the precision of object detection. Finally, the original prediction head is converted into a transformer prediction head to explore the prediction potential with a multi-head attention mechanism. The experimental results show that the CST-YOLOv5s model has a significant improvement in the detection of corn kernel breakage and mildew. Compared with the original YOLOv5s model, the average precision (AP) of corn kernel breakage and mildew recognition increased by 5.2% and 7.1%, respectively, and the mean average precision (mAP) of all kinds of corn kernel recognition is 96.1%, and the frame rate is 36.7 FPS. Compared with YOLOv4-tiny, YOLOv6n, YOLOv7, YOLOv8s, and YOLOv9-E detection model algorithms, the CST-YOLOv5s model has better overall performance in terms of detection accuracy and speed. This study can provide a reference for real-time detection of breakage and mildew kernels during the harvesting process of corn kernels.

Few-shot fault diagnosis of turnout switch machine based on flexible semi-supervised meta-learning network

The safety of train operations hinges on the reliability of the signal system, and the switch machine stands out as a pivotal component within it. Consequently, fault diagnosis of switch machines is of paramount importance. However, obtaining a substantial amount of fault data is challenging in reality, and labeled data is even scarcer, which makes the fault diagnosis model of the switch machine have low diagnostic accuracy and poor generalization ability. To address these problems, a flexible semi-supervised meta-learning network (FSMN) is proposed for the fault diagnosis of switch machines in this paper. Firstly, a dual-channel hetero-convolution kernel feature extractor (DHKFE) is efficiently proposed to extract the switch machine fault features at different levels from few-shot samples. Secondly, a flexible distance prototype corrector is employed to adaptively modify the distance function. It accomplishes this by rapidly identifying similarities among fault samples and harnessing the potential of unlabeled data to fine-tune prototype positions, which can enhance prototype stability and generalization, leading to more accurate fault classification. Finally, the A-phase current data collected in the real scene during the transition between the two states of the switch machine are utilized for the validation of FSMN, alongside a comparative assessment against five other methods. The results show that the accuracy in forward-reverse and reverse-forward of FSMN is up to 97.35% and 92.72%, respectively, which means FSMN is superior in few-shot fault diagnosis and can be applied to various switch machines.

Analysis of selected deep features with CNN-SVM-based for bread wheat seed classification

The main ingredient of flour is processed wheat. Wheat is an agricultural product that is harvested once a year. It may be necessary to choose the variety of wheat for growing wheat and efficient harvesting. The variety of wheat is important for its economic value, taste, and crop yield. Although there are many varieties of wheat, they are very similar in colour, size, and shape, and it requires expertise to distinguish them by eye. This is very time consuming and can lead to human error. Using computer vision and artificial intelligence, such problems can be solved more quickly and objectively. In this study, an attempt was made to classify five bread wheat varieties belonging to different cultivars using Convolutional Neural Network (CNN) models. Three approaches have been proposed for classification. First, pre-trained CNN models (ResNet18, ResNet50, and ResNet101) were trained for bread wheat cultivars. Second, the features extracted from the fc1000 layer of the pre-trained CNN models ResNet18, ResNet50, and ResNet101 were classified using a support vector machine (SVM) classifier with different kernel features from machine learning techniques for classification with different variants. Finally, SVM methods were used in the second stage to classify the features obtained from the fc1000 layer of the pre-trained CNN models with an optimal set of features that can represent all features using the minimum redundancy maximum relevance (mRMR) feature selection algorithm.The accuracies obtained in the first, second, and last phases are as follows. In the first phase, the most successful method in classifying wheat grains was the ResNet18 model with 97.57%. In the second phase, the ResNet18 + ResNet50 + ResNet101 + Quadratic SVM model was the most successful model in classification using the features obtained from the ResNet CNN models with 94.08%.The accuracy for classification with the 1000 most effective features selected by the feature selection algorithm was 94.51%. Although the classification with features is slightly lower than deep learning, the classification time is much shorter and is 93%. This result confirms the great effectiveness of CNN models for wheat grain classification.

Unraveling Convolution Neural Networks: A Topological Exploration of Kernel Evolution

Convolutional Neural Networks (CNNs) have become essential in deep learning applications, especially in computer vision, yet their complex internal mechanisms pose significant challenges to interpretability, crucial for ethical applications. Addressing this, our paper explores CNNs by examining their topological changes throughout the learning process, specifically employing persistent homology, a core method within Topological Data Analysis (TDA), to observe the dynamic evolution of their structure. This approach allows us to identify consistent patterns in the topological features of CNN kernels, particularly through shifts in Betti curves, which is a key concept in TDA. Our analysis of these Betti curves, initially focusing on the zeroth and first Betti numbers (respectively referred to as Betti-0 and Betti-1, which denote the number of connected components and loops), reveals insights into the learning dynamics of CNNs and potentially indicates the effectiveness of the learning process. We also discover notable differences in topological structures when CNNs are trained on grayscale versus color datasets, indicating the need for more extensive parameter space adjustments in color image processing. This study not only enhances the understanding of the intricate workings of CNNs but also contributes to bridging the gap between their complex operations and practical, interpretable applications.

HKFGCN: A novel multiple kernel fusion framework on graph convolutional network to predict microbe-drug associations

Accumulating clinical studies have consistently demonstrated that the microbes in the human body closely interact with the human host, actively participating in the regulation of drug effectiveness. Identifying the associations between microbes and drugs can facilitate the development of drug discovery, and microbes have become a new target in antimicrobial drug development. However, the discovery of microbe-drug associations relies on clinical or biological experiments, which are not only time-consuming but also financially burdensome. Thus, the utilization of computational methods to predict microbe-drug associations holds promise for reducing costs and enhancing the efficiency of biological experiments. Here, we introduce a new computational method, called HKFGCN (Heterogeneous information Kernel Fusion Graph Convolution Network), to predict the microbe-drug associations. Instead of extracting feature from a single network in previous studies, HKFGCN separately extracts topological information features from different networks, and further refines them by generating Gaussian kernel features. HKFGCN consists of three main steps. Firstly, we constructed two similarity networks and a microbe-drug association network based on numerous biological data. Second, we employed two types of encoders to extract features from these networks. Next, Gaussian kernel features were obtained from the drug and microbe features at each layer. Finally, we reconstructed the bipartite microbe-drug graph based on the learned representations. Experimental results demonstrate the excellent performance of the HKFGCN model across different datasets using the cross-validation scheme. Additionally, we conduced case studies on human immunodeficiency virus, and the results were corroborated by existing literatures. The prediction model’s code is available at https://github.com/roll-of-bubble/HKFGCN.

Reciprocal sensitivity kernels for receiver positioning

In their classical formulation, sensitivity kernels aim to evaluate the points where an infinitesimal variation in the properties of the medium would most affect a pressure (or displacement) measurement at a fixed location. These kernels feature typical cigar shapes, with characteristically low sensitivity on the direct path. In this presentation, the same approach is applied to the reciprocal problem, where we want to evaluate, at a constant source position, which points are the most (or least) sensitive to a given model change. These reciprocal kernels exhibit flared shapes centered on the areas where the model is perturbed. Applications to receiver positioning in acoustic are discussed.

Meta-learning with deep flow kernel network for few shot cross-domain remaining useful life prediction

Reliable prediction of the remaining useful life (RUL) is important for improving maintenance efficiency, equipment availability, and avoiding catastrophic accidents in complex industrial systems. Existing RUL prediction models have made some contribution, relying mainly on a large amount of degraded data with similar patterns or approximate distributions. However, in practical industrial systems, only a small amount of labeling data is usually available, which may also come from different devices and different working conditions, resulting in different distributions in the degraded data. This situation makes the existing RUL methods difficult to achieve satisfactory generalization performance. To address this challenge, this paper proposes a novel Meta-Learning with Deep Flow Kernel Network (MetaDFKN) model for RUL prediction under the few shot and cross-domain conditions. The model first learns kernel features in a data-driven manner and considers them as latent variables to improve the model’s representative ability of shared knowledge between tasks. Then, we introduce the conditional normalized flow technique to infer richer posterior distributions in the kernel features, which helps to obtain feature information with stronger discriminative power. Moreover, shared knowledge and task-specific information are integrated into the contextual inference process, which can mine the dependencies of related tasks and capture richer domain information. Finally, to evaluate the proposed model, we conduct extensive experiments on engine and bearings degradation data, and the results verify the superiority of the model.

MLBKFD: Probabilistic Model Methods to Infer Pseudo Trajectories from Single-cell Data

Cell trajectory inference is very important to understand the details of tissue cell development, state differentiation and gene dynamic regulation. However, due to the high noise and heterogeneity of the single-cell data, it is challenging to infer cell trajectory in complex biological processes. Here, we proposed a new trajectory inference method, called Metric Learning Bhattacharyya Kernel Feature Decomposition (MLBKFD). In MLBKFD, a statistical model was used to infer cell trajectory by calculating the Markov transition matrix and Bhattacharyya kernel matrix between cells. Before that, to expedite the matrix calculation in the statistical model, a deep feedforward neural network was used to perform dimensionality reduction on single-cell data. The MLBKFD was evaluated on four typical datasets as well as seven recent human fetal lung datasets. Comparisons with the two outstanding methods (i.e., DTFLOW and MARGARET) demonstrate that the MLBKFD is capable of accurately inferring cell development and differentiation trajectories from single-cell data with different sizes and sources. Notably, MLBKFD exhibits nearly twice the speed of DTFLOW while maintaining high precision, particularly when dealing with large datasets. MLBKFD provides accurate and efficient trajectory inference, empowering researchers to gain deeper insights into the complex dynamics of cell development and differentiation.

Edge‐Preserved Tversky Indexive Hellinger with Deep Perceptive Czekanowski‐Based Image Classification

The texture is identifiable in optical and easy ways. Texture classification is an imperative region in texture analysis, where it gives descriptors for classifying the images. The categorization of normal and abnormal matter by magnetic resonance (MR), computed tomography (CT), and texture images has made noteworthy evolution in modern years. Recently, different novel robust classification techniques have been introduced to classify the different kinds of images for prediction. However, the accuracy of classification was not improved with lesser time. To address these issues, the edge‐preserved Tversky indexive Hellinger and deep perceptive Czekanowski classifier (ETIH‐DPCC) technique is introduced to segment and classify the images with more accuracy. The ETIH‐DPCC technique includes diverse processes namely preprocessing, segmentation, feature extraction, as well as classification. At first, different types of images, such as magnetic resonance imaging, CT, and texture, are used as input. With the acquired input, edge‐preserving normalized adaptive bilateral filtering is employed to carry the image preprocessing. In this stage, the noisy pixels are removed and edges are preserved. Then, the Tversky‐indexed quantile regression is applied to segment the images into diverse texture regions. After that, the feature extraction is performed on the segmented region using Hellinger kernel feature extraction, where a more informative feature for image prediction is extracted. During this process, the irrelevant features are avoided to decrease the dimensionality and feature extraction time. These extracted features are finally classified into positive and negative classes for disease prediction using DPCC. DPCC comprises multiple layers to deeply analyze the association between training and testing features. With this, the prediction accuracy is improved. Experimental outcomes show that the ETIH‐DPCC technique efficiently enhances prediction accuracy and less time compared to conventional methods.

LSKF-YOLO: Large Selective Kernel Feature Fusion Network for Power Tower Detection in High-Resolution Satellite Remote Sensing Images

LSKF-YOLO: Large Selective Kernel Feature Fusion Network for Power Tower Detection in High-Resolution Satellite Remote Sensing Images