Training-free Method Research Articles

SF-GPT: A training-free method to enhance capabilities for knowledge graph construction in LLMs

Knowledge graphs (KGs) are constructed by extracting knowledge triples from text and fusing knowledge, enhancing information retrieval efficiency. Current methods for knowledge triple extraction include ”Pretrain and Fine-tuning” and Large Language Models (LLMs). The former shifts effort from manual extraction to dataset annotation and suffers from performance degradation with different test and training set distributions. LLMs-based methods face errors and incompleteness in extraction. We introduce SF-GPT, a training-free method to address these issues. Firstly, we propose the Entity Extraction Filter (EEF) module to filter triple generation results, addressing evaluation and cleansing challenges. Secondly, we introduce a training-free Entity Alignment Module based on Entity Alias Generation (EAG), tackling semantic richness and interpretability issues in LLM-based knowledge fusion. Finally, our Self-Fusion Subgraph strategy uses multi-response self-fusion and a common entity list to filter triple results, reducing noise from LLMs’ multi-responses. In experiments, SF-GPT showed a 55.5% increase in recall and a 32.6% increase in F1 score on the BDNC dataset compared to the UniRel model trained on the NYT dataset and achieved a 5% improvement in F1 score compared to GPT-4+EEF baseline on the WebNLG dataset in the case of a fusion round of three. SF-GPT offers a promising way to extract knowledge from unstructured information.

Training-free approach to constructing ensemble of local experts

Conventional machine learning methods assume that a central server holds the entire training dataset for training a global model. However, in many real-world situations, datasets are collected from individual local nodes in distributed environments. Gathering these datasets often poses challenges owing to various practical constraints such as data security and privacy issues, storage limits, and transmission costs. When data distributions differ among local nodes, local experts trained on individual local nodes exhibit poor generalization performance for new instances. In this study, we propose a training-free method for constructing an ensemble of local experts without accessing datasets. For a query instance, we apply uncertainty quantification and out-of-distribution (OOD) detection to calculate the uncertainty and OOD scores of the predictions from individual models. We then determine the weights of the models by assigning a lower weight to those with higher uncertainty and OOD scores. Using these weights, the predictions of the individual models are aggregated to obtain a final prediction. In contrast to existing methods that impose specific application requirements, the proposed method can be applied in situations in which only individual predictions from local models are available for use when making a prediction for a query instance. We evaluate the effectiveness of the proposed method using image classification benchmark datasets.

Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery

The substantial success of Vision Transformer (ViT) in computer vision tasks is largely attributed to the architecture design. This underscores the necessity of efficient architecture search for designing better ViTs automatically. As training-based architecture search methods are computationally intensive, there’s a growing interest in training-free methods that use zero-cost proxies to score ViTs. However, existing training-free approaches require expert knowledge to manually design specific zero-cost proxies. Moreover, these zero-cost proxies exhibit limitations to generalize across diverse domains. In this paper, we introduce Auto-Prox, an automatic proxy discovery framework, to address the problem. First, we build the ViT-Bench-101, which involves different ViT candidates and their actual performance on multiple datasets. Utilizing ViT-Bench-101, we can evaluate zero-cost proxies based on their score-accuracy correlation. Then, we represent zero-cost proxies with computation graphs and organize the zero-cost proxy search space with ViT statistics and primitive operations. To discover generic zero-cost proxies, we propose a joint correlation metric to evolve and mutate different zero-cost proxy candidates. We introduce an elitism-preserve strategy for search efficiency to achieve a better trade-off between exploitation and exploration. Based on the discovered zero-cost proxy, we conduct a ViT architecture search in a training-free manner. Extensive experiments demonstrate that our method generalizes well to different datasets and achieves state-of-the-art results both in ranking correlation and final accuracy. Codes can be found at https://github.com/lilujunai/Auto-Prox-AAAI24.

Lightweight multi-objective evolutionary neural architecture search with low-cost proxy metrics

Multi-Objective Evolutionary Neural Architecture Search (MOENAS) methods employ evolutionary algorithms to approximate a set of architectures representing optimal trade-offs between network performance and complexity. Directly estimating network performance via error rates or losses incurs long runtimes due to the computationally expensive network training procedure. Instead, low-cost metrics that require no network training have been proposed as a proxy for network performance. However, these metrics might exhibit inconsistent correlations with network performance across different search spaces. The influences of training-based and training-free metrics on the effectiveness and efficiency of MOENAS are still under-explored.We introduce the Enhanced Training-Free MOENAS (E-TF-MOENAS) that employs the widely-used NSGA-II as the search algorithm and optimizes multiple training-free performance metrics as separate objectives. Experiments on NAS-Bench-101 and NAS-Bench-201 show that E-TF-MOENAS outperforms training-free methods that use a single training-free performance metric and could obtain comparable results to training-based methods but with approximately 30 times less computation cost. E-TF-MOENAS obtains architectures in NAS-Bench-201 with state-of-the-art mean accuracies of 94.37%, 73.50%, and 46.62% for CIFAR-10, CIFAR-100, and ImageNet16-120, respectively, within less than 3 GPU hours. It is beneficial to utilize multiple training-free proxy metrics simultaneously and E-TF-MOENAS provides a convenient framework for building such an efficient NAS approach. The source code can be found at https://github.com/ELO-Lab/E-TF-MOENAS.

Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search

Zero-shot Neural Architecture Search has garnered attention due to its training-free nature and rapid search speed. However, existing zero-shot estimators commonly suffer from low consistency, which hampers their practicality. In this work, we theoretically analyze that network generalization and convergence are highly correlated with Sweet Gradient of Parameter, i.e., the number of parameters whose gradient absolute values are within a certain interval. Empirical results indicate that Sweet Gradient of Parameter brings a higher consistency than the overall number of parameters. Additionally, we demonstrate a positive correlation between the network depth and the proportion of parameters with sweet gradients in each layer. Based on the analysis, we propose a training-free method to find the Sweet Gradient interval and obtain an estimator, named Sweetimator. Furthermore, Sweet Gradient can be an effective and general approach to promote the consistency of zero-shot estimators. Experiments show that Sweetimator and Sweet-enhanced estimators have significant consistency improvement in multiple benchmarks. Our method achieves state-of-the-art performance with 256x speedup in NAS-Bench-201 and maintains high competitiveness in DARTS, MobileNet, and Transformer search spaces. The source code is available at https://github.com/xingxing-123/SweetGradient.

Performance Analysis of Six Semi-Automated Tumour Delineation Methods on [18F] Fluorodeoxyglucose Positron Emission Tomography/Computed Tomography (FDG PET/CT) in Patients with Head and Neck Cancer.

Background. Head and neck cancer (HNC) is the seventh most common neoplastic disorder at the global level. Contouring HNC lesions on [18F] Fluorodeoxyglucose positron emission tomography/computed tomography (FDG PET/CT) scans plays a fundamental role for diagnosis, risk assessment, radiotherapy planning and post-treatment evaluation. However, manual contouring is a lengthy and tedious procedure which requires significant effort from the clinician. Methods. We evaluated the performance of six hand-crafted, training-free methods (four threshold-based, two algorithm-based) for the semi-automated delineation of HNC lesions on FDG PET/CT. This study was carried out on a single-centre population of n=103 subjects, and the standard of reference was manual segmentation generated by nuclear medicine specialists. Figures of merit were the Sørensen-Dice coefficient (DSC) and relative volume difference (RVD). Results. Median DSC ranged between 0.595 and 0.792, median RVD between -22.0% and 87.4%. Click and draw and Nestle's methods achieved the best segmentation accuracy (median DSC, respectively, 0.792 ± 0.178 and 0.762 ± 0.107; median RVD, respectively, -21.6% ± 1270.8% and -32.7% ± 40.0%) and outperformed the other methods by a significant margin. Nestle's method also resulted in a lower dispersion of the data, hence showing stronger inter-patient stability. The accuracy of the two best methods was in agreement with the most recent state-of-the art results. Conclusions. Semi-automated PET delineation methods show potential to assist clinicians in the segmentation of HNC lesions on FDG PET/CT images, although manual refinement may sometimes be needed to obtain clinically acceptable ROIs.

Zero-Cost Operation Scoring in Differentiable Architecture Search

We formalize and analyze a fundamental component of dif- ferentiable neural architecture search (NAS): local “opera- tion scoring” at each operation choice. We view existing operation scoring functions as inexact proxies for accuracy, and we find that they perform poorly when analyzed empir- ically on NAS benchmarks. From this perspective, we intro- duce a novel perturbation-based zero-cost operation scor- ing (Zero-Cost-PT) approach, which utilizes zero-cost prox- ies that were recently studied in multi-trial NAS but de- grade significantly on larger search spaces, typical for dif- ferentiable NAS. We conduct a thorough empirical evalu- ation on a number of NAS benchmarks and large search spaces, from NAS-Bench-201, NAS-Bench-1Shot1, NAS- Bench-Macro, to DARTS-like and MobileNet-like spaces, showing significant improvements in both search time and accuracy. On the ImageNet classification task on the DARTS search space, our approach improved accuracy compared to the best current training-free methods (TE-NAS) while be- ing over 10× faster (total searching time 25 minutes on a single GPU), and observed significantly better transferabil- ity on architectures searched on the CIFAR-10 dataset with an accuracy increase of 1.8 pp. Our code is available at: https://github.com/zerocostptnas/zerocost operation score.

A Training-Free Infant Spontaneous Movement Assessment Method for Cerebral Palsy Prediction Based on Videos

Early diagnosis of infant cerebral palsy (CP) is very important for infant health. In this paper, we present a novel training-free method to quantify infant spontaneous movements for predicting CP. Unlike other classification methods, our method turns the assessment into a clustering task. First, the joints of the infant are extracted by the current pose estimation algorithm, and the skeleton sequence is segmented into multiple clips through a sliding window. Then we cluster the clips and quantify infant CP by the number of cluster classes. The proposed method was tested on two datasets, and achieved state-of-the-arts (SOTAs) on both datasets using the same parameters. What's more, our method is interpretable with visualized results. The proposed method can quantify abnormal brain development in infants effectively and be used in different datasets without training. Limited by small samples, we propose a training-free method for quantifying infant spontaneous movements. Unlike other binary classification methods, our work not only enables continuous quantification of infant brain development, but also provides interpretable conclusions by visualizing the results. The proposed spontaneous movement assessment method significantly advances SOTAs in automatically measuring infant health.

A Generalized Zero-Shot Learning Scheme for SSVEP-Based BCI System.

The steady-state visual evoked potential (SSVEP) has been widely used in building multi-target brain-computer interfaces (BCIs) based on electroencephalogram (EEG). However, methods for high-accuracy SSVEP systems require training data for each target, which needs significant calibration time. This study aimed to use the data of only part of the targets for training while achieving high classification accuracy on all targets. In this work, we proposed a generalized zero-shot learning (GZSL) scheme for SSVEP classification. We divided the target classes into seen and unseen classes and trained the classifier only using the seen classes. During the test time, the search space contained both seen classes and unseen classes. In the proposed scheme, the EEG data and the sine waves are embedded into the same latent space using convolutional neural networks (CNN). We use the correlation coefficient of the two outputs in the latent space for classification. Our method was tested on two public datasets and reached 89.9% of the classification accuracy of the state-of-the-art (SOTA) data-driven method, which needs the training data of all targets. Compared to the SOTA training-free method, our method achieved a multifold improvement. This work shows that it is promising to build an SSVEP classification system that does not need the training data of all targets.

Novel Signal-to-Signal translation method based on StarGAN to generate artificial EEG for SSVEP-based brain-computer interfaces

Generative adversarial networks (GANs) have shown promising performance in image-to-image translation. Inspired by StarGAN v2, which was introduced to address the multidomain image-to-image translation, the authors propose a novel multidomain signal-to-signal translation method to generate artificial steady-state visual evoked potential (SSVEP) signals from resting electroencephalograms (EEGs). The proposed StarGAN-based signal-to-signal translation model was trained using EEG data acquired from three subjects and could successfully generate sufficient numbers of artificial SSVEP signals of 15 test participants using resting EEG data acquired over an extremely short time period (∼16 s) from each test participant. The possibility of improving the performance of SSVEP-based brain–computer interfaces (BCIs) was investigated by incorporating the artificially generated SSVEP signals and Combined-ECCA, which is proposed in this study as an extended version of combined canonical correlation analysis (Combined-CCA). Fifteen healthy individuals participated in the SSVEP-based BCI study to distinguish four visual stimuli with different flickering frequencies. To evaluate the degree of performance improvement, the average classification accuracy and information transfer rate (ITR) obtained using the proposed methods were compared with those obtained using filter bank CCA (FBCCA), a state-of-the-art training-free method for SSVEP-based BCIs. The performances in terms of the classification accuracy and ITR of SSVEP-based BCIs were significantly improved by using the proposed methods (95.47% and 40.09 bit/min) compared with those using FBCCA (92.03% and 31.80 bit/min, p < 0.05). In addition, more than four individual training trials per class were necessary to achieve better classification accuracy than that of the proposed approach if the conventional individual training approach was employed.

Bayesian neural architecture search using a training-free performance metric

Recurrent neural networks (RNNs) are a powerful approach for time series prediction. However, their performance is strongly affected by their architecture and hyperparameter settings. The architecture optimization of RNNs is a time-consuming task, where the search space is typically a mixture of real, integer and categorical values. To allow for shrinking and expanding the size of the network, the representation of architectures often has a variable length. In this paper, we propose to tackle the architecture optimization problem with a variant of the Bayesian Optimization (BO) algorithm. To reduce the evaluation time of candidate architectures the Mean Absolute Error Random Sampling (MRS), a training-free method to estimate the network performance, is adopted as the objective function for BO. Also, we propose three fixed-length encoding schemes to cope with the variable-length architecture representation. The result is a new perspective on accurate and efficient design of RNNs, that we validate on three problems. Our findings show that (1) the BO algorithm can explore different network architectures using the proposed encoding schemes and successfully designs well-performing architectures, and (2) the optimization time is significantly reduced by using MRS, without compromising the performance as compared to the architectures obtained from the actual training procedure.

Training-free measures based on algorithmic probability identify high nucleosome occupancy in DNA sequences.

We introduce and study a set of training-free methods of an information-theoretic and algorithmic complexity nature that we apply to DNA sequences to identify their potential to identify nucleosomal binding sites. We test the measures on well-studied genomic sequences of different sizes drawn from different sources. The measures reveal the known in vivo versus in vitro predictive discrepancies and uncover their potential to pinpoint high and low nucleosome occupancy. We explore different possible signals within and beyond the nucleosome length and find that the complexity indices are informative of nucleosome occupancy. We found that, while it is clear that the gold standard Kaplan model is driven by GC content (by design) and by k-mer training; for high occupancy, entropy and complexity-based scores are also informative and can complement the Kaplan model.

Autonomous Wireless Systems With Artificial Intelligence: A Knowledge Management Perspective

This article discusses technology and opportunities available to embrace artificial intelligence (AI) in the design of autonomous wireless systems. A vision is presented for knowledge-driven wireless operation by means of AI disciplines, such as sensing, reasoning, knowledge management, and active learning. The aim is to provide readers with the motivation and a general big data-independent AI methodology for autonomous agents in the context of self-organization in real time by unifying knowledge management with sensing, reasoning, and active learning. Differences between training-based methods for matching problems and training-free methods for environment-specific problems are highlighted. Finally, we conceptually introduce the functions of an autonomous agent with knowledge management.

A novel training-free method for real-time prediction of femoral strain

Surrogate methods for rapid calculation of femoral strain are limited by the scope of the training data. We compared a newly developed training-free method based on the superposition principle (Superposition Principle Method, SPM) and popular surrogate methods for calculating femoral strain during activity. Finite-element calculations of femoral strain, muscle, and joint forces for five different activity types were obtained previously. Multi-linear regression, multivariate adaptive regression splines, and Gaussian process were trained for 50, 100, 200, and 300 random samples generated using Latin Hypercube (LH) and Design of Experiment (DOE) sampling. The SPM method used weighted linear combinations of 173 activity-independent finite-element analyses accounting for each muscle and hip contact force. Across the surrogate methods, we found that 200 DOE samples consistently provided low error (RMSE < 100 µε), with model construction time ranging from 3.8 to 63.3 h and prediction time ranging from 6 to 1236 s per activity. The SPM method provided the lowest error (RMSE = 40 µε), the fastest model construction time (3.2 h) and the second fastest prediction time per activity (36 s) after Multi-linear Regression (6 s). The SPM method will enable large numerical studies of femoral strain and will narrow the gap between bone strain prediction and real-time clinical applications.

Fully automatic robust adaptive beamforming using the constant modulus feature

The diagonal loading (DL) technique is the most widely used method to improve the robustness of the Capon beamformer in the presence of imprecise knowledge of the covariance matrix and the desired signal's steering vector. The selection of the DL level is challenging in practice and might depend on some user-defined parameters which are possibly hard to be determined. A fully automatic and training-free method for the DL level selection is herein presented to extract the desired signal with constant modulus, which is a common feature for communication signals. Simulated results of the beamforming performance have demonstrated the efficacy of the proposed method.

Training-free atomistic prediction of nucleosome occupancy.

Nucleosomes alter gene expression by preventing transcription factors from occupying binding sites along DNA. DNA methylation can affect nucleosome positioning and so alter gene expression epigenetically (without changing DNA sequence). Conventional methods to predict nucleosome occupancy are trained on observed DNA sequence patterns or known DNA oligonucleotide structures. They are statistical and lack the physics needed to predict subtle epigenetic changes due to DNA methylation. The training-free method presented here uses physical principles and state-of-the-art all-atom force fields to predict both nucleosome occupancy along genomic sequences as well as binding to known positioning sequences. Our method calculates the energy of both nucleosomal and linear DNA of the given sequence. Based on the DNA deformation energy, we accurately predict the in vitro occupancy profile observed experimentally for a 20,000-bp genomic region as well as the experimental locations of nucleosomes along 13 well-established positioning sequence elements. DNA with all C bases methylated at the 5 position shows less variation of nucleosome binding: Strong binding is weakened and weak binding is strengthened compared with normal DNA. Methylation also alters the preference of nucleosomes for some positioning sequences but not others.