Training Scheme Research Articles

A low cost neuromorphic learning engine based on a high performance supervised SNN learning algorithm

Spiking neural networks (SNNs) are more energy- and resource-efficient than artificial neural networks (ANNs). However, supervised SNN learning is a challenging task due to non-differentiability of spikes and computation of complex terms. Moreover, the design of SNN learning engines is not an easy task due to limited hardware resources and tight energy constraints. In this article, a novel hardware-efficient SNN back-propagation scheme that offers fast convergence is proposed. The learning scheme does not require any complex operation such as error normalization and weight-threshold balancing, and can achieve an accuracy of around 97.5% on MNIST dataset using only 158,800 synapses. The multiplier-less inference engine trained using the proposed hard sigmoid SNN training (HaSiST) scheme can operate at a frequency of 135 MHz and consumes only 1.03 slice registers per synapse, 2.8 slice look-up tables, and can infer about 0.03times {varvec{10}}^{varvec{9}} features in a second, equivalent to 9.44 giga synaptic operations per second (GSOPS). The article also presents a high-speed, cost-efficient SNN training engine that consumes only 2.63 slice registers per synapse, 37.84 slice look-up tables per synapse, and can operate at a maximum computational frequency of around 50 MHz on a Virtex 6 FPGA.

Hierarchical Trajectory Planning for Narrow-Space Automated Parking with Deep Reinforcement Learning: A Federated Learning Scheme.

Collision-free trajectory planning in narrow spaces has become one of the most challenging tasks in automated parking scenarios. Previous optimization-based approaches can generate accurate parking trajectories, but these methods cannot compute feasible solutions with extremely complex constraints in a limited time. Recent research uses neural-network-based approaches that can generate time-optimized parking trajectories in linear time. However, the generalization of these neural network models in different parking scenarios has not been considered thoroughly and the risk of privacy compromise exists in the case of centralized training. To address the above issues, this paper proposes a hierarchical trajectory planning method with deep reinforcement learning in the federated learning scheme (HALOES) to rapidly and accurately generate collision-free automated parking trajectories in multiple narrow spaces. HALOES is a federated learning based hierarchical trajectory planning method to fully exert high-level deep reinforcement learning and the low-level optimization-based approach. HALOES further fuse the deep reinforcement learning model parameters to improve the generalization capabilities with a decentralized training scheme. The federated learning scheme in HALOES aims to protect the privacy of the vehicle's data during model parameter aggregation. Simulation results show that the proposed method can achieve efficient automatic parking in multiple narrow spaces, improve planning time from 12.15% to 66.02% compared to other state-of-the-art methods (e.g., hybrid A*, OBCA) and maintain the same level of trajectory accuracy while having great model generalization.

End-to-end weakly-supervised single-stage multiple 3D hand mesh reconstruction from a single RGB image

In this paper, we consider the challenging task of simultaneously locating and recovering multiple hands from a single 2D image. Previous studies either focus on single hand reconstruction or solve this problem in a multi-stage way. Moreover, the conventional two-stage pipeline firstly detects hand areas, and then estimates 3D hand pose from each cropped patch. To reduce the computational redundancy in preprocessing and feature extraction, for the first time, we propose a concise but efficient single-stage pipeline for multi-hand reconstruction. Specifically, we design a multi-head auto-encoder structure, where each head network shares the same feature map and outputs the hand center, pose and texture, respectively. Besides, we adopt a weakly-supervised scheme to alleviate the burden of expensive 3D real-world data annotations. To this end, we propose a series of losses optimized by a stage-wise training scheme, where a multi-hand dataset with 2D annotations is generated based on the publicly available single hand datasets. In order to further improve the accuracy of the weakly supervised model, we adopt several feature consistency constraints in both single and multiple hand settings. Specifically, the keypoints of each hand estimated from local features should be consistent with the re-projected points predicted from global features. Extensive experiments on public benchmarks including FreiHAND, HO3D, InterHand2.6M and RHD demonstrate that our method outperforms the state-of-the-art model-based methods in both weakly-supervised and fully-supervised manners. The code and models are available at https://github.com/zijinxuxu/SMHR.

An adaptive link training based hybrid circuit topology for full‐duplex on‐chip interconnects

SummaryThis work proposes a link training based half‐rate dynamic current‐steering echo‐cancellation hybrid circuit topology for full‐duplex signaling that performs echo cancellation by deploying a bit error detection unit and backchannel adaptation. In addition, this work proposes a power‐efficient half‐rate dynamic current‐steering logic hybrid circuit for full‐duplex signaling over on‐chip interconnects for simultaneous transmission and reception. The half‐rate dynamic current‐steering hybrid circuit topology has low static power consumption compared with traditional current‐mode circuit topology implementations, thanks to the discrete nature of the current‐steering hybrid topology. The proposed scheme is implemented in 1.2 V, 65‐nm complementary metal‐oxide semiconductor (CMOS). The performance results show that the link training scheme sets the control bits for both the transmitter driver and hybrid circuit topology when the backchannel adaptation loop gets locked for 1‐ and 3‐mm interconnects for process, voltage, and temperature (PVT) variations. The proposed hybrid circuit topology is able to receive the required far‐end transmitted data with the link training scheme and has the differential recovered received signal voltage swing of 1.2 V for both 1‐ and 3‐mm interconnect at the 10‐Gb/s data rate. The total power consumption of the hybrid is 3 and 3.5 mW for 1‐ and 3‐mm interconnect with an energy efficiency of 0.6 and 0.7 pJ/bit, respectively, at 10‐Gb/s full‐duplex operation.

A deep supervised cross-attention strategy for ischemic stroke segmentation in MRI studies

The key component of stroke diagnosis is the localization and delineation of brain lesions, especially from MRI studies. Nonetheless, this manual delineation is time-consuming and biased by expert opinion. The main purpose of this study is to introduce an autoencoder architecture that effectively integrates cross-attention mechanisms, together with hierarchical deep supervision to delineate lesions under scenarios of remarked unbalance tissue classes, challenging geometry of the shape, and a variable textural representation. This work introduces a cross-attention deep autoencoder that focuses on the lesion shape through a set of convolutional saliency maps, forcing skip connections to preserve the morphology of affected tissue. Moreover, a deep supervision training scheme was herein adapted to induce the learning of hierarchical lesion details. Besides, a special weighted loss function remarks lesion tissue, alleviating the negative impact of class imbalance. The proposed approach was validated on the public ISLES2017 dataset outperforming state-of-the-art results, achieving a dice score of 0.36 and a precision of 0.42. Deeply supervised cross-attention autoencoders, trained to pay more attention to lesion tissue, are better at estimating ischemic lesions in MRI studies. The best architectural configuration was achieved by integrating ADC, TTP and Tmax sequences. The contribution of deeply supervised cross-attention autoencoders allows better support the discrimination between healthy and lesion regions, which in consequence results in favorable prognosis and follow-up of patients.

Few-shot classification with task-adaptive semantic feature learning

Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much information as possible from limited samples. This can be achieved by incorporating data from multiple modalities. However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of the support samples. It is utilized to predict semantic features for the query samples. Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query data, which significantly improves model generalization. We conduct extensive experiments on four benchmarks in both inductive and transductive settings. Results show that the proposed TasNet outperforms state-of-the-art methods with an improvement of 1% to 5% in classification accuracy, demonstrating the superiority of our method. The exhaustive ablation studies further validate the effectiveness of our framework. The code is available at: https://github.com/pmhDL/TasNet

Research on Minimum Non-Collision Distance and Protection Strategy for Normal Train to Avoid Rear-End Accidents With Braking-Fault Train

Generally, a train adopts the method of emergency stop to avoid risks if there is any fault in the railway system. However, for braking system faults in a train, the relevant documents and specifications don’t provide a theoretical basis or solutions to ensure the parking safety of the breaking-fault train. For the tracking scenario, when a braking-fault train tracks the normal train running ahead, a calculation model of the tracking operation between trains with a failure coefficient is proposed. By detecting the braking-fault train information and the operation parameters of the train ahead, the model can calculate the minimum non-collision distance or predict the collision speed and position of these two trains. Considering the Yumen station of the Lanzhou-Xinjiang high-speed railway as an example, a series of experimental analyses are conducted under multiple fault scenarios, and the operation schemes that can avoid collision accidents are investigated. The experimental results show that the actual distance between the faulty and the front trains and the maximum acceleration that the front train could achieve are the two most critical factors. Finally, based on the information obtained from the above model results, the process of calculating the anti-collision scheme of the front train is summarized, and a protection strategy is designed, which provides the calculation basis and overall execution framework for the dispatching system to ensure the safety of these two trains.

Adaptive Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization.

Weakly-supervised temporal action localization (W-TAL) aims to classify and localize all action instances in untrimmed videos under only video-level supervision. Without frame-level annotations, it is challenging for W-TAL methods to clearly distinguish actions and background, which severely degrades the action boundary localization and action proposal scoring. In this paper, we present an adaptive two-stream consensus network (A-TSCN) to address this problem. Our A-TSCN features an iterative refinement training scheme: a frame-level pseudo ground truth is generated and iteratively updated from a late-fusion activation sequence, and used to provide frame-level supervision for improved model training. Besides, we introduce an adaptive attention normalization loss, which adaptively selects action and background snippets according to video attention distribution. By differentiating the attention values of the selected action snippets and background snippets, it forces the predicted attention to act as a binary selection and promotes the precise localization of action boundaries. Furthermore, we propose a video-level and a snippet-level uncertainty estimator, and they can mitigate the adverse effect caused by learning from noisy pseudo ground truth. Experiments conducted on the THUMOS14, ActivityNet v1.2, ActivityNet v1.3, and HACS datasets show that our A-TSCN outperforms current state-of-the-art methods, and even achieves comparable performance with several fully-supervised methods.

Regression Transformer enables concurrent sequence regression and generation for molecular language modelling

Despite tremendous progress of generative models in the natural sciences, their controllability remains challenging. One fundamentally missing aspect of molecular or protein generative models is an inductive bias that can reflect continuous properties of interest. To that end, we propose the Regression Transformer (RT), a method that abstracts regression as a conditional sequence modelling problem. This introduces a new direction for multitask language models, seamlessly bridging sequence regression and conditional sequence generation. We demonstrate that, despite using a nominal-scale training objective, the RT matches or surpasses the performance of conventional regression models in property prediction of small molecules, proteins and chemical reactions. Critically, priming the same model with continuous properties yields a competitive conditional generative model that outperforms specialized approaches in a substructure-constrained, property-driven molecule generation benchmark. Our dichotomous approach is facilitated by an alternating training scheme that enables the model to decorate seed sequences on the basis of desired property constraints, for example, to optimize reaction yield. We expect that the RT’s capability to jointly tackle predictive and generative tasks in biochemistry can find applications in property-driven, local exploration of the chemical or protein space. Such multitask approaches will pave the road towards foundation models in materials design.

Specific Emitter Identification via Contrastive Learning

Specific emitter identification (SEI) methods via deep learning have shown significant progress in accuracy recently. However, these methods require a large amount of the labeled data. In this letter, contrastive learning is introduced to cope with the problem of the lack of the labeled data, which the network modules include the feature extractor, the projection head, and the classifier. We use a two-stage semi-supervised training scheme. In the first stage, the feature extractor extracts the features of the received signal, and the projection head deepens the network to reserve more information in the features. They are trained with the unlabeled samples via the self-supervised contrastive learning loss. In the second stage, the overall network is fine-tuned over a small amount of the labeled data using an alternative loss. This loss combines the original cross-entropy loss with the supervised contrastive learning loss. Numerical results on the FIT/CorteXlab dataset show that even with tens of the labeled samples, the accuracy of the proposed identifier attains around 90%, which outperforms the conventional supervised and semi-supervised identifiers.

Adaptive Cooperative Control Strategy for a Wrist Exoskeleton Using Model-Based Joint Impedance Estimation

Wrist rehabilitation exoskeletons have gained much attention over the last decades, striving to restore motor functions for patients with neuromuscular disorders. Electromyography signal has been employed to estimate the motion intention to achieve interactive training schemes. However, it is a challenging task to estimate the joint impedance in real time, as it is a crucial parameter for control of exoskeletons. This article proposes an adaptive cooperative control strategy for a wrist exoskeleton based on a real-time joint impedance estimation approach. By explicitly interpreting the underlying transformation in the muscular and skeletal systems, the proposed approach estimates the motion intention and the joint impedance of a human subject simultaneously without additional calibration procedures and regulates the training trajectories and assistance accordingly. Results indicate the proposed method outperforms other training protocols, including the trajectory tracking control and the fixed cooperative control. The proposed control strategy provides an additional 66.25% motion deviation when estimated joint torque increases 12.36%, which enhances the training effectiveness and the interaction safety and promotes subjects' active engagement.

Strengthening the competitiveness of the lentil seed system of Nepal: A value chain approach

A value chain approach is the most viable and pragmatic strategy to develop an effective, inclusive, competitive and sustainable seed system for many crops globally. This study was conducted in the Kailali, Dang and Siraha districts of the Terai and Inner Terai regions of Nepal to map and thoroughly analyze the value chain of the lentil seed system. Primary data were collected through a household survey, key stakeholder interviews and focus group discussions. Altogether, 300 lentil grain-producing and 100 seed-producing farmers were selected for the study. Descriptive statistics, value chain mapping, stakeholder price spread calculation and return on investment were the major tools for the study. Conventional cultivation practices along with incidence of biotic and abiotic stresses ensured non-significant differences in yield, income and benefit-cost ratio (B:C) between local seed users and improved seed users, resulting in a distorted demand pull chain and eventually leading to the lamentable lentil seed system. Total value addition from farm gate to end user was 63.84% of the total retail price. Seed conditioners and processors contributed 28.38% of total value addition. Return on Investment (ROI) per total cost was higher for seed-producing farmers (59.3%). However, ROI per added cost was higher (214.8%) for retailers followed by seed conditioners and processors (96.37%). The total price spread along the chain was 45.21% of the total consumer price, revealing the inefficient marketing system. Scaling up the major factors behind improved seed adoption, like package of practices, use of a specialized production area, the development of stress resilient varieties, technical assistance, training, and subsidy schemes based on production and efficient market system development, can economically benefit lentil-producing farmers, thereby reinforcing the major pulling force of the lentil seed value chain. Appropriate infrastructure, government prioritization, a coordinated policy environment and innovative public-private partnership models across the seed value chain are necessary for overall sectoral growth.

Self-paced ARIMA for robust time series prediction

For time series prediction tasks, the autoregressive integrated moving average (ARIMA) model is one of the most classical and popular linear models, and extended applications have achieved satisfying prediction accuracy in many domains. However, less work has focused on the influence of noisy data, which causes the instability of ARIMA and a sharp decline in performance. In this paper, we propose a robust time series prediction framework named spARIMA. To reduce noise interference in the training process, we design a sequential training scheme in batches based on the degree of noise and the contribution to the correct modeling. spARIMA relies on the differential prediction model in ARIMA and absorbs the advantages of the gradual training scheme in self-paced learning (SPL) so that spARIMA can effectively address the instability caused by noise. Moreover, considering that the similarity of selected samples may generate local optimal solutions, we extend spARIMA to diversity selection, which utilizes a weighted local complexity–similarity distance expression to represent the diversity of noisy data. We test the performance of spARIMA on twelve datasets compared with existing ARIMA models on two gradient descent algorithms. The results show that our method is not only suitable for noisy data but also efficient for normal data, which indicates the generalization ability of spARIMA.

Self-supervised speech denoising using only noisy audio signals

In traditional speech denoising tasks, clean audio signals are often used as the training target, but absolutely clean signals are collected from expensive recording equipment or in studios with the strict environments. To overcome this drawback, we propose an end-to-end self-supervised speech denoising training scheme using only noisy audio signals, named Only-Noisy Training (ONT), without extra training conditions. The proposed ONT strategy constructs training pairs only from each single noisy audio, and it contains two modules: training audio pairs generated module and speech denoising module. The first module adopts a random audio sub-sampler on each noisy audio to generate training pairs. The sub-sampled pairs are then fed into a novel complex-valued speech denoising module. Experimental results show that the proposed method not only eliminates the high dependence on clean targets of traditional audio denoising tasks, but also achieves on-par or better performance than other training strategies. Availability—ONT is available at https://github.com/liqingchunnnn/Only-Noisy-Training

Comparative study of deep learning models on the images of biopsy specimens for diagnosis of lung cancer treatment

ObjectiveTo compare the application value of different traditional deep learning models in diagnosing and classifying lung cancer. MethodsAccording to the biopsy samples of our hospital from January 2018 to November 2022, 37 patients treated in this department were selected as the study subjects. Nonsmall and small cell lung cancer biopsy specimens were obtained and stained. Two experienced pathologists diagnosed the biopsy specimens. Multiple in-depth learning methods were used to distinguish between cancer and noncancer biopsies. In this study, we compared the application value of traditional deep learning models in lung cancer diagnosis and Classification. ResultsThe study tested several popular CNN architectures based on image block classification: AlexNet, VGG, ResNet, and SqueezeNet, comparing two types of training schemes: training from scratch and fine-tuning the entire pretrained network. The AUC of the deep learning model is more reasonable (0. 8808–0. 9121). Except for Resnet-50, the AUC of the training is higher than that of the fine-tuning of the whole network. ConclusionDeep learning analysis can accelerate the detection speed of whole section images (WSI) and maintain a similar detection rate with pathologists.

Energy skills as a driver to sustainable construction: current status, needs and proposed actions

During the last two decades, intensified actions toward energy efficiency in construction have provoked the implementation of a significant number of EU-funded and national projects that have developed knowledge in the field of energy-efficient (EE) design and construction of new and refurbished buildings. The current study contributes to state-of-the-art research of the best practices in the field of energy skills at the EU level. The methodology used comprised of desk research on the status of the development of energy skills in construction and business, the status of demand, the level of awareness, and the current training schemes and certifications which are offered. The research is based on published reports, publications, and surveys, resulting in the identification of over 500 sources of information relevant to EE skills. Key findings address the challenges related to the transition of EE skills from the national to the inter-regional level, due to methodological differences, language barriers, and organizational issues. Special attention is given to identifying existing practices for the recognition of previous learning (RPL), and to mitigate barriers to further implementation of this method for upskilling and certification of professionals and workers. In addition, we provide a base that shall allow us to evaluate the impact of existing initiatives on EE and digital skills management in the construction sectors through the survey conducted on an inter-regional level, including five different European countries. The survey results show regional divergence in the adoption of energy skills, but also concentrate on more general conclusions that will pave the way for the creation of upcoming training schemes for upskilling of the construction workforce involving both blue collars and white collars, and noting digital skills, cross-craft competences, and skills in renewable sources as the most in-demand to ensure sustainable construction

Privacy-enhanced and non-interactive linear regression with dropout-resilience

Linear regression, the most basic and widely used machine learning algorithm, has played an essential role in many areas such as healthcare, economics, and weather prediction. However, in practice, regression training requires a large dataset, usually stored in a distributed form from multiple data owners. Moreover, it is difficult to federate data among users to train a global model because of privacy issues and communication limitations. To address these challenges, we propose a non-interactive privacy-enhanced training scheme for linear regression based on functional encryption - FELR. In particular, based on the secure aggregation scheme and Paillier inner-product functional encryption, two cloud servers can train the model on the ciphertext. The solution does not require any interaction between the cloud servers and data owners. Throughout the training period, the cloud servers do not know any local private data. A detailed security analysis demonstrates that FELR can provide high security for data owners. The experimental conclusions also demonstrate that the performance of the training scheme on the data owners' side is approximately 25% higher than that of the existing solution, which significantly decreases the computational costs.

Adaptive Image Processing: First Order PDE Constraint Regularizers and a Bilevel Training Scheme

A bilevel training scheme is used to introduce a novel class of regularizers, providing a unified approach to standard regularizers TGV^2 and NsTGV^2. Optimal parameters and regularizers are identified, and the existence of a solution for any given set of training imaging data is proved by Gamma -convergence under a conditional uniform bound on the trace constant of the operators and a finite-null-space condition. Some first examples and numerical results are given.

A Study of Handgrip Force Prediction Scheme Based on Electrical Impedance Myography

Handgrip force prediction is widely used in the rehabilitation of the arm and prosthetic control. To investigate the effects of different measurement positions and feature parameters on the results of handgrip force prediction, a model based on electrical impedance myography (EIM) and long short-term memory (LSTM) networks was proposed to compare and determine a better scheme for handgrip force prediction. We conducted the signal acquisition experiments of impedance and handgrip force on the anterior forearm muscles and brachioradialis muscle. Afterwards, three evaluation metrics were introduced to compare the prediction results of various models, and the variability between models was analyzed using paired sample t-tests. The results showed that the model of handgrip force prediction based on anterior forearm muscles exhibited better performance in predicting. The evaluation metrics of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathbf {R^{2}}$</tex-math></inline-formula> , explained variance score (EVS) and normalized mean square error (NMSE) for the model fusing the feature parameters resistance (R) and reactance (X) were 0.9023, 0.9173 and 0.0114, respectively. Therefore, the feature parameters fusing R and X are the optimal input for the handgrip force prediction model. The anterior forearm muscles are the preferred position for impedance measurement over the brachioradialis muscle. This paper validated the feasibility of EIM for handgrip force prediction and provided a new reference and implementation scheme for muscle rehabilitation training and prosthetic control.

Open-Set Fault Diagnosis via Supervised Contrastive Learning With Negative Out-of-Distribution Data Augmentation

Fault diagnosis in an open world refers to the diagnosis tasks that need to cope with previously unknown faults in the online stage. It faces a great challenge yet to be addressed—that is, the online data of unknown faults may be classified as normal samples with a high probability. In this article, we develop an effective solution for this challenge by using supervised contrastive learning to learn a discriminative and compact embedding for the known normal situation and fault situations. Specifically, in addition to contrasting a given sample with other instances as is the case in conventional contrastive learning methods, our training scheme contrasts the normal samples with negative augmentations of themselves. The negative out-of-distribution data is generated by the Soft Brownian Offset sampling method to simulate the previously unknown faults. Computational experiments are conducted on the Tennessee Eastman Process benchmark dataset and a practical plasma etching process dataset. The proposed method achieves significant improvement compared with four existing methods under three open-set fault diagnosis circumstances, i.e., balanced open-set fault diagnosis, imbalanced fault diagnosis, and few-shot fault diagnosis. This demonstrates its great potentials in real world fault diagnosis applications.