Lengthy Training Time Research Articles

TaylorNet: A novel approach for spectral filter learning on graph data

Graphs are indispensable data structures used to model interconnected nodes and their relationships. However, devising filters for graph data is challenging. Previous approaches, including BernNet, utilized Bernstein polynomials to approximate spectral graph convolutions but suffered from lengthy training times compared to alternatives. To address this, we propose TaylorNet, a novel graph neural network built upon rigorous mathematical principles and theoretical foundations. TaylorNet provides an efficient approach for learning arbitrary graph spectral filters. Our methodology revolves around utilizing Kth order Taylor polynomials to estimate filters on the normalized Laplacian operator of graphs. By carefully setting the coefficients of these polynomials, we achieve effective filter design. Additionally, TaylorNet personalizes filter learning for specific graphs by leveraging observed graph data and signals to determine these coefficients. Extensive experimentation demonstrates TaylorNet’s ability to learn arbitrary spectral filters, surpassing current state-of-the-art methods, particularly for handling large graphs. Its exceptional performance extends to real-world graph modeling tasks, highlighting its versatility and applicability. For interested researchers, the complete implementation of TaylorNet is available at https://github.com/12chen20/TaylorNet.

Dynamic robot routing optimization: State–space decomposition for operations research-informed reinforcement learning

There is a growing interest in implementing artificial intelligence for operations research in the industrial environment. While numerous classic operations research solvers ensure optimal solutions, they often struggle with real-time dynamic objectives and environments, such as dynamic routing problems, which require periodic algorithmic recalibration. To deal with dynamic environments, deep reinforcement learning has shown great potential with its capability as a self-learning and optimizing mechanism. However, the real-world applications of reinforcement learning are relatively limited due to lengthy training time and inefficiency in high-dimensional state spaces. In this study, we introduce two methods to enhance reinforcement learning for dynamic routing optimization. The first method involves transferring knowledge from classic operations research solvers to reinforcement learning during training, which accelerates exploration and reduces lengthy training time. The second method uses a state–space decomposer to transform the high-dimensional state space into a low-dimensional latent space, which allows the reinforcement learning agent to learn efficiently in the latent space. Lastly, we demonstrate the applicability of our approach in an industrial application of an automated welding process, where our approach identifies the shortest welding pathway of an industrial robotic arm to weld a set of dynamically changing target nodes, poses and sizes. The suggested method cuts computation time by 25% to 50% compared to classic routing algorithms.

Deep Reinforcement Learning for sim-to-real policy transfer of VTOL-UAVs offshore docking operations

This paper proposes a novel Reinforcement Learning (RL) approach for sim-to-real policy transfer of Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTOL-UAV). The proposed approach is designed for VTOL-UAV landing on offshore docking stations in maritime operations. VTOL-UAVs in maritime operations encounter limitations in their operational range, primarily stemming from constraints imposed by their battery capacity. The concept of autonomous landing on a charging platform presents an intriguing prospect for mitigating these limitations by facilitating battery charging and data transfer. However, current Deep Reinforcement Learning (DRL) methods exhibit drawbacks, including lengthy training times, and modest success rates. In this paper, we tackle these concerns comprehensively by decomposing the landing procedure into a sequence of more manageable but analogous tasks in terms of an approach phase and a landing phase. The proposed architecture utilizes a model-based control scheme for the approach phase, where the VTOL-UAV is approaching the offshore docking station. In the Landing phase, DRL agents were trained offline to learn the optimal policy to dock on the offshore station. The Joint North Sea Wave Project (JONSWAP) spectrum model has been employed to create a wave model for each episode, enhancing policy generalization for sim2real transfer. A set of DRL algorithms have been tested through numerical simulations including value-based agents and policy-based agents such as Deep Q Networks (DQN) and Proximal Policy Optimization (PPO) respectively. The numerical experiments show that the PPO agent can learn complicated and efficient policies to land in uncertain environments, which in turn enhances the likelihood of successful sim-to-real transfer.

Hybrid supervised and reinforcement learning for the design and optimization of nanophotonic structures.

From higher computational efficiency to enabling the discovery of novel and complex structures, deep learning has emerged as a powerful framework for the design and optimization of nanophotonic circuits and components. However, both data-driven and exploration-based machine learning strategies have limitations in their effectiveness for nanophotonic inverse design. Supervised machine learning approaches require large quantities of training data to produce high-performance models and have difficulty generalizing beyond training data given the complexity of the design space. Unsupervised and reinforcement learning-based approaches on the other hand can have very lengthy training or optimization times associated with them. Here we demonstrate a hybrid supervised learning and reinforcement learning approach to the inverse design of nanophotonic structures and show this approach can reduce training data dependence, improve the generalizability of model predictions, and significantly shorten exploratory training times. The presented strategy thus addresses several contemporary deep learning-based challenges, while opening the door for new design methodologies that leverage multiple classes of machine learning algorithms to produce more effective and practical solutions for photonic design.

Human Activity Recognition through Smartphone Inertial Sensors with ML Approach

Human Activity Recognition (HAR) has several applications in healthcare, security, and assisted living systems used in smart homes. The main aim of these applications or systems is to classify body movement read from the built in sensors such as accelerometers and gyroscopes. Some actions could be performed in response to the output of these HAR systems. The number of smartphone users increases, whereas the sensors are widely available in different sizes and shapes (internal or external sensors). Recent advances in sensor technology and machine learning have led researchers to conduct studies on sensor technology such as HAR. HAR systems typically use a combination of sensors, such as accelerometers, gyroscopes, and cameras, to collect images or signal data that can be classified by machine learning algorithms. HAR research has focused on several key challenges including dealing with variability in sensor data, handling missing data or noise, and dealing with large amounts of sensor-generated data. In this work, several machine learning algorithms were tested in predefined settings using the KU-HAR dataset in a series of experiments. Subsequently, various performance metrics were calculated to assess the chosen algorithms’ performance. The experimental findings showed that the LightGBM classifier surpassed the other machine learning algorithms in performance metrics, such as accuracy, F1 score, precision, and recall. Although Gradient Boosting has lengthy training time, the other classifiers complete their training in an acceptable time period.

A Review and Benchmark on State-of-the-Art Steel Defects Detection

Steel, a critical material in construction, automobile, and railroad manufacturing industries, often presents defects that can lead to equipment failure, significant safety risks, and costly downtime. This research aims to evaluate the performance of state-of-the-art object detection models in detecting defects on steel surfaces, a critical task in industries such as railroad and automobile manufacturing. The study addresses the challenges of limited defect data and lengthy model training times. Five existing state-of-the-art object detection models (faster R-CNN, deformable DETR, double head R-CNN, Retinanet, and deformable convolutional network) were benchmarked on the Northeastern University (NEU) steel dataset. The selection of models covers a broad spectrum of methodologies, including two-stage detectors, single-stage detectors, transformers, and a model incorporating deformable convolutions. The deformable convolutional network achieved the highest accuracy of 77.28% on the NEU dataset following a fivefold cross-validation method. Other models also demonstrated notable performance, with accuracies within the 70–75% range. Certain models exhibited particular strengths in detecting specific defects, indicating potential areas for future research and model improvement. The findings provide a comprehensive foundation for future research in steel defect detection and have significant implications for practical applications. The research could improve quality control processes in the steel industry by automating the defect detection task, leading to safer and more reliable steel products and protecting workers by removing the human factor from hazardous environments.

Fault diagnosis research of laser gyroscope based on optimized-kernel extreme learning machine

Traditional diagnostic models for laser gyroscopes, widely which are commonly employed as high-precision angular velocity sensors in aerospace applications, often encounter challenges in terms of reliability and accuracy. These challenges arise from difficulties in feature extraction, high computational costs, and lengthy training times. In light of these challenges, the present study proposes a new method for diagnosing faults in laser gyroscopes using the Kernel Extreme Learning Machine (KELM). Specifically, the proposed method utilizes Wavelet Packet Decomposition (WPD) to efficiently extract features from the laser gyroscope signal, which are then used as input for our diagnostic model. Furthermore, the KELM model is trained for fault diagnosis. Afterward, we utilize the Improved Dung Beetle Optimizer (IDBO) algorithm to optimize its parameters for improved optimization performance. According to the experimental results, our proposed IDBO-KELM model demonstrates a 3.68% improvement in diagnostic accuracy compared to traditional approaches. Additionally, it offers the advantages of shorter training time and increased precision.

Face Super-Resolution via Joint Edge Information and Attention Aggregation Network

Face super-resolution (SR) is a specialized image super-resolution problem that is particularly relevant in various intelligent transportation scenes such as video surveillance and identification systems. Deep learning-steered solutions for face images often leverage facial priors (i.e., face parsing and landmark) to restore intricate facial components and have demonstrated promising performance. However, these methods typically rely on abundant manually labeled data and require lengthy training times. In this work, we introduce a meaningful approach called the Joint Edge Information and Attention Aggregation Network (JEANet) for the face SR problem. JEANet is established on our carefully designed Face Attention Aggregation Module (FAAM), which incorporates parallel connections of both channel-wise and spatial-wise features. Specifically, we integrate an attention fusion strategy into the residual blocks and elaborately employ edge blocks to extract edge information. Additionally, we introduce adaptive shortcuts that interpolate reconstruction parts at multiple scales. With the aid of adaptive shortcuts, JEANet effectively restores detailed images textures by leveraging contextual features optimally. These considerations enable the convolutional operations to flexibly distill information related to primary facial structures and progressively incorporate edge information to assemble local and global meaningful structures. Qualitative and quantitative evaluations demonstrate that our proposed method excels in recovering highly realistic face images compared to several competitive face SR methods.

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively injest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps --- accelerating personalization from dozens of minutes to seconds , while preserving quality. Code and trained encoders will be available at our project page.

Intrusion Detection Model Based on Improved Transformer

This paper proposes an enhanced Transformer-based intrusion detection model to tackle the challenges of lengthy training time, inaccurate detection of overlapping classes, and poor performance in multi-class classification of current intrusion detection models. Specifically, the proposed model includes the following: (i) A data processing strategy that initially reduces the data dimension using a stacked auto-encoder to speed up training. In addition, a novel under-sampling method based on the KNN principle is introduced, along with the Borderline-SMOTE over-sampling method, for hybrid data sampling that balances the dataset while addressing the issue of low detection accuracy in overlapping data classes. (ii) An improved position encoding method for the Transformer model that effectively learns the dependencies between features by embedding the position information of features, resulting in better classification accuracy. (iii) A two-stage learning strategy in which the model first performs rough binary prediction (determining whether it is an illegal intrusion) and then inputs the prediction value and original features together for further multi-class prediction (predicting the intrusion category), addressing the issue of low accuracy in multi-class classification. Experimental results on the official NSL-KDD test set demonstrate that the proposed model achieves an accuracy of 88.7% and an F1-score of 88.2% in binary classification and an accuracy of 84.1% and an F1-score of 83.8% in multi-class classification. Compared to existing intrusion detection models, our model exhibits higher accuracy and F1-score and trains faster than other models.

Optimization Strategy of a Stacked Autoencoder and Deep Belief Network in a Hyperspectral Remote-Sensing Image Classification Model

Improvements in hyperspectral image technology, diversification methods, and cost reductions have increased the convenience of hyperspectral data acquisitions. However, because of their multiband and multiredundant characteristics, hyperspectral data processing is still complex. Two feature extraction algorithms, the autoencoder (AE) and restricted Boltzmann machine (RBM), were used to optimize the classification model parameters. The optimal classification model was obtained by comparing a stacked autoencoder (SAE) and a deep belief network (DBN). Finally, the SAE was further optimized by adding sparse representation constraints and GPU parallel computation to improve classification accuracy and speed. The research results show that the SAE enhanced by deep learning is superior to the traditional feature extraction algorithm. The optimal classification model based on deep learning, namely, the stacked sparse autoencoder, achieved 93.41% and 94.92% classification accuracy using two experimental datasets. The use of parallel computing increased the model’s training speed by more than seven times, solving the model’s lengthy training time limitation.

A Two-stage Learning-based method for Large-scale On-demand pickup and delivery services with soft time windows

With the rapid growth of the on-demand logistics industry, large-scale pickup and delivery with soft time windows has become widespread in various time-critical scenarios. This problem has proven to be an NP-hard problem. Hence, the computation time and resources required to solve it increase exponentially with the growth of size. As a result, it is burdensome for the exact algorithm and heuristic method to generate a high-quality solution instantly. Machine learning seems to be a possible option due to the advantage of offline training. However, it is difficult to solve large-scale problems due to the lengthy training time, heavy computational cost, and training instability. Thus, this paper proposes the two-stage learning-based method composed of the clustering stage and the routing stage to tackle this problem. The clustering stage builds upon the attention mechanism by introducing graph convolutional network to the input, which can keep the match of pickup and paired delivery customers and classify them into different vehicles, while the routing stage adopts a well-trained model to generate a route for each capacitated vehicle. Furthermore, the well-trained model is utilized to train another problem inspired by transfer learning. Experiments show that the model, trained on small-scale problems, generalizes well to larger-scale problems, and achieves superior performance compared with the heuristic method and Google OR-Tools, with an extremely short computing time. In addition, the favorable transferability of this model is verified through contrast experiment, which can save a significant amount of training time.

Multi-Level Pooling Model for Fingerprint-Based Gender Classification

It has been widely reported that CNN (Convolutional Neural Network) has shown satisfactory results in classifying images. The strength of CNN lies in the type and the number of layers that construct it. However, the most apparent drawbacks of CNN are the requirement for a large labeled dataset and its lengthy training time. Although datasets are available, labeling that data is a significant problem. This work mimics the CNN model but only utilizes its pooling layers. The novelty of this model is removing convolution layers and directly processing fingerprint images using pooling layers. Three pooling layer models, namely maximum pooling, average pooling, and minimum pooling, are used to generate fingerprint features to classify their owner gender. These pooling layers are arranged consecutively up to eight levels. Removing convolution layers makes the process straightforward, and the computation is much faster. This study utilized 200 fingerprint datasets from the NIST (National Institute of Standards and Technology), with male and female fingerprints of 100 samples each. The extracted features were then classified using K-NN (K-Nearest Neighbors) algorithm. The proposed method resulted in an accuracy of 61% to 71.5% or an average of 66.25%.

The Report of the 2021-2022 AACP Research and Graduate Affairs Committee

EXECUTIVE SUMMARY. The work of the 2021-2022 AACP Research and Graduate Affairs Committee (RGAC) focused on barriers to graduate education and research-related careers in pharmacy education. AACP President Stuart Haines charged the RGAC with identifying the critical barriers that hinder current PharmD students/recent graduates as well as under-represented groups (e.g., Black and Latino) from pursuing advanced degrees and research-related career paths in the pharmaceutical, social & behavioral, and clinical sciences and recommending changes that might address these barriers - this may include recommendations to change the fundamental structure of graduate education.The committee began its work with a literature review to survey current perspectives on these barriers and assess the supporting evidence for effective solutions and programs, including their relevance to pharmacy education. Based on the review, the committee was able to identify numerous obstacles to entry into and progression through research training, for both underrepresented learners and student pharmacists. Obstacles are individual, e.g., lack of exposure to and self-efficacy in research, financial constraints, structural, e.g., lengthy training time, programmatic rigidity, and institutional, e.g., implicit and explicit bias. The committee found evidence of effective approaches and programs to address these barriers that could be applied in pharmacy schools. These approaches include improvements to existing practices in recruitment, admissions and hiring practices as well as creation of new programs and structural changes to existing programs to increase accessibility to learners. The committee also recognized a need for more research and development of additional approaches to address these barriers.The committee makes a series of recommendations that AACP develop resource guides and programs to address key issues in the recruitment and retention of underrepresented students and student pharmacists into graduate education and research careers, including as faculty. The committee also proposes new AACP policies to support innovative graduate programs and early, longitudinal engagement of learners from elementary school onward to increase access to graduate education and to support environments and cultures of commitment to accessibility, diversity, equity, inclusiveness, antiracism in pharmacy education.

Deep Residual Network Combined with Transfer Learning Based Fault Diagnosis for Rolling Bearing

Fault diagnosis of rolling bearings is significant for mechanical equipment operation and maintenance. Presently, the deep convolutional neural network (CNN) is increasingly used for fault diagnosis of rolling bearings, but CNN has challenges with incomplete training and lengthy training times. This paper proposes a residual network combined with the transfer learning (ResNet-TL) based diagnosis method for rolling bearing, which can preprocess the one-dimensional data of vibration signals into image data. Then, the transfer learning theory in parameter transfer is applied to the training of the network model, and the ResNet34 network is pre-trained and re-trained; the image data are selected to be the inputs of the fault diagnosis model. The experimental validation of the rolling bearing fault dataset collected from the practical bench and Case Western Reserve University shows the superiority of the ResNet34-TL model compared with other classification models.

A Minimum Error-Based PCA for Improving Classifier Performance in Detecting Financial Fraud

The main challenge of data mining approaches to detect fraud in financial transaction data is the imbalance of data classes in available datasets, with a much smaller fraud class proportion than the non-fraud. This imbalance affects the f1-score to be low due to unbalanced precision and recall. Therefore, the model can predict one class well, but it does not apply to another class. In addition, the lengthy training time duration and high computational resource requirements in implementing data mining also make them a particular concern. Therefore, solely handling imbalanced data is still insufficient to produce the expected performance. Reduction of data dimensions can be a solution to increase the speed of the process. However, this method actually reduces the classifier’s performance when it comes to classification. Furthermore, this study intends to improve the performance of the data mining approach based on the Support Vector Machine (SVM) classifier aiming at detecting financial fraud transactions. The SVM performance was refined by tuning the kernel and hyperparameter integrated with the Random Under Sampling (RUS) and our Minimum error-based Principal Component Analysis (MebPCA). The RUS was used to handle imbalanced data, while MebPCA modified data dimension reduction techniques based on classification errors to speed up computational time without disturbing the performance of SVM. This combination improves the classifier's performance in detecting fraud effectively with a precision improvement of 29.31% and f1-score of 19.8%, and efficiently reduces the duration of training time significantly by 36.39% compared to previous research regarding the SVM method for fraud detection.

Alterations in facial expressions of emotion: Determining the promise of ultrathin slicing approaches and comparing human and automated coding methods in psychosis risk.

Alterations in facial expressions of emotion are a hallmark of psychopathology and may be present before the onset of mental illness. Technological advances have spurred interest in examining alterations based on "thin slices" of behavior using automated approaches. However, questions remain. First, can alterations be detected in ultrathin slices of behavior? Second, how do automated approaches converge with human coding techniques? The present study examined ultrathin (i.e., 1-min) slices of video-recorded clinical interviews of 42 individuals at clinical high risk (CHR) for psychosis and 42 matched controls. Facial expressions of emotion (e.g., joy, anger) were examined using two automated facial analysis programs and coded by trained human raters (using the Expressive Emotional Behavior Coding System). Results showed that ultrathin (i.e., 1-min) slices of behavior were sufficient to reveal alterations in facial expressions of emotion, specifically blunted joy expressions in individuals at CHR (with supplementary analyses probing links with attenuated positive symptoms and functioning). Furthermore, both automated analysis programs converged in the ability to detect blunted joy expressions and were consistent with human coding at the level of both second-by-second and aggregate data. Finally, there were areas of divergence across approaches for other emotional expressions beyond joy. These data suggest that ultrathin slices of behavior can yield clues about emotional dysfunction. Further, automated approaches (which do not require lengthy training and coder time but do lend well to mobile assessment and computational modeling) show promise, but careful evaluation of convergence with human coding is needed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

An Online Offline Framework for Anomaly Scoring and Detecting New Traffic in Network Streams

Network data constantly evolves with new network applications and protocols. There is a need for robust techniques to detect anomalous behaviour. Offline models trained with static data lose validity when new variants of traffic emerge. They require retraining but the need for ground truth and lengthy training times make this task challenging. Meanwhile, online models which detect outliers in streaming data are susceptible to the curse of dimensionality and natural variability. Today’s anomalies may be tomorrow’s new traffic and existing methods do not provide a way to differentiate between them. We propose a framework that makes the most of both approaches: an offline deep learning model extracts features of normal traffic and provides a bias for an online outlier detection model to select data for training. The online model retains its previously learnt knowledge and retrains itself with new data. Online thresholds are updated in a drifting manner and the Mann-Whitney U test is incorporated to prevent inaccurate updates. We perform analysis on the scores, develop heuristics to detect new traffic and evaluate using three deep learning models and four outlier detection methods on the UNSW-NB15 and CTU-13 datasets. The framework improves upon any individual offline or online models in isolation.

Toward Near-Real-Time Training With Semi-Random Deep Neural Networks and Tensor-Train Decomposition

In recent years, deep neural networks have shown to achieve state-of-the-art performance on several classification and prediction tasks. However, these networks demand undesirable lengthy training times coupled with high computational resources (memory, I/O, processing time). In this work, we explore semi-random deep neural networks to achieve near real-time training and less computational resource usage. Although many works enhance the underlying hardware for real-time training, this work focuses on algorithmic optimization. It is shown that random projection networks with additional skipped connectivity and randomly weighted layers can boost the overall network performance while enabling for real-time training. Additionally, a tensor-train decomposition technique is leveraged to further reduce the model complexity of these networks. Our investigation accomplishes the following: 1) Tensor-train decomposition decreases the complexity of random projection networks, 2) compression of the fully connected hidden layer leads to a minimum ~ 40× decrease in memory size, and 3) training under random projection networks can be achieved in near-real time.

A Computationally Efficient Single-Channel EEG Sleep Stage Scoring Approach using Simple Structured CNN

Automatic sleep stage classification has been a hot trend since hand-crafted feature engineering is highly inefficient. However, current studies of automatic sleep stage scoring focus more on designing complex neural network structures to improve the model performance while neglecting the model efficiency. This causes both lengthy training time and highly demanding hardware are needed for model training, which is not favorable for future industrial applications. This work proves the concept that the simple model, such as a shallow Convolutional Neural Network (CNN) combining the proper data processing techniques, can achieve a comparable model performance (overall accuracy of 79.0 %) to the complex model (overall accuracy of 74.9-82.0 %). The designed model in this work also significantly improves the model efficiency by reducing the number of learnable parameters in the neural network. This approach provides a new insight into automatic sleep stage scoring study as well as other deep learning studies that the data processing and the model design are equally important.