Supervised Training Data Research Articles

Deep‐Learning Phase‐Onset Picker for Deep Earth Seismology: PKIKP Waves

AbstractBody waves traversing the Earth's interior from a seismic source to receivers on the surface carry rich information about its internal structures. Their travel time measurements have been widely used in seismology to constrain Earth's interior at the global scale by mapping the time anomaly along their ray paths. However, picking the travel time of global seismic waves, suitable for studying Earth's fine‐scale structures, requires highly skilled personnel and is often fairly subjective. Here, we report the development of an automatic picker for PKIKP waves traveling through the inner core (IC), especially nearly along Earth's diameters, based on the latest advances in supervised deep learning. A convolutional neural network (CNN) we develop automatically determines the PKIKP onset on vertical seismograms near its theoretical prediction of cataloged earthquakes. As high‐quality manual onset picks of global seismic phases are limited, we employ a scheme to generate a synthetic supervised training data set containing 300,000 waveforms. The PKIKP onsets picked by our trained CNN automatic picker exhibit a mean absolute error of ∼0.5 s compared to 1,503 manual picks, comparable to the estimated human‐picking error. In an integration test, the automatic picks obtained from an extended waveform data set yield a cylindrically anisotropic IC model that agrees well with the models inferred from manual picks, which illustrates the success of this pilot model. This is a significant step closer to harvesting an unprecedented volume of travel time measurements for studying the IC or other regions of the Earth's deep interior.

Few-Shot Relation Extraction With Automatically Generated Prompts.

Relation extraction (RE) tends to struggle when the supervised training data is few and difficult to be collected. In this article, we elicit relational and factual knowledge from large pretrained language models (PLMs) for few-shot RE (FSRE) with prompting techniques. Concretely, we automatically generate a diverse set of natural language templates and modulate PLM's behavior through these prompts for FSRE. To mitigate the template bias which leads to unstableness of few-shot learning, we propose a simple yet effective template regularization network (TRN) to prevent deep networks from over-fitting uncertain templates and thus stabilize the FSRE models. TRN alleviates the template bias with three mechanisms: 1) an attention mechanism over mini-batch to weight each template; 2) a ranking regularization mechanism to regularize the attention weights and constrain the importance of uncertain templates; and 3) a template calibration module with two calibrating techniques to modify the uncertain templates in the lowest-ranked group. Experimental results on two benchmark datasets (i.e., FewRel and NYT) show that our model has robust superiority over strong competitors. For reproducibility, we will release our code and data upon the publication of this article.

Concept of Chatbot as an Intelligent Dialogue Assistant

The purpose of the study is to compare chatbots, assess their ability to perform tasks, analyze their impact on consumers, and determine the areas of application and risks associated with the use of intelligent dialogue assistants. The research methods used in the study include analysis and systematisation of information, synthesis and generalization of theoretical data based on a review of current research, which made it possible to assess the risks and benefits of using chatbots for businesses and ordinary users. The scientific novelty lies in the analysis of technologies for creating intelligent dialogue assistants and determining the areas of application of chatbots. A study of the areas of application of modern intelligent dialogue systems has shown the high efficiency of using chatbots to solve typical user problems. The article provides statistical data on how satisfied Internet users are with contacts with chatbots and describes recommendations for improving such dialogue systems. An analysis of the role of artificial intelligence in the operation of dialogue systems has shown possible risks from the impact of technology on the minds of users. Conclusions. One of the most obvious trends in chatbots in 2023 is that their use will become more widespread and chatbots themselves will become more sophisticated. Their advantage is free customer service and data collection, which can then be used for marketing research. Over time, chatbots will be used in such areas as marketing, recruitment, education, and medicine. Their ability to perform a wide range of tasks makes chatbots attractive for e-commerce stores, B2B companies, real estate, healthcare, and education. In the future, intelligent dialogue agents will work with huge, dynamic, heterogeneous data streams, providing powerful capabilities for adaptive and flexible interaction. Dialogue systems have become successful and reliable due to the large amount of real-world user data available to their developers. However, these systems still exhibit rather limited communication behaviour modelled on information retrieval tasks. Chatbots are developed for research purposes, but they are often limited to a narrow, manually created domain. The newest trend in conversational agent development involves neural networks and models trained on huge collections of dialogue data without detailed specifications of dialogue states. These models lack tractability and interpretability due to their black-box nature. They also require extensive supervised training data to be competitive.

Learning Domain‐specific Semantic Representation from Weakly Supervised Data to Improve Research Dataset Retrieval

AbstractAlong with the development of the data‐driven research paradigm, there are exponentially increasing datasets, which bring challenges to researchers in the efficient retrieval of relevant datasets. Previous studies mainly focused on query expansion methods based on sparse retrieval models to improve the accuracy and recall in retrieval. We investigated the use of semantically rich information to retrieve relevant datasets and the benefits of using domain‐specific dense vector representation as opposed to general representation. First, we used pairs of metadata fields that have semantic relevance to construct the domain‐specific weakly supervised training data. Then, a pre‐trained transformer‐based deep learning model is fine‐tuned on the training data using the contrastive learning method. Finally, dense vector representations of the queries and datasets are obtained based on the fine‐tuned model. The relevance of a dataset to a query is measured by the similarity between the vectors. To evaluate the performance of the proposed model, we collected 104,683 datasets from 13 research data repositories, recruited volunteers to design research‐oriented queries, and annotated the retrieval results. The experimental results show that compared with the domain‐independent fine‐tuned model, our proposed method can improve the NDCG@10 score by about 5%.

Task adapted reconstruction for inverse problems

The paper considers the problem of performing a post-processing task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and post-processing as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a differentiable parametrization of the family of estimators for both steps. These networks are combined and jointly trained against suitable supervised training data in order to minimize a joint differentiable loss function, resulting in an end-to-end task adapted reconstruction method. The suggested framework is generic, yet adaptable, with a plug-and-play structure for adjusting both the inverse problem and the post-processing task at hand. More precisely, the data model (forward operator and statistical model of the noise) associated with the inverse problem is exchangeable, e.g., by using neural network architecture given by a learned iterative method. Furthermore, any post-processing that can be encoded as a trainable neural network can be used. The approach is demonstrated on joint tomographic image reconstruction, classification and joint tomographic image reconstruction segmentation.

Large-scale protein-protein post-translational modification extraction with distant supervision and confidence calibrated BioBERT

MotivationProtein-protein interactions (PPIs) are critical to normal cellular function and are related to many disease pathways. A range of protein functions are mediated and regulated by protein interactions through post-translational modifications (PTM). However, only 4% of PPIs are annotated with PTMs in biological knowledge databases such as IntAct, mainly performed through manual curation, which is neither time- nor cost-effective. Here we aim to facilitate annotation by extracting PPIs along with their pairwise PTM from the literature by using distantly supervised training data using deep learning to aid human curation.MethodWe use the IntAct PPI database to create a distant supervised dataset annotated with interacting protein pairs, their corresponding PTM type, and associated abstracts from the PubMed database. We train an ensemble of BioBERT models—dubbed PPI-BioBERT-x10—to improve confidence calibration. We extend the use of ensemble average confidence approach with confidence variation to counteract the effects of class imbalance to extract high confidence predictions.Results and conclusionThe PPI-BioBERT-x10 model evaluated on the test set resulted in a modest F1-micro 41.3 (P =5 8.1, R = 32.1). However, by combining high confidence and low variation to identify high quality predictions, tuning the predictions for precision, we retained 19% of the test predictions with 100% precision. We evaluated PPI-BioBERT-x10 on 18 million PubMed abstracts and extracted 1.6 million (546507 unique PTM-PPI triplets) PTM-PPI predictions, and filter approx 5700 (4584 unique) high confidence predictions. Of the 5700, human evaluation on a small randomly sampled subset shows that the precision drops to 33.7% despite confidence calibration and highlights the challenges of generalisability beyond the test set even with confidence calibration. We circumvent the problem by only including predictions associated with multiple papers, improving the precision to 58.8%. In this work, we highlight the benefits and challenges of deep learning-based text mining in practice, and the need for increased emphasis on confidence calibration to facilitate human curation efforts.

Cross-Modality Transfer Learning for Image-Text Information Management

In the past decades, information from all kinds of data has been on a rapid increase. With state-of-the-art performance, machine learning algorithms have been beneficial for information management. However, insufficient supervised training data is still an adversity in many real-world applications. Therefore, transfer learning (TF) was proposed to address this issue. This article studies a not well investigated but important TL problem termed cross-modality transfer learning (CMTL). This topic is closely related to distant domain transfer learning (DDTL) and negative transfer. In general, conventional TL disciplines assume that the source domain and the target domain are in the same modality. DDTL aims to make efficient transfers even when the domains or the tasks are entirely different. As an extension of DDTL, CMTL aims to make efficient transfers between two different data modalities, such as from image to text. As the main focus of this study, we aim to improve the performance of image classification by transferring knowledge from text data. Previously, a few CMTL algorithms were proposed to deal with image classification problems. However, most existing algorithms are very task specific, and they are unstable on convergence. There are four main contributions in this study. First, we propose a novel heterogeneous CMTL algorithm, which requires only a tiny set of unlabeled target data and labeled source data with associate text tags. Second, we introduce a latent semantic information extraction method to connect the information learned from the image data and the text data. Third, the proposed method can effectively handle the information transfer across different modalities (text-image). Fourth, we examined our algorithm on a public dataset, Office-31. It has achieved up to 5% higher classification accuracy than “non-transfer” algorithms and up to 9% higher than existing CMTL algorithms.

PhotoApp

Photorealistic editing of head portraits is a challenging task as humans are very sensitive to inconsistencies in faces. We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination (parameterised with an environment map) in a portrait image. This requires our method to capture and control the full reflectance field of the person in the image. Most editing approaches rely on supervised learning using training data captured with setups such as light and camera stages. Such datasets are expensive to acquire, not readily available and do not capture all the rich variations of in-the-wild portrait images. In addition, most supervised approaches only focus on relighting, and do not allow camera viewpoint editing. Thus, they only capture and control a subset of the reflectance field. Recently, portrait editing has been demonstrated by operating in the generative model space of StyleGAN. While such approaches do not require direct supervision, there is a significant loss of quality when compared to the supervised approaches. In this paper, we present a method which learns from limited supervised training data. The training images only include people in a fixed neutral expression with eyes closed, without much hair or background variations. Each person is captured under 150 one-light-at-a-time conditions and under 8 camera poses. Instead of training directly in the image space, we design a supervised problem which learns transformations in the latent space of StyleGAN. This combines the best of supervised learning and generative adversarial modeling. We show that the StyleGAN prior allows for generalisation to different expressions, hairstyles and backgrounds. This produces high-quality photorealistic results for in-the-wild images and significantly outperforms existing methods. Our approach can edit the illumination and pose simultaneously, and runs at interactive rates.

Three-Dimensional Object Co-Localization From Mobile LiDAR Point Clouds

Recently, 3D deep learning technologies require a large amount of supervised 3D point-cloud data to learn statistical models for various ITS-related tasks, e.g. object classification, object detection, object segmentation, etc. However, manually annotating 3D point-cloud data is time-consuming and labor-intensive. Therefore, this paper aims at co-locating 3D objects from mobile LiDAR point clouds without any help of supervised training data. To realize it, we propose a new framework to implement 3D object co-localization for automatically extracting the objects of the same category from different point-cloud scenes. Specifically, to search and exploit the co-information from objects in different point-cloud scenes, we formulate a 3D object co-localization problem as a maximal subgraph matching problem. During the graph construction procedure, to handle the inconsistent representation of objects in different scenes, we propose a multi-scale clustering method to represent objects by a pyramid structure. In addition, because the maximal subgraph matching problem is NP-hard, we propose a stochastic search algorithm to generate the co-localization results. Extensive experiments on the point-cloud data collected by the Reigl VMX450 mobile LiDAR system demonstrate the promising performance of the proposed framework.

Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models

For incremental machine-learning applications it is often important to robustly estimate the system accuracy during training, especially if humans perform the supervised teaching. Cross-validation and interleaved test/train error are here the standard supervised approaches. We propose a novel semi-supervised accuracy estimation approach that clearly outperforms these two methods. We introduce the Configram Estimation (CGEM) approach to predict the accuracy of any classifier that delivers confidences. By calculating classification confidences for unseen samples, it is possible to train an offline regression model, capable of predicting the classifier’s accuracy on novel data in a semi-supervised fashion. We evaluate our method with several diverse classifiers and on analytical and real-world benchmark data sets for both incremental and active learning. The results show that our novel method improves accuracy estimation over standard methods and requires less supervised training data after deployment of the model. We demonstrate the application of our approach to a challenging robot object recognition task, where the human teacher can use our method to judge sufficient training.

Optimasi nilai k dan parameter lag algoritme k-nearest neighbor pada prediksi tingkat hunian hotel

Hotel occupancy rates are the most important factor in hotel business management. Prediction of the rates for the next few months determines the manager's decision to arrange and provide all the needed facilities. This study performs the optimization of lag parameters and k values of the k-Nearest Neighbor algorithm on hotel occupancy history data. Historical data were arranged in the form of supervised training data, with the number of columns per row according to the lag parameter and the number of prediction targets. The kNN algorithm was applied using 10-fold cross-validation and k-value variations from 1-30. The optimal lag was obtained at intervals of 14-17 and the optimal k at intervals of 5-13 to predict occupancy rates of 1, 3, 6, 9, and 12 months later. The obtained k-value does not follow the rule at the square root of the number of sample data.

Domain Adversarial Transfer Network for Cross-Domain Fault Diagnosis of Rotary Machinery

Recently, deep learning-based intelligent fault diagnosis techniques have obtained good classification performance with amount of supervised training data. However, domain shift problem between the training and testing data usually occurs due to variation in operating conditions and interferences of environment noise. Transfer learning provides a promising tool for handling the cross-domain diagnosis problems by leveraging knowledge from the source domain to help learning in the target domain. Most existing studies attempt to learn both domain features in a common feature space to reduce the domain shift, which are not optimal on specific discriminative tasks and can be limited to small shifts. This article proposes a novel domain adversarial transfer network (DATN), exploiting task-specific feature learning networks and domain adversarial training techniques for handling large distribution discrepancy across domains. First, two asymmetric encoder networks integrating deep convolutional neural networks are designed for learning hierarchical representations from the source domain and target domain. Then, the network weights learned in source tasks are transferred to improve training on target tasks. Finally, domain adversarial training with inverted label loss is introduced to minimize the difference between source and target distributions. To validate the effectiveness and superiority of the proposed method in the presence of large domain shifts, two fault data sets from different test rigs are investigated, and different fault severities, compound faults, and data contaminated by noise are considered. The experimental results demonstrate that the proposed method achieves the average accuracy of 96.45% on the bearing data set and 98.92% on the gearbox data set, which outperforms other algorithms.

Deep learning-based adversarial multi-classifier optimization for cross-domain machinery fault diagnostics

Despite the recent success in data-driven machinery fault diagnosis, cross-domain diagnostic tasks still remain challenging where the supervised training data and unsupervised testing data are collected under different operating conditions. In order to address the domain shift problem, minimizing the marginal domain distribution discrepancy is considered in most of the existing studies. While improvements have been achieved, the class-level alignments between domains are generally neglected, resulting in deteriorations in testing performance. This paper proposes an adversarial multi-classifier optimization method for cross-domain fault diagnosis based on deep learning. Through adversarial training, the overfitting phenomena of different classifiers are exploited to achieve class-level domain adaptation effects, facilitating extraction of domain-invariant features and development of cross-domain classifiers. Experiments on three rotating machinery datasets are carried out for validations, and the results suggest the proposed method is promising for cross-domain fault diagnostic tasks.

Gradient boosting in crowd ensembles for Q-learning using weight sharing

Reinforcement learning (RL) is a double-edged sword: it frees the human trainer from having to provide voluminous supervised training data or from even knowing a solution. On the other hand, a common complaint about RL is that learning is slow. Deep Q-learning (DQN), a somewhat recent development, has allowed practitioners and scientists to solve tasks previously thought unsolvable by a reinforcement learning approach. However DQN has resulted in an explosion in the number of model parameters which has further exasperated the computational needs of Q-learning during training. In this work, an ensemble approach which improves the training time, in terms of the number of interactions with the training environment, is proposed. In the presented experiments, it is shown that the proposed approach improves stability of during training, results in improved average performance, results in more reliable training, and faster learning of features in convolutional layers.

Online Vehicle Routing With Neural Combinatorial Optimization and Deep Reinforcement Learning

Online vehicle routing is an important task of the modern transportation service provider. Contributed by the ever-increasing real-time demand on the transportation system, especially small-parcel last-mile delivery requests, vehicle route generation is becoming more computationally complex than before. The existing routing algorithms are mostly based on mathematical programming, which requires huge computation time in city-size transportation networks. To develop routes with minimal time, in this paper, we propose a novel deep reinforcement learning-based neural combinatorial optimization strategy. Specifically, we transform the online routing problem to a vehicle tour generation problem, and propose a structural graph embedded pointer network to develop these tours iteratively. Furthermore, since constructing supervised training data for the neural network is impractical due to the high computation complexity, we propose a deep reinforcement learning mechanism with an unsupervised auxiliary network to train the model parameters. A multisampling scheme is also devised to further improve the system performance. Since the parameter training process is offline, the proposed strategy can achieve a superior online route generation speed. To assess the proposed strategy, we conduct comprehensive case studies with a real-world transportation network. The simulation results show that the proposed strategy can significantly outperform conventional strategies with limited computation time in both static and dynamic logistic systems. In addition, the influence of control parameters on the system performance is investigated.

Flow graphs as data structures for inducing classifiers

This paper describes an empirical research work based on the use of a suitable data structure, named Flow Graph (FG), that can be induced from a supervised training data set. A FG can be approached as a weighted and labeled digraph that summarizes a

Robust Histopathology Image Analysis: to Label or to Synthesize?

Detection, segmentation and classification of nuclei are fundamental analysis operations in digital pathology. Existing state-of-the-art approaches demand extensive amount of supervised training data from pathologists and may still perform poorly in images from unseen tissue types. We propose an unsupervised approach for histopathology image segmentation that synthesizes heterogeneous sets of training image patches, of every tissue type. Although our synthetic patches are not always of high quality, we harness the motley crew of generated samples through a generally applicable importance sampling method. This proposed approach, for the first time, re-weighs the training loss over synthetic data so that the ideal (unbiased) generalization loss over the true data distribution is minimized. This enables us to use a random polygon generator to synthesize approximate cellular structures (i.e., nuclear masks) for which no real examples are given in many tissue types, and hence, GAN-based methods are not suited. In addition, we propose a hybrid synthesis pipeline that utilizes textures in real histopathology patches and GAN models, to tackle heterogeneity in tissue textures. Compared with existing state-of-the-art supervised models, our approach generalizes significantly better on cancer types without training data. Even in cancer types with training data, our approach achieves the same performance without supervision cost. We release code and segmentation results on over 5000 Whole Slide Images (WSI) in The Cancer Genome Atlas (TCGA) repository, a dataset that would be orders of magnitude larger than what is available today.

Quality Index of Supervised Data for Convolutional Neural Network-Based Localization

Automated guided vehicles (AGVs) are important in modern factories. The main functions of an AGV are its own localization and object detection, for which both sensor and localization methods are crucial. For localization, we used a small imaging sensor named a single-photon avalanche diode (SPAD) light detection and ranging (LiDAR), which uses the time-of-flight principle and arrays of SPADs. The SPAD LiDAR works both indoors and outdoors and is suitable for AGV applications. We utilized a deep convolutional neural network (CNN) as a localization method. For accurate CNN-based localization, the quality of the supervised data is important. The localization results can be poor or good if the supervised training data are noisy or clean, respectively. To address this issue, we propose a quality index for supervised data based on correlations between consecutive frames visualizing the important pixels for CNN-based localization. First, the important pixels for CNN-based localization are determined, and the quality index of supervised data is defined based on differences in these pixels. We evaluated the quality index in indoor-environment localization using the SPAD LiDAR and compared the localization performance. Our results demonstrate that the index correlates well to the quality of supervised training data for CNN-based localization.

Snore-GANs: Improving Automatic Snore Sound Classification With Synthesized Data.

One of the frontier issues that severely hamper the development of automatic snore sound classification (ASSC) associates to the lack of sufficient supervised training data. To cope with this problem, we propose a novel data augmentation approach based on semi-supervised conditional generative adversarial networks (scGANs), which aims to automatically learn a mapping strategy from a random noise space to original data distribution. The proposed approach has the capability of well synthesizing "realistic" high-dimensional data, while requiring no additional annotation process. To handle the mode collapse problem of GANs, we further introduce an ensemble strategy to enhance the diversity of the generated data. The systematic experiments conducted on a widely used Munich-Passau snore sound corpus demonstrate that the scGANs-based systems can remarkably outperform other classic data augmentation systems, and are also competitive to other recently reported systems for ASSC.

Deep Learning with Unsupervised Data Labeling for Weed Detection in Line Crops in UAV Images

In recent years, weeds have been responsible for most agricultural yield losses. To deal with this threat, farmers resort to spraying the fields uniformly with herbicides. This method not only requires huge quantities of herbicides but impacts the environment and human health. One way to reduce the cost and environmental impact is to allocate the right doses of herbicide to the right place and at the right time (precision agriculture). Nowadays, unmanned aerial vehicles (UAVs) are becoming an interesting acquisition system for weed localization and management due to their ability to obtain images of the entire agricultural field with a very high spatial resolution and at a low cost. However, despite significant advances in UAV acquisition systems, the automatic detection of weeds remains a challenging problem because of their strong similarity to the crops. Recently, a deep learning approach has shown impressive results in different complex classification problems. However, this approach needs a certain amount of training data, and creating large agricultural datasets with pixel-level annotations by an expert is an extremely time-consuming task. In this paper, we propose a novel fully automatic learning method using convolutional neuronal networks (CNNs) with an unsupervised training dataset collection for weed detection from UAV images. The proposed method comprises three main phases. First, we automatically detect the crop rows and use them to identify the inter-row weeds. In the second phase, inter-row weeds are used to constitute the training dataset. Finally, we perform CNNs on this dataset to build a model able to detect the crop and the weeds in the images. The results obtained are comparable to those of traditional supervised training data labeling, with differences in accuracy of 1.5% in the spinach field and 6% in the bean field.