Self-supervised pre-training on industrial time-series

Abstract

Machine Learning (ML) and Deep Learning (DL) techniques have achieved remarkable results in many domains resulting in excellent performances in fields including Natural Language Processing (NLP), Computer Vision (CV) and Speech Recognition (SR). In light of these achievements, the application of ML algorithms to the industry world is appealing and very interesting results have already been produced by the scientific community. However, industrial data are often significantly different from those commonly employed in the disciplines where DL has had a larger impact. Industrial datasets are often limited in size and are constituted by large amounts of unlabelled data. In contrast, in CV and NLP, large deep neural networks can benefit from large labelled datasets, that can thus be used to train DL models in a fully-supervised fashion. In this work, we adopt a self-supervised approach to process heterogeneous dataset characterized by a relatively large number of unlabelled data and few labelled ones. We apply the proposed approach to two industrial time-series data and show that leveraging unlabelled data results in improved performance on classification downstream tasks.