A “big-data” algorithm for KNN-PLS

Abstract

A well known issue regarding PLS lies in the difficulty to apprehend nonlinearities. As a solution, an extension of the method, “KNN-PLS”, was developed. However, this solution is based on a neighbourhood selection algorithm whose execution time is highly dependent on the size of the database, leading to prohibitive response times. This article proposes, as an alternative, a new method designed to process large data volumes: “parSketch-PLS”. This method combines a “big-data domain” neighbour selection method, called “parSketch”, and the PLS method. Essentially, this paper presents a feasibility study, regarding the adaptation of big-data principles for spectral datasets, in non-linear contexts. The parSketch method has not been studied in the context of chemometrics and considering the specific properties of spectral data. This method is based on the approximation of sample neighbourhoods, based on spectral distances. It is then necessary to investigate the relevance of these neighbourhoods for PLS models and predictions. This article compares PLS and KNN-PLS methods with the parSketch-PLS method. In this context, PLS allows to process large volumes of data quickly but performs poorly in prediction while the KNN-PLS method returns accurate predictions, yet with much higher computational time. This paper shows that the proposed pairing offers a good operational trade-off between prediction performances and computational cost. In addition a comprehensive study of the input parameters of parSketch-PLS is conducted. The objective is to understand the influence of these parameters on the prediction performances. This article proposes a framework to interpret the neighbourhoods returned by comparing their relative sizes with the evolution of performances and the input parameters of parSketch.