Data-driven soft sensors that predict the primary variables of a process by using the secondary measurements have drawn increased research interests recently. They are easy to develop and only require a good historical data set. Among them, the partial least squares (PLS) based soft sensor is the most commonly used approach for industrial applications. As industrial processes often experience time-varying changes, it is desirable to update the soft sensor model with the new process data once the soft sensor is implemented online. Because the PLS algorithms are sensitive to outliers in the dataset, outlier detection and handling plays a critical role in the development of the PLS based soft sensors. In this work, we develop a multivariate approach for online outlier detection. In addition, to differentiate outliers caused by erroneous readings from those caused by process changes, we propose a Bayesian supervisory approach to analyze and classify the detected outliers. Finally, to address time-varying nature of industrial processes, we proposed a simple yet effective scheme to update the detection threshold. Both simulated and industrial case studies of the Kamyr digesters are used to demonstrate the effectiveness of the proposed approaches.