Data-driven anomaly detection is one of the central issues for the implementation of predictive maintenance in cyber-physical systems (CPS). The increasing nonstationary dynamics in CPS lead to complex shapes of collected process data, e.g., convex and nonconvex. Algorithms with the ability of handling nonconvex data are desired for anomaly detection tasks in CPS. Archetypal analysis selects extreme points (archetypes) to represent a dataset-mainly to improve run times, e.g., for anomaly detection. The classic archetypal analysis uses convex combinations of archetypes to represent a set of observations (data points). This leads to the performance depression of the classic archetypal analysis methods for anomaly detection tasks on nonconvex datasets. Such nonconvex sets are typical for CPS. In this article, the anomaly detection tasks are considered as one-class classification problem due to the lack of abnormal samples. A novel nonconvex archetypal one-class classification algorithm is proposed to address the challenge of nonconvex data, which combines the random projection and the AdaBoost algorithm. The major advantages of this method are its high efficiency, flexibility, and its ability to handle both convex and nonconvex datasets, i.e., it can be applied to CPS analysis tasks. The usefulness of the presented approach is evaluated for fault diagnosis tasks in CPS.