Automatic modulation classification (AMC) plays a crucial role in the stages of processing signals from unknown sources and monitoring the airwaves. This paper presents an AMC method based on machine learning (ML) using constellation diagrams, distribution test function and high-order cumulants (HOCs) and novel filtering and regeneration technique. A data clustering approach based on the signal phase characteristics is utilized to perform filtering, regeneration, and testing distribution functions. NI PXIe-1065 was used as a source of modulated signals, which is a multifunctional experimental complex for generating signals with various types of modulation. In this work, 5 types of modulation schemes were employed, specifically BPSK, QPSK, PSK-8, PSK-16 and PSK-32. The proposed classification method consists of three main stages. At the first stage, the constellation diagram is partitioned into clusters. At the second stage, the phase distribution function of signals within each cluster is analyzed, HOCs were calculated and novel filtering and regeneration techniques were employed. The main concept of the proposed approach is compressing angular ranges depicted on the constellation diagram. This involves filtering points and subsequently regenerating, which means relocating points that fall within adjacent phase ranges. Also, a statistical analysis of the phase transition distribution function in clusters is conducted at this step. At the last stage the obtained data set was trained using XGBOOST ML method. The proposed approach shows excellent results and can be a strong contender to existing NN based AMC methods, achieving 95 % accuracy at an SNR value of 9 dB.