Removing fat from whole milk and adding water to milk to increase its volume are among the most common food fraud practices that alter the characteristics of milk. Usually, deviations from the expected fat content can indicate adulteration. Infrared spectroscopy is a commonly used technique for distinguishing pure milk from adulterated milk, even when it comes from different animal species. More recently, portable spectrometers have enabled in situ analysis with analytical performance comparable to that of benchtop instruments. Partial Least Square (PLS) analysis is the most popular tool for developing calibration models, although the increasing availability of portable near infrared spectroscopy (NIRS) has led to the use of alternative supervised techniques, including support vector machine (SVM). The aim of this study was to develop and implement a method based on the combination of a compact and low-cost Fourier Transform near infrared (FT-NIR) spectrometer and variable cluster-support vector machine (VC-SVM) hybrid model for the rapid classification of milk in accordance with EU Regulation EC No. 1308/2013 without any pre-treatment. The results obtained from the external validation of the VC-SVM hybrid model showed a perfect classification capacity (100% sensitivity, 100% specificity, MCC = 1) for the radial basis function (RBF) kernel when used to classify whole vs. not-whole and skimmed vs. not-skimmed milk samples. A strong classification capacity (94.4% sensitivity, 100% specificity, MCC = 0.95) was also achieved in discriminating semi-skimmed vs. not-semi-skimmed milk samples. This approach provides the dairy industry with a practical, simple and efficient solution to quickly identify skimmed, semi-skimmed and whole milk and detect potential fraud.