The escalating prevalence of diabetes mellitus underscores the critical need for non-invasive screening tools capable of early disease detection. Present diagnostic techniques depend on invasive procedures, which highlights the need for advancement of non-invasive alternatives for initial disease detection. Machine learning in integration with the optical sensing technology can effectively analyze the signal patterns associated with diabetes. The objective of this research is to develop and evaluate a non-invasive optical-based method combined with machine learning algorithms for the classification of individuals into normal, prediabetic, and diabetic categories. A novel device was engineered to capture real-time optical vascular signals from participants representing the three glycemic states. The signals were then subjected to quality assessment and preprocessing to ensure data reliability. Subsequently, feature extraction was performed using time-domain analysis and wavelet scattering techniques to derive meaningful characteristics from the optical signals. The extracted features were subsequently employed to train and validate a suite of machine learning algorithms. An ensemble bagged trees classifier with wavelet scattering features and random forest classifier with time-domain features demonstrated superior performance, achieving an overall accuracy of 86.6% and 80.0% in differentiating between normal, prediabetic, and diabetic individuals based on the optical vascular signals. The proposed non-invasive optical-based approach, coupled with advanced machine learning techniques, holds promise as a potential screening tool for diabetes mellitus. The classification accuracy achieved in this study warrants further investigation and validation in larger and more diverse populations.