Abstract Seed varietal purity is vital to establish a uniform plant population. If the seeds are impure, it creates an unhealthy plant population that brings labor-intensive crop production. In this study, a rapid inspection method was established to classify the paddy seed based on varietal purity using a machine vision technique with multivariate analysis methods. Three varieties of paddy seeds were taken, namely - BR 11, BRRI dhan 28 and BRRI dhan 29. The individual paddy seed image was captured using an RGB camera with white LED lighting conditions in the laboratory. An image processing algorithm was developed for extracting 20 important features (seven color features, nine morphological features, and four textural features) from 375 paddy seed images. In the next step, the significant difference of extracted features data among the paddy varieties was studied using variance analysis. Also, the principal component analysis was performed to explore the separability of paddy seed varieties. Accordingly, the paddy seed variety classification models were developed for the combination of paddy varieties and selected feature data using partial least squares-discriminant analysis (PLS-DA), Support vector machine-classification (SVM-C) and K-Nearest Neighbors (KNN) algorithm. During model development, it was seen that the morphological image features were more significant compare to color and textural image features. The accuracy of 83.8%, 93.9%, and 87.2% was achieved using combined selected features of color, morphological, and textural for the PLS-DA, SVM-C, and KNN model, respectively. Finally, it was stated that the SVM-C algorithm with selected features of color, morphological, and textural could be used to classify the paddy seed variety.