The seed is an inevitable element for agricultural and industrial production. The non-destructive paddy seed variety identification is essential to assure paddy purity and quality. This research is aimed at developing a computer vision-based system to identify paddy varieties using multiple heterogeneous features, exploiting textural, external, and physical properties. We captured the paddy seed images without any fixed setup to make the system user friendly at both industry and farmer levels, which can lead to illumination problems in the images. To overcome this problem, we introduced a modified histogram oriented gradient (T20-HOG) feature that can describe the illumination, scale, and rotational variations of a paddy image. We also utilized the existing Haralick and traditional features and the dimensionality of the features is reduced by the Lasso feature selection technique. The selected features are used to train the feed-forward neural network (FNN) to predict the paddy variety. The experiments conducted on two different datasets: BDRICE, and VNRICE. Results of our method are shown in terms of four standard evaluation metrics, namely, accuracy, precision, recall, and F_1 score, and achieved 99.28%, 98.64%, 98.48%, and 98.56% score, respectively. We also compared our system efficiency with existing studies. The experimental results demonstrate that our proposed features are effective to identify paddy variety and achieved a new state-of-the-art performance. And we also observed that our newly proposed T20-HOG features have a major impact on overall system performance.