Rock mineral recognition is a costly and time-consuming task when using traditional methods, during which physical and chemical properties are tested at micro- and macro-scale in the laboratory. As a solution, a comprehensive recognition model of 12 kinds of rock minerals can be utilized, based upon the deep learning and transfer learning algorithms. In the process, the texture features of images are extracted and a color model for rock mineral identification can also be established by the K-means algorithm. Finally, a comprehensive identification model is made by combining the deep learning model and color model. The test results of the comprehensive model reveal that color and texture are important features in rock mineral identification, and that deep learning methods can effectively improve identification accuracy. To prove that the comprehensive model could extract effective features of mineral images, we also established a support vector machine (SVM) model and a random forest (RF) model based on Histogram of Oriented Gradient (HOG) features. The comparison indicates that the comprehensive model has the best performance of all.