This study aimed to investigate the performance and factors affecting the species classification of convolutional neural network (CNN) architecture using whole-part and earlywood-part cross-sectional datasets of six Korean Quercus species. The accuracy of species classification for each condition was analyzed using the datasets, data augmentation, and optimizers—stochastic gradient descent (SGD), adaptive moment estimation (Adam), and root mean square propagation (RMSProp)—based on a CNN architecture with three to four convolutional layers. The model trained with the augmented dataset yielded significantly superior results in terms of classification accuracy compared to the model trained with the non-augmented dataset. The augmented dataset was the only factor affecting classification accuracy in the final five epochs. In contrast, four factors in the entire epoch, such as the Adam and SGD optimizers and the earlywood-part and whole-part datasets, affected species classification. The arrangement of earlywood vessels, broad rays, and axial parenchyma was identified as a major influential factor in the CNN species classification using gradient-weighted class activation mapping (Grad-CAM) analysis. The augmented whole-part dataset with the Adam optimizer achieved the highest classification accuracy of 85.7% during the final five epochs of the test phase.