Grape farming is one of the most lucrative agricultural enterprises in India. There are several biotic and abiotic stress conditions that may adversely affect the yield if not tackled at the right time. It is crucial that the farmer can correctly identify and monitor the type of stress so that steps can be taken to prevent undesirable outcomes. We have gathered a dataset of these stress conditions on grape berries and categorized them into eight classes, necrosis, shriveling, and honeydew by mealybug, mealybug incidence, spray injury, thrips scarring, pink berry, and powdery mildew. Transfer learning was used to test the performance of six major deep learning image classification architectures (namely MobileNet-v2, Inception-v3, Inception-ResNet-v2, ResNet-v2, NASNet, and PNASNet) with variations in training conditions and hyper parameters. The results were compared to determine the most feasible and accurate deep learning architecture and its hyper parameters for the given problem statement. The experiment shows that Inception-ResNet-v2 obtained maximum classification accuracy of 88.75% when learning rate of 0.035 and minibatch size of 10 were applied using 8000 training steps. This result will act as a pre-requisite for the development of an application for mapping vineyard stress conditions on berries and give automated advisory.