As one of the most important agricultural production types in the world, orchards have high economic, ecological, and cultural value, so the accurate and timely mapping of orchards is highly demanded for many applications. Selecting a remote-sensing (RS) data source is a critical step in efficient orchard mapping, and it is hard to have a RS image with both rich temporal and spatial information. A trade-off between spatial and temporal resolution must be made. Taking grape-growing regions as an example, we tested imagery at different spatial and temporal resolutions as classification inputs (including from Worldview-2, Landsat-8, and Sentinel-2) and compared and assessed their orchard-mapping performance using the same classifier of random forest. Our results showed that the overall accuracies improved from 0.6 to 0.8 as the spatial resolution of the input images increased from 58.86 m to 0.46 m (simulated from Worldview-2 imagery). The overall accuracy improved from 0.7 to 0.86 when the number of images used for classification was increased from 2 to 20 (Landsat-8) or approximately 60 (Sentinel-2) in one year. The marginal benefit of increasing the level of details (LoD) of temporal features on accuracy is higher than that of spatial features, indicating that the classification ability of temporal information is higher than that of spatial information. The highest accuracy of using a very high-resolution (VHR) image can be exceeded only by using four to five medium-resolution multi-temporal images, or even two to three growing season images with the same classifier. Combining the spatial and temporal features from multi-source data can improve the overall accuracies by 5% to 7% compared to using only temporal features. It can also compensate for the accuracy loss caused by missing data or low-quality images in single-source input. Although selecting multi-source data can obtain the best accuracy, selecting single-source data can improve computational efficiency and at the same time obtain an acceptable accuracy. This study provides practical guidance on selecting data at various spatial and temporal resolutions for the efficient mapping of other types of annual crops or orchards.