The weather condition is the major external factor that affects annual building energy usage and determines the reliability of building energy assessment and simulation models. In city-scale studies, buildings’ energy assessment and simulation rely on one universal Typical Meteorological Year weather condition for the whole city. However, due to the unique morphological features and environmental conditions, each location in a city has its unique microclimate conditions. To predict and simulate more accurate and realistic building energy usage, this study developed a deep transfer learning neural network that can integrate morphological features and frontal areas' depth distribution. The frontal areas are represented as stripes with different cutting methods to reflect the impact of depth distribution. To validate the proposed method, this study conducted an on-campus experiment with a wireless sensing system. The results suggest that the proposed stripe-enabled geometry segmentation method can effectively improve the accuracy of local microclimate condition prediction. Also, the combined vertical and horizontal cutting method shows the highest performance.