Residential buildings account for a large proportion of global energy consumption in both low- and high- income countries. Efficient planning to meet building energy needs while increasing operational, economic, and environmental efficiency requires accurate, high spatial resolution information on energy consumption. Such information is difficult to acquire and most models for estimating residential building energy consumption require detailed knowledge of individual homes and communities which are unlikely to be available at a large scale.To address this need, we introduce a methodology for automatically estimating individual building energy consumption from overhead imagery (e.g. satellite, aerial) and demonstrate the effect of spatial aggregation for further improving accuracy. We use a three-step estimation process by which we (1) automatically segment buildings in overhead imagery using a convolutional neural network and classify them by type (residential or commercial), (2) extract features (e.g. area, perimeter, building density) from those identified residential buildings, and (3) use random forests regression to estimate building energy consumption from those features.The predictive capability of this approach is evaluated in two locations: Gainesville, Florida, and San Diego, California. The building detector correctly identifies 84% and 88% of buildings in Gainesville and San Diego, respectively. The type of building is classified successfully 99% of the time for residential buildings and 74% of the time for commercial buildings. With residential buildings identified, this approach predicted individual building-level energy consumption with an R2 of 0.28 and 0.38 for Gainesville and San Diego, respectively. Aggregating the energy consumption estimates across small neighborhoods of size 200 × 200 m and 1000 × 1000 m in Gainesville results in an R2 of 0.91 and 0.97, respectively. We also explore the sensitivity of estimates in San Diego and Gainesville to the training data and its size. Our results suggest that using overhead imagery to estimate the size of buildings has a higher predictive power in estimating residential building energy consumption than common alternatives.