Cities account for over 75% of primary energy use in the world, with buildings making up a significant share of this energy use. Previous simulation-based research has established that building energy use can be greatly impacted by surrounding urban systems such as other buildings, vegetation, and roads. Understanding these relationships is thus critical to enhancing the efficiency of energy-intensive urban environments. Taking advantage of the recent profusion of urban data, this paper proposes a novel Context-aware Urban Energy Analytics (CUE-A) framework to empirically extract and quantify the relationships between building energy use and the spatial proximity of multiple surrounding urban systems. We apply the CUE-A framework to a case study of 477 buildings in a mid-size U.S. city to demonstrate its merits and the statistical significance of explored relationships. Results show that spatial proximity of other buildings, trees, and roads is associated with varied and significant changes in both the central tendency and variability of building energy use, indicating that empirical frameworks, which are a growing field of work, can serve as useful complements to existing simulation models. Further, our paper demonstrates that energy-aware urban planning and design has the potential to unlock energy efficiency and low-carbon pathways for cities around the world.