This paper focuses on the day-ahead forecasting of the aggregate power of a pool of smart buildings equipped with thermostatically-controlled loads. We first propose the modeling of the aggregate behavior of its power trajectory by using a geometric approach. Specifically, we assume that the aggregate power is a homothet of a prototype building, whose physical and technical parameters are chosen to be the mean of those in the pool. This allows us to preserve the building thermal dynamics of the pool. We then apply inverse optimization to estimate the homothetic parameters with bilevel programming. The lower level characterizes the price-response of the ensemble by a set of marginal utility curves and a homothet of the prototype building, which, in turn, are inferred in the upper-level problem. The upper level minimizes the mean absolute error over a training sample. This bilevel program is transformed into a regularized nonlinear problem that is initialized with the solution given by an efficient heuristic procedure. This heuristic consists in solving two linear programs and its solution is deemed a suitable proxy for the original bilevel problem. The results have been compared to state-of-the-art methodologies.