Building energy simulation (BES) models rely on a variety of different input data, and the more accurate the input data are, the more accurate the model will be in predicting energy use. The objective of this paper is to show a method for obtaining higher accuracy in building energy simulations of existing buildings by combining time diaries with data from logged measurements, and also to show that more variety is needed in template values of user input data in different kinds of buildings. The case studied in this article is a retirement home in Linköping, Sweden. Results from time diaries and interviews were combined with logged measurements of electricity, temperature, and CO2 levels to create detailed occupant behavior schedules for use in BES models. Two BES models were compared, one with highly detailed schedules of occupancy, electricity use, and airing, and one using standardized input data of occupant behavior. The largest differences between the models could be seen in energy losses due to airing and in household electricity use, where the one with standardized user input data had a higher amount of electricity use and less losses due to airing of 39% and 99%, respectively. Time diaries and interviews, together with logged measurements, can be great tools to detect behavior that affects energy use in buildings. They can also be used to create detailed schedules and behavioral models, and to help develop standardized user input data for more types of buildings. This will help improve the accuracy of BES models so the energy efficiency gap can be reduced.