With the advent of flexible working arrangements, we are observing a dramatic shift in how buildings are occupied today, which presents an opportunity to optimize Heating, Ventilation, and Air Conditioning system temperature setpoints based on variations in occupancy. Guidelines often suggest the adoption of the highest or lowest setpoint or setback to minimize energy consumption in hot or cold climates, respectively. However, at outdoor temperatures where variations in occupancy heat loads prompt buildings to fluctuate across cooling, free-running, and heating mode, optimal setpoints and setbacks are not always the lowest or highest. In addition, the perturbations caused by rapid switching between setpoint and setback could diminish energy savings due to system destabilization. This paper aims to systematically compare the potential energy savings from fixed and optimal setpoints and setbacks across wide-ranging occupancy scenarios (four occupancy rates and 14 patterns). Energy simulations were conducted using the Department of Energy reference models for small, medium, and large office buildings to enable an exhaustive search of optimal setpoint/setbacks in 17 climate zones. Explored setpoints were 19.5 °C to 25.5 °C with intervals of 1 °C, and setbacks were 17 °C/19 °C for heating and 26 °C/28 °C for cooling. The findings indicate that, on average, while lower occupancy heat loads results in 5.48% energy reduction, a conventional fixed setpoint and setback strategy provides an additional 11.80%, and optimal selection of setpoints and setbacks could provide an additional 34.36–38.08%, emphasizing the untapped potential energy saving. To facilitate practical applications, this paper presents an interactive graphical interface: Optimal Temperature Setpoint Tool.