In this paper, a new algorithm named the improved transient search optimization algorithm (ITSOA) is utilized to solve classical test functions, optimize the consumption of building energy, and optimize hybrid energy system production. The conventional TSOA draws inspiration from the fleeting behavior of electrical circuits with energy storage components. Rosenbrock’s direct rotation technique is used to improve the traditional TSOA performance against exploration and exploitation unbalance. First, the ITSOA performance is investigated in solving 23 classical benchmark functions, and the outcomes have shown the superior capability of the recommended algorithm in comparison with the conventional TSOA, DMO, SHO, GA, MRFO, and PSO methods. Also, the ITSOA proficiency is verified in solving the building energy optimization (BEO) problem for minimizing the energy usage of two simple and detailed buildings. The optimization results showed that the optimized solutions of ITSOA in single and multi-objective optimizations compared to conventional TSOA, DMO, SHO, GA, MRFO, and PSO obtained a lower value of the cost function. Also, the superiority of ITSOA has been confirmed to solve the BEO compared to previous methods. Moreover, the multi-objective optimization results have shown that ITSOA is able to determine the ultimate solution among the Pareto front set based on the fuzzy decision-making approach and building energy utilization decisions.