For a data-driven building energy prediction model, the insufficient data available for new buildings and existing buildings with limited data pose a challenge to achieving the expected prediction performance. Transfer learning (TL) and continual learning (CL) are two widely applied effective methods for addressing the shortage of building data. The actual building projects typically exist in a scenario of data increment, wherein their energy consumption data gradually accumulates with usage. Previous studies have indicated that the TL strategy can transfer the learned data distribution patterns from source buildings to target buildings of the same type. Meanwhile, the CL strategy can extract the dynamic characteristics from continuously accumulated data. Both strategies can be employed to enhance the performance of energy consumption prediction models for buildings with insufficient data. However, existing studies on data scarcity in energy prediction primarily focus on buildings with fixed data quantities, and have not provided sufficient analysis for newly constructed buildings in a data increment scenario. Therefore, this study proposes a TL-to-CL strategy to enhance the energy prediction performance of newly constructed buildings in a data increment scenario. When the available data is limited, TL should be utilized. But as the data volume increases to be sufficient, TL should be transformed to CL at the reasonable time. Additionally, this study provides a reference for determining the transform time for the TL-to-CL strategy. The validation was conducted using two years of energy consumption data from 36 buildings in the Genome Project 2 dataset. The results demonstrate that the proposed TL-to-CL strategy (transform time = 4-week) obtain the overall better prediction performance than pure TL, pure CL and combined TL-CL without transformation between the two) strategies. The prediction improvement ratio (PIR) can be as high as 0.9 compared with the traditional LSTM.