Building energy consumption prediction is important for sustainable building and city construction. However, traditional data-driven prediction methods weakly consider real-time interactions among humans, energy systems, and the environment, which reduces forecasting accuracies. In addition, compared to traditional machine learning algorithms, more suitable algorithms may be required to improve forecasting accuracy and accelerate convergence speed. In this study, an improved energy-consumption prediction system is proposed that includes input system modification and algorithm improvement. For inputs, the probability of sockets, lighting, and air-conditioning related behaviors are included as inputs. Compared with previous studies, the duration of behaviors instead of the number of behavior occurrences is used in this study to solve the behavior probability, which can more accurately reflect occupant-building interactions during forecasting. For algorithms, swarm intelligence algorithms with circle mapping and self-adaptive weight adjustment are used, retaining the diversity of the population and optimizing the weight of the neural network, thereby resulting in better algorithm performance. Results showed that the case using occupant-building interaction inputs and improved swarm intelligence algorithms had the best performance. The coefficients of determination are 0.9588–0.9901, and the mean absolute percentage errors are 4.44%–11.60%. When using the same algorithms and optimization, the modified inputs can improve the prediction performance by 9.15%–21.59%. Also, when using the same inputs, the improved algorithms can improve the precision performance by 5.46%–22.37%. Also, this study discusses the priority of selecting a combination of inputs and algorithms, which will help users to identify different inputs and algorithms when there are different data collection conditions and accuracy requirements. All cases in this study had coefficients of determination above 0.95, indicating good fits. In addition, models of occupant behavior probability are discussed and may be used to help develop occupant behavior research, particularly for random behavior.