Accurate probabilistic forecasting of photovoltaic (PV) power is crucial for optimizing energy scheduling in smart buildings and ensuring the low-carbon, efficient operation of building energy management systems (BEMS). However, existing feature selection techniques fail to guarantee that the selected features genuinely impact the output of forecasting models. Additionally, traditional black-box deep learning models lack clarity on whether their output truly relies on those selected features. These challenges limit the accuracy of forecasting models. To address these challenges, a novel methodology named temporal importance model explanation and temporal fusion transformers (TIME-TFT) model is proposed. Firstly, the TIME method is employed for feature selection, and interpretable outputs are used to identify important global features. Secondly, the TFT model is then utilized for forecasting tasks, providing interpretable outputs to track back to the features that TFT model pays attention to. Finally, the consistency between the interpretable outputs of TIME method and TFT model is examined to confirm predictions are based on genuinely learned selected features. Empirical studies demonstrate the superiority of the proposed TIME-TFT system, outperforming comparable models with an R2 of 0.9546. In summary, the interpretable outputs not only improve the accuracy of predictions but also provides visual evidence for predictions, thereby bolstering effectiveness and credibility in engineering practices.