The fracture energy of lightweight aggregate concrete (LWAC) is comparatively lower than that of normal weight concrete (NWC), primarily due to the propensity for fracture exhibited by lightweight aggregates and their characteristics that modify the damage softening mechanism at the crack tip. Few studies have been done on how to precisely predict the LWAC fracture energy. In this work, 111 sets of three-point bending test data were gathered, and the impact of aggregate types and size, concrete density, water-cement (w/c) ratio, and compressive strength on the total average fracture energy (GF) and the initial fracture energy (Gf) was thoroughly evaluated. Five fracture energy prediction methods’ applicability were compared, and a fracture energy prediction expression appropriate for LWAC was presented. Finally, the three-point bending test was used to confirm the prediction. The results show that the fracture energy of LWAC is significantly influenced by changes in aggregate and mortar characteristics. The compressive strength has substantial correlations with the fracture energy of LWAC. After adjusting the aggregate and mortar coefficients, the prediction formula can correctly predict Gf and GF.