Lead-free ferroelectric electrocaloric (EC) materials are promising candidates for the next generation of cooling materials. Bi0.5Na0.5TiO3 (BNT)-based relaxor ferroelectrics, one of the most representative EC materials, exhibit a superior electrocaloric effect (ECE) near the depolarization temperature (Td). However, there are more and more concerns over the reliability of the widely used indirect measurement method. Here, to comprehend the difference between indirectly and directly obtained ECE results in BNT-based system, the applicability of indirect measurement (based on Maxwell equation) was critically analyzed with respect to the reliable direct measurement (based on heat flow). As evidenced in prototypical BNT-based compositions with different Td values, the dynamic behaviors of local polar structures including ferroelectric domains and polar nano-sized regions (PNRs), can largely affect the measured polarization and thus affect the reliability of indirect measurement. After systematically comparing and analyzing the composition-, temperature-, electric field (E-field)-dependent ECE results obtained from indirect and direct ways, the applicable working regions for the indirect method were determined. This work reveals the underlying mechanism behind direct and indirect ECE measurements, providing the “indirect + direct” experimental strategy for effectively evaluating the ECE performance in BNT-based relaxor ferroelectrics.