The electrocaloric effect (ECE) of lead-free BaTiO3 based materials has drawn considerable attention for application in next-generation refrigeration, which has been widely investigated through indirect measurement methods; however, this method is sometimes ineffective because negative and positive ECEs coexist in one material. The paradox of ECE phenomena should be clarified; therefore, in this work, the ECEs of modified BaTiO3 were investigated through both direct and indirect measurement methods. The investigated BaTiO3 compositions herein are (Ba1-xCax)(Ti0.9Zr0.1)O3 (0.05 ≤ x ≤ 0.20), abbreviated as BZT-100xCa. Through indirect measurements, the ECE reached its maximum value around the Curie temperature TC, and the coexistence of negative and positive ECEs was observed in BZT-5Ca, BZT-10Ca, and BZT-15Ca. A laboratory-built ECE direct measurement system utilizing a platinum resistor was used to measure the ECE of BZT-100xCa. Through direct measurement, the maximum ECE appears several Kelvins above the TC, which coincides with the ECE through the indirect measurement, but no negative ECE is observed. Meanwhile, the ECE measured through direct measurement was smaller than that measured through indirect measurement. Through direct measurements, BZT-15Ca exhibited the largest ΔT of 0.291 K and ΔT/ΔE of 0.182 × 10−6 K m/V under 16 kV/cm, indicating that BZT-15Ca has a better ECE performance than previously reported modified BaTiO3 ceramics. By comparison, it can be concluded that the ECE through the indirect measurement method can reflect the approximate variation tendency of the actual ECE, and the negative ECE measured indirectly should be certified through direct measurement method, which doesn't influence the reliability of ECE through the indirect measurement.