We report the effect of impurity doping at the Mn-site on magnetization and magnetic entropy change (ΔSm) in Pr0.6Ca0.4Mn0.96B0.04O3 with B cations having different electronic and spin states [B=Al3+(d0), Cr3+(d3), Fe3+(d5), Co2+(d7), Ni2+(d8), and Ru4+/5+(d4/3)]. The charge ordered antiferromagnetic insulator Pr0.6Ca0.4MnO3 transforms into a ferromagnetic metal in the absence of an external magnetic field (H) for B=Cr, Ni, Co, and Ru, whereas B=Fe and Al samples remain as antiferromagnetic insulators. It is found that the applied magnetic field induced an abrupt increase in the magnetization above a critical value of H (field-induced metamagnetic transition) in the paramagnetic state for B=Cr, Ni, and Co but not for B=Ru, which has the highest Curie temperature in the series (TC=205 K). The B=Co sample shows the maximum value of the magnetic entropy (−ΔSm=7.37 J/kg K under ΔH=5 T), which is followed by −ΔSm=6.77,6.0,5.07,5.04,2.72 J/kg K for B=Ni, Cr, Ru, Fe, and Al, respectively. We have also found a large magnetic entropy change (−ΔSm=8.42 J/kg K under ΔH=5) by differential scanning calorimetric study in B=Co sample. The large ΔSm values in B=Co, Cr, and Ni compared to Ru originate from the distinct field dependence of the magnetization above TC. It is suggested that both charge-orbital fluctuations and ferromagnetic polarons exist in the paramagnetic phase of B=Co, Cr, and Ni samples in the absence of a magnetic field, and the external magnetic field suppresses charge-orbital fluctuations and enhances size of ferromagnetic polarons, which leads to observed large ΔSm values. Our study suggests that creation of charge-orbital fluctuations and ferromagnetic polarons by dilution of the Mn-site could be a simple way to enhance the magnetocaloric effect in charge ordered manganites.