Context. High-resolution millimeter and submillimeter (mm and submm) polarization observations have opened a new era in the understanding of how magnetic fields are organized in star forming regions, unveiling an intricate interplay between the magnetic fields and the gas in protostellar cores. However, to assess the role of the magnetic field in the process of solar-type star formation, it is important to understand to what extent the polarized dust emission is a good tracer of the magnetic field in the youngest protostellar objects. Aims. In this paper, we present a thorough investigation of the fidelity and limitations of using dust polarized emission to map the magnetic field topologies in low-mass protostars. Methods. To assess the importance of these effects, we performed an analysis of magnetic field properties in 27 realizations of magnetohydrodynamics (MHD) models following the evolution of physical properties in star-forming cores. Assuming a uniform population of dust grains the sizes of which follow the standard MRN size distribution, we analyzed the synthetic polarized dust emission maps produced when these grains align with the local B-field because of radiative torques (B-RATs). Results. We find that mm and submm polarized dust emission is a robust tracer of the magnetic field topologies in inner protostellar envelopes and is successful at capturing the details of the magnetic field spatial distribution down to radii ~100 au. Measurements of the line-of-sight-averaged magnetic field line orientation using the polarized dust emission are precise to <15° (typical of the error on polarization angles obtained with observations from large mm polarimetric facilities such as ALMA) in about 75%–95% of the independent lines of sight that pass through protostellar envelopes. Large discrepancies between the integrated B-field mean orientation and the orientation reconstructed from the polarized dust emission are mostly observed in (i) lines of sight where the magnetic field is highly disorganized and (ii) those that probe large column densities. Our analysis shows that the high opacity of the thermal dust emission and low polarization fractions could be used to avoid using the small fraction of measurements affected by large errors.