A large step on the initial magnetization curve has been widely observed in Nd-Fe-B hot-deformed magnets. This behavior has been considered as that the first- and latter-half parts of the initial magnetization curve correspond to the magnetization reversals of multi- and single-domain grains, respectively. In this study, the detailed magnetic domain structure of a Nd-Fe-B hot-deformed magnet has been studied for the initial magnetization and demagnetization processes by using a soft X-ray magnetic circular dichroism (XMCD) microscopy. The observed magnetization reversal behavior for the initial magnetization process is quite different from that previously considered. The multi-domain states move from grains to grains through the gradual displacement of domain wall passing through many grains, and then the massive domain wall displacement takes place. The latter part of domain wall displacement is similar with that found in the demagnetization process near the coercivity. Moreover, we found that there are several strong pinning sites which are identical for both the initial magnetization and the demagnetization processes.