We have theoretically studied the domain pinning effect from the internal bias field generated by aligned dipolar defects. By ordering defect dipoles along the polarization direction, the experimentally observed ‘hard’ properties in ferroelectric materials, including shifted polarization–electric field (P–E) loops, biased strain-electric field (S–E) loops, suppressed piezoelectric and dielectric responses in sufficiently poled and aged ferroelectric system, can be well described by our model. As expected, the P–E loops and S–E loops become more asymmetric with the increase of defect concentration. The local domain evolution around dipolar defects under an electric field shows the physical process for the domain pinning effect, i.e. the difficulty of domain switching due to the presence of defect dipoles. Finally, the domain memory effect during heating/cooling cycle is simulated. Interestingly, domain configuration can be memorized when a ferroelectric system with dipolar defects goes through the ferro-para-ferro phase transition cycle, and the simulated time steps of domain recovery are strongly correlated with the defect dipole concentration. All theoretical results from our model are consistent with the experimental results. We can conclude that the local interaction between dipole defects and domains is the main mechanism of ferroelectric hardening.