One fundamental challenge in diffusion magnetic resonance imaging (dMRI) harmonization is to disentangle the contributions of scanner-related effects from the variable brain anatomy for the observed imaging signals. Conventional harmonization methods rely on establishing an atlas space to resolve anatomical variability and generate a unified inter-site mapping function. However, this approach is limited in accounting for the misalignment of neuroanatomy that still widely persists even after registration, especially in regions close to cortical boundaries. To overcome this challenge, we propose a personalized framework in this paper to more effectively address the confounding from the misalignment of neuroanatomy in dMRI harmonization. Instead of using a common template representing site-effects for all subjects, the main novelty of our method is the adaptive computation of personalized templates for both source and target scanning sites to estimate the inter-site mapping function. We integrate our method with the rotation invariant spherical harmonics (RISH) features to achieve the harmonization of dMRI signals between sites. In our experiments, the proposed approach is applied to harmonize the dMRI data acquired from two scanning platforms: Siemens Prisma and GE MR750 from the Adolescent Brain Cognitive Development dataset and compared with a state-of-the-art method based on RISH features. Our results indicate that the proposed harmonization framework achieves superior performance not only in reducing inter-site variations due to scanner differences but also in preserving sex-related biological variability in original cohorts. Moreover, we assess the impact of harmonization on the estimation of fiber orientation distributions and show the robustness of the personalized harmonization procedure in preserving the fiber orientation of original dMRI signals.