Osteocalcin (Ocn), also known as bone Gla protein, is synthesized by osteoblasts and thought to regulate energy metabolism, testosterone synthesis and brain development. However, its function in bone is not fully understood. Mice have three Ocn genes: Bglap, Bglap2 and Bglap3. Due to the long span of these genes in the mouse genome and the low expression of Bglap3 in bone, researchers commonly use Bglap and Bglap2 knockout mice to investigate the function of Ocn. However, it is unclear whether Bglap3 has any compensatory mechanisms when Bglap and Bglap2 are knocked out. Considering the controversy surrounding the role of Ocn in bone, we constructed an Ocn-deficient mouse model by knocking out all three genes (Ocn-/-) and analyzed bone quality by Raman spectroscopy (RS), Scanning electron microscopy (SEM), Fourier transform infrared spectroscopy (FTIR) and MicroCT (μCT). The RS test showed that the alignment of hydroxyapatite crystals and collagen fibers was significantly poorer in Ocn-/- mice than in wild-type (WT) mice. Ocn deficiency resulted in a looser surface structure of bone particles and a larger gap area proportion. FTIR analysis showed few differences in bone mineral index between WT and Ocn-/- mice, while μCT analysis showed no significant difference in cortical and trabecular regions. However, under tail-suspension simulating bone loss condition, the disorder of hydroxyapatite and collagen fiber alignment in Ocn-/- mice led to more obvious changes in bone mineral composition. Collectively, our results revealed that Ocn is necessary for regulating the alignment of minerals parallel to collagen fibrils.