Venous malformation (VM) is a type of vascular morphogenic defect in humans with an incidence of 1%. Although gene mutation is considered as the most common cause of VM, the pathogenesis of those without gene mutation remains to be elucidated. Here, we aimed to explore the relation of bone morphogenetic protein 9 (BMP9) and development of VM. At first, we found serum and tissue BMP9 expression in VM patients was significantly lower than that in healthy subjects, detected via enzyme-linked immunosorbent assay. Next, with wound healing assay, transwell assay and tube formation assay, we discovered BMP9 could inhibit migration and enhance tube formation activity of human umbilical vein endothelial cells (HUVECs) via receptor activin receptor-like kinase 1 (ALK1). Besides, BMP9 improved the expression of structural proteins alpha-smooth muscle actin (α-SMA) and Desmin in human umbilical vein smooth muscle cells (HUVSMCs) via activation of the SMAD1/5-ID1 pathway, determined by RNA-based next-generation sequencing, qPCR, immunofluorescence and western blotting. Intriguingly, this effect could be blocked by receptor ALK1 inhibitor, SMAD1/5 inhibitor and siRNAs targeting ID1, verifying the BMP9/ALK1/SMAD1/5/ID1/α-SMA pathway. Meanwhile, knocking out BMP9 in C57BL/6 mice embryo led to α-SMA scarcity in walls of lung and mesenteric vessels, as well as walls of small trachea. BMP9−/− zebrafish also exhibited abnormal vascular maturity, indicating a critical role of BMP9 in vascular maturity and remodeling. Finally, a VM mice model revealed that BMP9 might have therapeutic effect in VM progression. Our study discovered that BMP9 might inhibit the occurrence of VM by strengthening the vessel wall and maintaining endothelium quiescence. These findings provide promising evidences of new therapeutic targets that might be used for the management of VM.