X chromosome inactivation triggers a dramatic reprogramming of transcription and chromosome architecture. However, how the chromatin organization of inactive X chromosome is established de novo in vivo remains elusive. Here, we identified an Xist-separated megadomain structure (X-megadomains) on the inactive X chromosome in mouse extraembryonic lineages and extraembryonic endoderm (XEN) cell lines, and transiently in the embryonic lineages, before Dxz4-delineated megadomain formation at later stages in a strain-specific manner. X-megadomain boundary coincides with strong enhancer activities and cohesin binding in an Xist regulatory region required for proper Xist activation in early embryos. Xist regulatory region disruption or cohesin degradation impaired X-megadomains in extraembryonic endoderm cells and caused ectopic activation of regulatory elements and genes near Xist, indicating that cohesin loading at regulatory elements promotes X-megadomains and confines local gene activities. These data reveal stepwise X chromosome folding and transcriptional regulation to achieve both essential gene activation and global silencing during the early stages of X chromosome inactivation.