This paper aims to discuss the fragilities of hierarchical structures of Chinese financial market, and find out the possible faults. We propose an analysis approach to explore the structure characteristics of financial market based on the manifold structure of financial data. First, we extract the underlying manifold embedded in financial time series data by manifold learning, which governs the dynamics of financial system. Second, the surface curvature of manifold is used to serve the quantitative analysis of manifold structure, in which the rate of curvature change is employed to measure the structure fragility. Finally, the dynamics of manifold structures are discussed by Lyapunov exponents, which further explore the fragilities of markets and confirm the conclusions of curvature analysis. In empirical research, we select CSI 300 index as the overall trend indicator of China's financial market, and nine industry indexes as hierarchical market indicators. Our research results indicate that Chinese financial market has less chaotic than the real economy industries, in spite of its high fragility. Some real economic industries are most likely to crash first in the event of a crisis. The findings contribute to present the states of financial markets and provide decision support for investors.