The microstructural properties of fracture surfaces are a remarkable reflection of the nonlinear mechanical properties of rocks, which have not been thoroughly evaluated to date. In order to understand the contributions of microstructure size and packing to the nonlinear mechanical response, both micro- (field emission scanning electron microscopy (SEM) experiments on failure specimen section) and macro-scale (unequal amplitude cycle experiment) tests were carried out on different marbles. The size and arrangement properties of microstructures on the fracture surface were determined using digital image recognition technology based on micro-scale experiments. Remarkable differences were observed among different types of marble in terms of breakage evolution, fracture combining form, and nonlinear evolution of energy. The micro-deformation of marbles is characterized by the deformation distribution and microstructure inhomogeneity in strong deformation zones of failure. The microstructures of brittle-ruptured marbles exhibit mechanical fragmentation and fracture of mineral grains. The ductile deformation of marble is characterized by the presence of solid flow traces within the mineral crystals, as well as the loose arrangement of mineral crystal grains in the strongly deformed zones. The constitutive model for marble samples with fine mineral crystals can be derived by nonlinearly separating the Helmholtz free energy during the breakage process and defining the relationship between breakage force and dissipated energy in the absence of residual strength after fracture. The obtained nonlinear breakage constitutive equation is in good agreement with the experimental results. These findings may provide basis and inspiration for engineering construction and geological disaster prevention.