In engineering, most structural components suffer from local damage in service or fabrication. Under different loading conditions, such local damage will further expand to large cracks and structural collapse. For ensuring longevity and reliability, the study of brittle crack growth is important. In structural damage prediction, various methodologies are commonly used and have demonstrated success in the past. Most traditional simulation methods, however, have the problems of large calculation quantities and long calculation times. At the same time, the mutual influence of crack opening stress and crack image on crack widening is rarely considered in existing works based on the deep learning framework. Herein, a deep learning model is developed to predict the spread of fracture of brittle materials under stress to improve the aforementioned problems. By developing a nonlinear relationship between crack stress and damage effects, this approach predicts crack growth, thus avoiding the loss of required information on crack growth behavior. The model emphasizes repeatability and extensibility, which is verified using extended finite element method (XFEM) Abaqus software through multiple sets of material crack data generated. This model's potential applications are material design and evaluation of residual life.