Accident reports provide information to understand why and how events occur. Learning from past accident reports is critical for preventing accidents or injuries in construction safety management. However, there are two issues: (1) manual analysis of such accident reports is time-consuming and labor-intensive; and (2) previous research mainly focused on analyzing the causal factors of accidents. Not much research concentrates on the injury effect in an accident and the influential relationship between accident cause and injury effect. To tackle this problem, a graph-based deep learning framework is proposed to identify accident-injury type and bodypart factors automatically to enable managers to make timely and better-informed decisions to prevent accidents and injuries for on-site safety. In this framework, a graph-based deep learning approach (specifically, the Graph Convolutional Network) is developed to automatically classify accident reports labeled with accident_type and injury_type, whereas the traversal method is developed to identify the bodypart factors. To further intuitively visualize these safety risk factors (e.g., accident_type, injury_type, and bodypart factors), the co-occurrence networks are drawn to further intuitively reveal the interdependency in accident-injury and injury-bodypart types respectively. From the perspective of theoretical and practical contributions, the framework proposed in this study not only represents a substantial data-driven advancement in construction accident report classification and keyword extraction tasks, but also enables managers to get knowledge of construction safety performance (i.e., accident causes and injury effects) and further formulate corresponding strategies to prevent accidents and injuries in on-site safety management.