The task of event relation extraction (ERE) aims to organize multiple events and their relations as a directed graph. However, existing ERE methods exhibit two limitations: (1) Events in a document are typically expressed with merely a trigger word or phrase neglecting the rich semantic structure of event arguments and roles. (2) Various event relations such as coreference, temporal, causal, and subevent relations are correlated with each other, while little work has explored the benefits of extracting all relations in a joint manner. In this work, we investigate an event-relation graph propagation network and a novel document-level semantically-enriched event representation for joint ERE. First, we address the lack of event argument annotations by extracting cross-sentence implicit arguments based on the AMR graph. Then triggers and argument roles are aggregated with a structure-aware encoder. Second, based on the rich event information, event relation interactions are incorporated with a subgraph propagation and aggregation mechanism. During training, we further develop a novel triadic contrastive loss to capture high-order event pair relationships. We conduct experiments based on the public MAVEN-ERE benchmark and the results show that our model achieves 60.7%, 37.4%, 32.9% F1-scores on temporal, causal and subevent relations, and 86.1% MUC-score on coreferences, outperforming the current joint models by a large margin. Further in-depth analysis shows the effectiveness of our model in capturing event-event dependencies in document context. The proposed model can be used for event graph construction and storyline understanding.