Autonomous Vehicles (AVs) have the potential to revolutionize transportation systems by enhancing traffic safety. Safety testing is undoubtedly a critical step for enabling large-scale deployment of AVs. High-risk scenarios are particularly important as they pose significant challenges and provide valuable insights into the driving capabilities of AVs. This study presents a novel approach to assess the safety of AVs using in-depth crash data, with a particular focus on real-world crash scenarios. First, based on the high-definition video recording of the whole process prior to the crash occurrences, 453 real-world crashes involving 596 passenger cars from China In-depth Mobility Safety Study-Traffic Accident (CIMSS-TA) database were reconstructed. Pertinent static and dynamic elements needed for the construction of the testing scenarios were extracted. Subsequently, 596 testing scenarios were created via each passenger car’s perspective within the simulation platform. Following this, each of the crash-involved passenger cars was replaced with Baidu Apollo, a famous automated driving system (ADS), for counterfactual simulation. Lastly, the safety performance of the AV was assessed using the simulation results. A logit model was utilized to identify the fifteen crucial scenario elements that have significant impacts on the test results. The findings demonstrated that the AV could avoid 363 real-world crashes, accounting for approximately 60.91% of the total, and effectively mitigated injuries in the remaining 233 unavoidable scenarios compared to a human driver. Moreover, the AV maintain a smoother speed in most of the scenarios. The common feature of these unavoidable scenarios is that the AV is in a passive state, and the crashes are not caused by the AV violating traffic rules, but rather caused by abnormal behavior exhibited by the human drivers. Additionally, seven specific scenarios have been identified wherein AVs are unable to avoid a crash. These findings demonstrate that, compared to human drivers, AVs can avoid crashes that are difficult for humans to avoid, thereby enhancing traffic safety.