Pedestrians and bicyclists from marginalized and underserved populations experienced disproportionate fatalities and injury rates due to traffic crashes in the US. This disparity among road users of different races and the increasing trend of traffic risk for underserved racial groups called for an urgent agenda for transportation policy making and research to ensure equity in roadway safety. Pedestrian and bicyclist crashes involved drivers and pedestrians/bicyclists; the latter were usually victims. Traditional safety studies did not account for the interaction between the two parties and assumed that they were independent from each other. In this study we paired the driver and pedestrian/bicyclist involved in the same crash to understand the socioeconomic and demographic make-up of the two parties involved in crashes and assessed the geographic distribution of these crashes and crash-contributing factors. For this purpose, we applied thelatent class clustering analysis (LCA) to classify different crash types and analyze the patterns of the crashes based on the income and ethnicity of both drivers and victims involved in pedestrian and bicyclist crashes. We then used random forest algorithms and partial dependence plots (PDPs) to model and interpreted the contributing factors of the clusters in both pedestrian and bicyclist models. The clustering results showed a pattern of social segregation in pedestrian and bicyclist crashes that drivers and victims with similar socioeconomic characteristics tend to be involved in one crash. Pedestrian/bicyclist exposure, driver’s age, victim’s age, year of the car in use, annual average daily traffic (AADT), speed limit, roadbed width, and lane width were the most influential factors contributing to this pattern. Crashes that involved drivers and victims with lower income and non-white ethnicity tended to happen in the location with higher pedestrian/bicyclist exposure, higher speed limit, and wider road. The findings of this research can help to inform the decision-making process for improving safety to ensure equitable and sustainable safety for all road users and communities.