The accumulation of waterborne large woody debris is a critical issue facing bridges spanning active waterways. In addition to collision forces, drift buildup constricts flow, producing increased hydraulic pressures and exacerbating scour, the cause of nearly one-third of bridge failures. Accurate methods of predicting debris generation, transport, entrapment, and dimensions are consequently necessary to improve public safety, bridge designs, and informed decision-making. However, debris buildup characteristics are site-specific, varying with the regional vegetative, hydraulic, topographical, and geotechnical properties. As sufficient data on these properties is limited to create empirical predictive models, a probabilistic physics-based framework incorporating the risk assessment of local vegetation to estimate debris accumulation and scour is developed. Based on the upstream tree, soil, weather, and river conditions, fragility assessment of the riparian trees from landslides and windthrow is conducted to estimate the probability of debris generation and subsequent entrapment. The framework is demonstrated in a case study of a bridge in Vermont subject to Hurricane Irene. Sensitivity analysis is conducted to quantify the importance of various contributing factors, including slope, rainfall, wind gust, flow rate, and pier geometry. The experimental results highlight the importance of including the debris accumulation in the scour projections.