Bed load particles in bedrock streams receiving lateral input from hillslopes may or may not show a clear, monotonic pattern of size reduction in the downstream direction. Both abrasion and selective sorting may play important roles in generating downstream fining. The objective of this study is to develop a physically based model of downstream fining in bedrock streams with lateral input based on both processes. A surface‐based gravel transport relation for the size mixture is employed to account for the effect of selective sorting (differential transport). While the model produces silt and sand by abrasion, it is also assumed to loosely capture particle fracturing via a lumped abrasion coefficient embodied in Sternberg's law. The model is here tested against field data from Vieux Habitants River in Guadeloupe Island, which is located in the Caribbean Sea. The river shows clear downstream fining, and this pattern is captured reasonably well by the model. The model results indicate that abrasion (including fracturing) is solely responsible for the downstream fining pattern for most of the study reach of the Vieux Habitants River. Sensitivity analysis of the model has also been performed. The model results suggest that, in general, selective sorting by differential transport can play a role in downstream fining only in cases of streams with relatively fine gravel sizes and lower slopes. The results also indicate that abrasion (including fracturing) and selective sorting can be equally important for downstream fining in bedrock rivers. The results also suggest that future work should explicitly consider fracturing (comminution) separately from abrasion (wear) rather than lumping them in a Sternberg‐type coefficient.