Anthropogenic alteration of physical habitat structure in streams and rivers is increasingly recognized as a major cause of impairment worldwide. As part of their assessment of the status and trends in the condition of rivers and streams in the U.S., the U.S. Environmental Protection Agency’s (USEPA) National Aquatic Resource Surveys (NARS) quantify and monitor channel size and slope, substrate size and stability, instream habitat complexity and cover, riparian vegetation cover and structure, anthropogenic disturbance activities, and channel-riparian interaction. Like biological assemblages and water chemistry, physical habitat is strongly controlled by natural geoclimatic factors that can obscure or amplify the influence of human activities. We developed a systematic approach to estimate the deviation of observed river and stream physical habitat from that expected in least-disturbed reference conditions. We applied this approach to calculate indices of anthropogenic alteration of three aspects of physical habitat condition in the conterminous U.S. (CONUS): streambed sediment size and stability, riparian vegetation cover, and instream habitat complexity. The precision and responsiveness of these indices led the USEPA to use them to evaluate physical habitat condition in CONUS rivers and streams. The scores of these indices systematically decreased with greater anthropogenic disturbance at river and stream sites in the CONUS and within ecoregions, which we interpret as a response of these physical habitat indices to anthropogenic influences. Although anthropogenic activities negatively influenced all three physical habitat indices in the least-disturbed sites within most ecoregions, natural geoclimatic and geomorphic factors were the dominant influences. For sites over the full range of anthropogenic disturbance, analyses of observed/expected sediment characteristics showed augmented flood flows and basin and riparian agriculture to be the leading predictors of streambed instability and excess fine sediments. Similarly, basin and riparian agriculture and non-agricultural riparian land uses were the leading predictors of reduced riparian vegetation cover complexity in the CONUS and within ecoregions. In turn, these reductions in riparian vegetation cover and complexity, combined with reduced summer low flows, were the leading predictors of instream habitat simplification. We conclude that quantitative measures of physical habitat structure are useful and important indicators of the impacts of human activities on stream and river condition.