Problem, research strategy, and findings One promising policy approach to addressing housing needs is liberalizing accessory dwelling unit (ADU) development. Yet, understanding the impact of such policy efforts is fundamentally constrained by the inability to quantify and characterize unpermitted ADUs, which may expose homeowners and tenants to legal, financial, and safety risks and confound policy evaluations. We addressed this gap by leveraging computer vision and human annotations to estimate the population of detached ADU constructions in San José (CA). Our contributions are threefold: 1) We estimated the proportion of unpermitted ADU constructions from 2016 to 2020; 2) we describe the demographic, housing market, and parcel characteristics associated with these informal ADUs; and 3) we provide a data set of labeled small buildings, excluding unpermitted detections, for further research. We found that informal ADU construction was substantial—approximately three to four informal units for every formal unit—and more likely in more diverse, dense, and overcrowded neighborhoods. Though our study was limited to analyzing detached ADUs during one time period, we set the stage for further investigations of informal housing across different typologies and over time. Takeaway for practice Our approach demonstrates the promise of computer vision and human annotations to enable more robust, comprehensive, and reliable understanding of actual—not just permitted—housing units. We urge planners and other policymakers to consider the growth patterns of unpermitted ADUs to more optimally and equitably address housing needs.