An unprecedented Household Solid Waste (HSW) separation program was launched in 46 cities and some pilot rural areas in China in 2019. This study examines the antecedents of waste separation behavior using the extended theory of planned behavior (TPB) to identify beliefs that are useful targets for interventions, and compares the urban-rural differences. Based on interviews with rural and urban residents, we identified residents’ beliefs influencing waste separation behaviors, including four behavioral, seven normative, and eight control beliefs. Then, we tested the conceptual model with a two-stage questionnaire survey using a sample of 604 residents from urban and rural areas. Evidence from structural equation modeling supports the extended TPB in that it predicts waste separation intention and behavior on the whole. Moral norm and self-identity were found to independently predict intention and may prove a useful addition to the TPB; however, the path from attitude to intention is not supported in both urban and rural models. Moreover, the specific difference of belief between urban and rural residents was examined. The behavioral beliefs associated to money, including earning, free waste bin or bag, and fine, are significant only in the rural group. The normative beliefs of family, Chinese Communist Party (CCP) members, and government can affect residents’ subjective norm in both urban and rural, while the beliefs of relatives, friends, and neighbors are significant only in the rural group. Implications for managers of operating waste separation programs were discussed. Further investigation into the potential of the model to intervene in waste separation behavior is required.