Primary care physicians (PCPs) suffered from heavy workloads and health problems during COVID-19 pandemics, and building their confidence in pandemic response has great potential to improve their well-being and work performance. We identified the organizational factors associated with their confidence in pandemic response and proposed potential management levers to guide primary care response for the pandemic. We conducted a cross-sectional survey with 224 PCPs working in 38 community health centers in China. Guided by self-efficacy theory, organization-level factors (organizational structure and organizational culture) and physician-level factors (job skill variety, perceived organizational support, work-family conflict, and professional fulfillment) were selected, and two-level ordinal logit models were built to examine their association with PCPs' confidence in pandemic response. We found that hierarchical culture (OR = 3.51, P<0.05), perceived organizational support (OR = 2.36, P<0.05), job skill variety (OR = 1.86, P<0.05), and professional fulfillment (OR = 2.26, P<0.05) were positively associated with PCPs' confidence in pandemic response. However, the influence of organization structure and work-family conflict seemed limited. The study not only increases our understanding of the influence of organizational context on PCPs' pandemic response confidence, but also points out potential management levers for front-line primary care managers to enhance primary care pandemic response capacity.