Affective factors have been shown to play a crucial role in second language acquisition. However, they seem to be overlooked by most English as a foreign language teachers during classroom practices. Teachers’ oral English is an essential input for EFL students because they are rarely able to use and practice English outside classrooms. Compared with majors, non-English major students can have lower levels of motivation, which may lead to their lack of interest during lessons. Hence, the enhancement of English teaching efficiency in non-major EFL classes seems to be harder. This qualitative study investigates how affective factors are perceived to influence oral English communication in EFL classes where English is taught as a non-major subject at two public colleges in Vietnam. Data were collected via individual interviews with lecturers and focus group interviews with students. Findings reveal that symmetrical relationship, strong sense of belonging, and deep concern and understanding are the three affective factors that support EFL learning. Among these factors, concern and understanding about students influence both lecturers’ pedagogical decisions and students’ learning motivation. Therefore, EFL teachers are suggested to nurture a strongly emotional classroom culture to lower students’ English anxiety and enhance their learning.