“Human-elephant conflict (HEC)”, the alarming issue, in present day context has attracted the attention of environmentalists and policy makers. The rising conflict between human beings and wild elephants is common in Buxa Tiger Reserve (BTR) and its adjoining area in West Bengal State, India, making the area volatile. People’s attitudes towards elephant conservation activity are very crucial to get rid of HEC, because people’s proximity with wild elephants’ habitat can trigger the occurrence of HEC. The aim of this study is to conduct an in-depth investigation about the association of people’s attitudes towards HEC with their locational, demographic, and socio-economic characteristics in BTR and its adjoining area by using Pearson’s bivariate chi-square test and binary logistic regression analysis. BTR is one of the constituent parts of Eastern Doors Elephant Reserve (EDER). We interviewed 500 respondents to understand their perceptions to HEC and investigated their locational, demographic, and socio-economic characteristics including location of village, gender, age, ethnicity, religion, caste, poverty level, education level, primary occupation, secondary occupation, household type, and source of firewood. The results indicate that respondents who are living in enclave forest villages (EFVs), peripheral forest villages (PFVs), corridor village (CVs), or forest and corridor villages (FCVs), mainly males, at the age of 18–48 years old, engaged with agriculture occupation, and living in kancha and mixed houses, have more likelihood to witness HEC. Besides, respondents who are illiterate or at primary education level are more likely to regard elephant as a main problematic animal around their villages and refuse to participate in elephant conservation activity. For the sake of a sustainable environment for both human beings and wildlife, people’s attitudes towards elephants must be friendly in a more prudent way, so that the two communities can live in harmony.