E-health has the potential to promote health accessibility, performance and cost-saving. However, the adoption and penetration of e-health in underprivileged areas remains insufficient. We aim to investigate patients' and doctors' perception, acceptance, and utilization of e-health in a rural, spatially isolated and poverty-stricken county in southwestern China. A retrospective analysis based on a cross-sectional survey of patients and doctors in 2016 was conducted. Participants were recruited through convenience and purposive sampling, and questionnaires were self-designed and validated by investigators. The utilization, intention to use and preference of four e-health services were evaluated, including e-appointment, e-consultation, online drug purchase, and telemedicine. Predictors of utilization and intention to use e-health services were investigated by multivariable logistic regression. A total of 485 patients were included. The utilization rate of any type of e-health services was 29.9%, ranging from 6% in telemedicine to 18% in e-consultation. Additionally, 13.9%-30.3% of respondents as non-users revealed their willingness to use such services. Users and potential users of e-health services were inclined to specialized care from county, city or province hospitals, and they were most concerned with the quality, ease of use and price of e-health service. Patients' utilization and intention to use e-health could be associated with education and income level, cohabitants, working location, previous medical utilization, and access to digital device and internet. There remained 53.9%-78.3% of respondents reluctant to use e-health services, mainly due to perceived inability to use them. Of 212 doctors, 58% and 28% had provided online consultation and telemedicine before, and over 80% of county-hospital doctors (including actual providers) indicated their willingness to provide such services. Reliability, quality and ease of use were doctors' major concerns regarding e-health. Doctors' actual provision of e-health was predicted by their professional title, number of years in work, satisfaction with the wage incentive system, and self-rated health. Nevertheless, their willingness to adopt was only associated with the possession of smartphone. E-health is still in its infancy in western and rural China, where health resources are most scarce, and where e-health could prove most beneficial. Our study reveals the wide gaps between patients' low usage and their certain willingness to use e-health, as well as gaps between patients' moderate attention to use and physician's high preparedness to adopt e-health. Patients' and doctors' perceptions, needs, expectations, and concerns should be recognized and considered to promote the development of e-health in these underprivileged regions.