Chronic diseases are the leading causes of disability and premature death among the elderly population in India. The study, using data from the 75th round of the NSSO survey (N = 44,631), examined the prevalence and determinants of chronic diseases among the population aged 60+ in India by applying bivariate and logistic regression analyses and used a non-linear decomposition technique to understand the urban-rural differences in the prevalence of chronic diseases. About 21% of the elderly in India reportedly have at least one chronic disease. Seventeen percent elderly in rural areas and 29% in urban areas suffer from a chronic disease. Hypertension and diabetes account for about 68% of all chronic diseases. The prevalence of chronic diseases is the highest in Kerala (54%), followed by Andhra Pradesh (43), West Bengal (36), and Goa (32). Those with higher levels of education, staying in urban areas, those who are economically dependent on others, staying alone or without spouse and children, and belonging to wealthy households have a higher likelihood of having a chronic disease. The probability of having a chronic disease is 1.15 times higher among urban residents as compared to their rural counterparts. Elderly rural women, compared to elderly rural men, and never-married, widowed, and divorced elderly urban women, compared to married elderly urban men, are significantly more likely to suffer from chronic ailments. Differences in education, wealth status, and caste are the three most significant contributors to the urban-rural gap in chronic diseases. The high risk of chronic diseases among certain subsets of the elderly population must be recognized as a key public health concern. The findings of our study will likely help promote healthy ageing in India.