There are over 150 tribal communities in South India. Many of them live in remote and forested areas. However, historically, they have had a close relationship with caste society. The dominance of caste is evident in their everyday contexts, economic relations and social status within the local social hierarchy. The article reflects on how the situation of tribes in South India cannot be fully comprehended without acknowledging the overbearing influence of caste ideology and politics on their lives. It will focus on two Scheduled Tribes in South India, the Paniyan and the Koraga, to illustrate practices of untouchability and social deprivation among them. Despite their Scheduled Tribe status, the Paniyan and the Koraga continue to face caste-based discrimination and cannot access basic services provided by the state. There is a deep sense of dispiritedness among them and, increasingly, they are isolating themselves from ‘others’ in their vicinity. Isolation has become an active mode of distancing from caste and asserting their tribal identity. It expresses their rejection of tribal integration, and associated politics, responsible for their marginalisation.