Information about rapidly changing slum areas may support the development of appropriate interventions by concerned authorities. Often, however, traditional data collection methods lack information on the spatial distribution of slum-dwellers. Remote sensing based methods could be used for a rapid inventory of the location and physical composition of slums. (Semi-)automatic detection of slums in image data is challenging, owing to the high variability in appearance and definitions across different contexts. This paper develops an ontological framework to conceptualize slums using input from 50 domain-experts covering 16 different countries. This generic slum ontology (GSO) comprises concepts identified at three levels that refer to the morphology of the built environment: the environs level, the settlement level and the object level. It serves as a comprehensive basis for image-based classification of slums, in particular, using object-oriented image analysis (OOA) techniques. This is demonstrated by with an example of local adaptation of GSO and OOA parameterization for a study area in Kisumu, Kenya. At the object level, building and road characteristics are major components of the ontology. At the settlement level, texture measures can be potentially used to represent the contrast between planned and unplanned settlements. At the environs level, factors which extend beyond the site itself are important indicators, e.g. hazards due to floods plains and marshy conditions. The GSO provides a comprehensive framework that includes all potentially relevant indicators that can be used for image-based slum identification. These characteristics may be different for other study areas, but show the applicability of the developed framework.