Context. Many star forming regions are known to have associated dispersed populations, whose members are located away from known current star forming sites. Their origin is unclear, and the identification of the members through relatively short-lived signatures of youth can miss them. Aims. We aim at confirming membership of a sample of cool stars identified in a previous work (Comer´ on et al. 2009, A&A 500, 1045) in the Lupus 1, 3 and 4 clouds as candidate members. Most of them do not display near- or mid-infrared excess or other easily recognizable signatures of youth. Methods. We use low-resolution spectroscopy in the red, including the H region, to accurately determine spectral types and probe surface gravity-sensitive features that provide reliable discrimination criteria among cool giant stars, young stellar objects, and evolved dwarf stars. Results. Most of the candidate members of a possible dispersed population around Lupus 1 are found to be background K or early M giants. However, about half of the observed members of Lupus 3 are confirmed as young objects, including both low-mass stars and brown dwarfs. The distributed population is compared to the sample of lightly obscured members projected closer to the densest parts of the Lupus 3 star forming cloud, and the estimated ages of the members of both sets are found to be consistent with a single distribution. However, we find statistical indications (although at a low significance level) of a decrease in the frequency of infrared excesses in the distributed population. Some non-giant members are also identified with gravity-sensitive features typical of more evolved stars, and we argue that these may belong to an older population associated with the Gould Belt, similar to that observed in the direction of other nearby star forming regions. We also confirm two additional very low-mass members of Lupus 4. Conclusions. Although some of its members were already known previous to this work, our results emphasize the richness of the low-mass distributed population around Lupus 3 and the existence of much less numerous dispersed populations around Lupus 1 and Lupus 4. The apparent spatial segregation as a function of the abundance of circumstellar material favors dynamical ejection from the main star forming cloud as the mechanism giving rise to the dispersed population.