Rough set theory and vague set theory are powerful tools for managing uncertain, incomplete and imprecise information. This paper extends the rough vague set model based on equivalence relations and the rough fuzzy set model based on fuzzy relations to vague sets. We mainly focus on the lower and upper approximation operators of vague sets based on vague relations, and investigate the basic properties of approximation operators on vague sets. Specially, we give some essential characterizations of the lower and upper approximation operators generated by reflexive, symmetric, and transitive vague relations. Finally, we structure a parameterized roughness measure of vague sets and similarity measure methods between two rough vague sets, and obtain some properties of the roughness measure and similarity measures. We also give some valuable counterexamples and point out some false properties of the roughness measure in the paper of Wang et al.