A quandle is an algebraic system originated in knot theory, and can be regarded as a generalization of symmetric spaces. The inner automorphism group of a quandle is defined as the group generated by the point symmetries (right multiplications). In this paper, starting from any simple graphs, we construct quandles whose inner automorphism groups are abelian. We also prove that the constructed quandle is homogeneous if and only if the graph is vertex-transitive. This shows that there is a wide family of quandles with abelian inner automorphism groups, even if we impose the homogeneity. The key examples of such quandles are realized as subquandles of oriented real Grassmannian manifolds.