The polymer quantization of cosmological backgrounds provides an alternative path to the original Wheeler-de Witt (WdW) quantum cosmology, based on a different representation the commutation relations of the canonical variables. This polymer representation allows to capture the lattice like structure of the quantum geometry and leads to a radically different quantum cosmology compared to the WdW construction. This new quantization scheme has attracted considerable attention due to the singularity resolution it allows in a wide class of symmetry reduced gravitational systems, in particular where the WdW scheme fails. However, as any canonical quantization scheme, ambiguities in the construction of the quantum theory, being regularization or factor-ordering ones, can drastically modify the resulting quantum dynamics. In this work, we propose a new criteria to restrict the quantization ambiguities in the simplest model of polymer quantum cosmology, for homogeneous and isotropic General Relativity minimally coupled to a massless scalar field. This new criteria is based on an underlying SL(2,ℝ) structure present in the phase space of this simple cosmological model. By preserving the symmetry of this cosmological system under this 1d conformal group, we derive a new regularization of the phase space. We perform both its polymer quantization and a quantization scheme directly providing a representation of the SL(2,ℝ) group action. The resulting quantum cosmology can be viewed as a lattice-like quantum mechanics with an SL(2,ℝ) invariance. This provides a new version of Loop Quantum Cosmology consistent with the conformal symmetry. This alternative construction opens new directions, among which a possible mapping with the conformal quantum mechanics as well as with recent matrix or tensor models constructions for quantum cosmological space-time.