Spontaneous polarization (Ps) is the fundamental for ferroelectrics. Compared with the relative explicit origin of Ps of perovskite ferroelectrics and bismuth layer structured ferroelectrics, that of the perovskite layer structured ferroelectrics (PLSFs) is still unclear and controversial due to their unique layer structures. Sr2Nb2O7 compound with a high Curie temperature of 1342 °C is a typical representative in the family of PLSFs. Here, we report our investigations about the phase transitions and the mechanism of ferroelectricity of Sr2Nb2O7 ceramics from the viewpoint of layered crystal structure experimentally and theoretically. Sr2Nb2O7 has two phase transitions at about 215 °C and 1340 °C. Selected area electron diffraction (SAED) images show superlattice along the [001] zone axis related to an incommensurate modulation and confirm it's incommensurate-commensurate phase transition at around 215 °C. The latter on at 1340 °C corresponds to the paraelectric-ferroelectric phase transition which is originated from oxygen octahedral rotation along c-axis, named a0b0c+ in Glazer's notation. Phonon spectra shows the origin of Ps for Sr2Nb2O7 is from mainly the oxygen octahedral rotations and partially the displacement of Sr2+ ions. This study not only demonstrates the rotation of oxygen octahedron is the domination that affects ferroelectric polarization but also opens up a new direction to design high ferroelectric and piezoelectric PLSF ceramics.