We propose that the topological phase transitions can occur in the square-octagon lattice, when the Rashba spin-orbit coupling and the staggered potential are taken into account. By means of the Fukui-Hatsugai method, topological invariant Z2 is evaluated and the phase diagram is presented as a function of the Rashba spin-orbit coupling and the staggered potential. We find that the square-octagon lattice can support different phases, such as semimetal, normal insulator and topological insulator. Particularly, the competition between the Rashba spin-orbit coupling and staggered potential can lead to the topological phase transition for all the filling fractions. To clearly support the identification of topological nontrivial phase, helical edge states are given numerically, the number of which is demonstrated by the spin chern number. Finally, we analyze and discuss the density distribution and spin polarization to further confirm the topological properties of the system.