In tensor network representation, the partition function of a generalized two-dimensional XY spin model with topological integer and half-integer vortex excitations is mapped to a tensor product of one-dimensional quantum transfer operator, whose eigenequation can be solved by an algorithm of variational uniform matrix product states. Using the singularities of the entanglement entropy, we accurately determine the complete phase diagram of this model. Both the integer vortex-antivortex binding and half-integer vortex-antivortex binding phases are separated from the disordered phase by the usual Berezinskii-Kosterlitz-Thouless (BKT) transitions, while a continuous topological phase transition exists between two different vortex binding phases, exhibiting a logarithmic divergence of the specific heat and exponential divergence of the spin correlation length. A hybrid BKT and Ising universality class of topological phase transition is thus established. We further prove that three phase transition lines meet at a multicritical point from which a deconfinement crossover line extends into the disordered phase.