The SYK model has a wormhole-like solution after averaging over the fermionic couplings in the nearly $AdS_2$ space. Even when the couplings are fixed the contribution of these wormholes continues to exist and new saddle points appear which are interpreted as "half-wormholes". In this paper, we will study the fate of these wormholes in a model without quenched disorder namely a tensor model with $O(N)^{q-1}$ gauge symmetry whose correlation function and thermodynamics in the large $N$ limit are the same as that of the SYK model. We will restate the factorization problem linked with the wormhole threaded Wilson operator, in terms of global charges or non-trivial cobordism classes associated with disconnected wormholes. Therefore for the partition function to factorize especially at short distances, there must exist certain topological defects which break the global symmetry associated with wormholes and make the theory devoid of global symmetries. We will interpret these wormholes with added topological defects as our "half-wormholes". We will also comment on the late time behavior of the spectral form factor, particularly its leading and sub-leading order contributions coming from higher genus wormholes in the gravitational sector. Finally we will show how, the other non-trivial saddles from "half-wormhole" dominate and give rise to unusual thermodynamics in the bulk sector due to non-perturbative effects.