More than 300 supermassive black holes have been detected at redshifts larger than six, and they are abundant in the centers of local galaxies. Their formation mechanisms, however, are still rather unconstrained. A possible origin of these supermassive black holes could be mergers in dense black hole clusters, forming as a result of mass segregation within nuclear star clusters at the center of galaxies. In this study, we present the first systematic investigation of the evolution of such black hole clusters in which the effect of an external potential is taken into account. Such a potential could be the result of gas inflows into the central region; for example, as a result of galaxy mergers. We show here that the efficiency of the formation of a massive central object is mostly regulated by the ratio of cluster velocity dispersion divided by the speed of light, potentially reaching efficiencies of 0.05–0.08 in realistic systems. Our results show that this scenario is potentially feasible and may provide black hole seeds of at least 103 M⊙. We conclude that the formation of seed black holes via this channel should be taken into account in statistical assessments of the black hole population.