Ultra-dense small cell networks represent a key future network solution that can help meet the exponentially rising traffic requirements of modern wireless networks. Backhauling these small cells are an emerging challenge to the extent that various cells are likely to have different backhaul constraints. The user-centric backhaul scheme has been proposed in the literature to jointly exploit the diversity in users’ requirement and backhaul constraints. In this paper, we propose a novel scheme, termed the memory-based hybrid scheme, which additionally also exploits the predictability in a user’s mobility. We compare the novel scheme to two variants of memory-less user-centric backhaul implementations and show significant gains in convergence time (15%), user-centric KPIs (51% and 82%) at the negligible cost 2% loss in cumulative throughput. The novel scheme requires additional memory in user-devices to store learned values, which is nonetheless well justified in view of the considerable gains achieved.