Models of universal quantum computation in which the required interactions between register (computational) qubits are mediated by some ancillary system are highly relevant to experimental realizations of a quantum computer. We introduce such a universal model that employs a $d$-dimensional ancillary qudit. The ancilla-register interactions take the form of controlled displacements operators, with a displacement operator defined on the periodic and discrete lattice phase space of a qudit. We show that these interactions can implement controlled phase gates on the register by utilizing geometric phases that are created when closed loops are traversed in this phase space. The extra degrees of freedom of the ancilla can be harnessed to reduce the number of operations required for certain gate sequences. In particular, we see that the computational advantages of the quantum bus (qubus) architecture, which employs a field-mode ancilla, are also applicable to this model. We then explore an alternative ancilla-mediated model which employs a spin ensemble as the ancillary system and again the interactions with the register qubits are via controlled displacement operators, with a displacement operator defined on the Bloch sphere phase space of the spin coherent states of the ensemble. We discuss the computational advantages of this model and its relationship with the qubus architecture.