In this paper we initiate a study of the topological group $PPQI(G,H)$ of pattern-preserving quasi-isometries for $G$ a hyperbolic Poincare duality group and $H$ an infinite quasiconvex subgroup of infinite index in $G$. Suppose $\partial G$ admits a visual metric $d$ with $dim_H < dim_t +2$, where $dim_H$ is the Hausdorff dimension and $dim_t$ is the topological dimension of $(\partial G,d)$. a) If $Q_u$ is a group of pattern-preserving uniform quasi-isometries (or more generally any locally compact group of pattern-preserving quasi-isometries) containing $G$, then $G$ is of finite index in $Q_u$. b) If instead, $H$ is a codimension one filling subgroup, and $Q$ is any group of pattern-preserving quasi-isometries containing $G$, then $G$ is of finite index in $Q$. Moreover, (Topological Pattern Rigidity) if $L$ is the limit set of $H$, $\LL$ is the collection of translates of $L$ under $G$, and $Q$ is any pattern-preserving group of {\it homeomorphisms} of $\partial G$ preserving $\LL$ and containing $G$, then the index of $G$ in $Q$ is finite. We find analogous results in the realm of relative hyperbolicity, regarding an equivariant collection of horoballs as a symmetric pattern in a {\it hyperbolic} (not relatively hyperbolic) space. Combining our main result with a theorem of Mosher-Sageev-Whyte, we obtain QI rigidity results. An important ingredient of the proof is a version of the Hilbert-Smith conjecture for certain metric measure spaces, which uses the full strength of Yang's theorem on actions of the p-adic integers on homology manifolds. This might be of independent interest.