This paper aims to address the low-temperature dynamics issue for the $p=2$ spin dynamics with confining potential, focusing especially on quartic and sextic cases. The dynamics are described by a Langevin equation for a real vector $q_i$ of size $N$, where disorder is materialized by a Wigner matrix and we especially investigate the self consistent evolution equation for effective potential arising from self averaging of the square length $a(t)\equiv \sum_i q_i^2(t)/N$ for large $N$. We first focus on the static case, assuming the system reached some equilibrium point, and we then investigate the way the system reach this point dynamically. This allows to identify a critical temperature, above which the relaxation toward equilibrium follows an exponential law but below which it has infinite time life and corresponds to a power law decay.