Acceleration of ultrathin foils by the laser radiation pressure promises a compact alternative to the conventional ion sources. Among the challenges on the way to practical realization, one fundamental is a strong transverse plasma instability, which develops density perturbations and breaks the acceleration. In this Letter, we develop a theoretical model supported by three-dimensional numerical simulations to explain the transverse instability growth from noise to wave breaking and its crucial effect on stopping the acceleration. The wave-broken nonlinear mode triggers rapid stochastic heating that finally explodes the target. Possible paths to mitigate this problem for getting efficient ion acceleration are discussed.