In this paper we intend to discuss in detail the question of an instability process of modulated-strain structures in alloys that undergo a first-order phase transformation. In addition, we aim at characterizing the formation of nonlinear structures in the pretransformation regime produced by the instability mechanism. The model is based on a two-dimensional lattice including nonlinear and competing interactions, which play a key role in the instability of a homogeneous solution. Although the main body of the study is devoted to the nonlinear dynamics of a lattice model, an excursion in linear analysis provides us with necessary results about the critical behavior of the system. Indeed, the linear problem leads us to the study of the phonon-dispersion branch, and the existence of a critical point of the phonon-dispersion curve is then shown for a particular value of the elastic coefficient (here, the control parameter of the phase transition). This critical behavior is, in fact, related to the softening of the dispersion curve at a nonzero wave number. The nonlinear analysis becomes essential when the system is linearly unstable in the vicinity of the critical point. An amplitude equation of the Ginzburg-Landau type is next deduced in a semidiscrete approach by using a multiple-scale technique. The examination of the stability of steady solutions allows one to determine the nature of the bifurcation near the critical point. The study of the bifurcating stationary solutions shows an instability process for long-wavelength modulations taking place in the transverse direction of the two-dimensional system. The mechanism of self-generated nonlinear structures in the two-dimensional lattice near the critical region of the phonon dispersion is numerically investigated. Nontrivial localized structures are then demonstrated. By way of conclusion, some emphasis is placed on the pretransformation phenomena in martensitic materials or ferroelastic crystals.