A theoretical study of patterns of the evolution and formation of dislocation structures during the plastic deformation of crystals is carried out. A nonlinear theory of the formation of cellular dislocation structures in an ensemble of screw dislocations has been developed. The nonlinear dynamics of an ensemble of dislocations is investigated in a two-dimensional domain, taking into account periodic boundary conditions of the Born - Karman type imposed on the initial equation. The local kinetics of dislocations is chosen in the form of multiplication of dislocations by means of their double transverse sliding and annihilation. A homogeneous stationary solution of the system (thermodynamic branch) is found. It is established that at a critical deviation from the thermodynamic branch, an instability of the homogeneous state occurs in the system due to the correlation interaction of dislocations. To obtain solutions in the domain of instability, the system of evolutionary equations is transformed to a system of equations for collective (mode ) variables. The expediency of such transformation lies in the fact that the system can be divided into subsystems of unstable and damped modes and it makes possible to apply the principle of adiabatic exclusion of unstable variables (the principle of subordination). Using the smallness of the values characterizing the increments of unstable modes, the principle of subordination is applied for the system of collective variables. In this case, it is shown that the system can be reduced to solving differential equations for a relatively small number of variables (order parameters). In the vicinity of the bifurcation point, two stable solutions are obtained for the order parameters. The first one is a consequence of the competition of modes and it leads, in the soft excitation regime, to a periodic one-dimensional structure for the dislocation density, the second parameter is a result of the cooperation of unstable modes and it leads to the formation of a hexagonal structure in the hard regime of emergence. The question is solved, which of these two structures is implemented, when the system reaches the bifurcation point. The equations for the order parameters are written in the variational form and the corresponding potential function is determined. Its analysis at the points of minima showed that the hexagonal configuration is more likely at the moment of instability occurrence. As the bifurcation parameter increases, the single-mode structure becomes more likely. Thus, the formation of a dissipative cellular structure serves as an indicator of the attaining of non-equilibrium critical conditions in a local volume, when a deformable crystal begins to change its defective structure minimizing its elastic energy.