A nonlinear problem of mean-square approximation of a real nonnegative continuous function with respect to two variables by the modulus of double Fourier integral dependent on two real parameters with use of the smoothing functional is studied. Finding the optimal solutions of this problem is reduced to solution of the Hammerstein type two-dimensional nonlinear integral equation. The numerical algorithms to find the branching lines and branching-off solutions of this equation are constructed and justified. Numerical examples are presented.