The problem of diophantine equations solving is considered in this article. This problem canbe applied in cryptography and cryptanalysis. The description of the genetic algorithm solvingdiophantine equations is stated briefly in the article. The rule of calculation the value of fitnessfunction of chromosome is determined, the coding system in the genetic algorithm is described.The genetic operators used in the algorithm are mentioned and the conditions for their executionare determined. The criterion for stopping the genetic algorithm is described. One of the shortcomingsof the genetic algorithm is analyzed. The shortcoming of the algorithm lies in its attemptsto solve any diophantine equation, including one that has no solutions. A method eliminating thisshortcoming in some cases is proposed. This method is based on number theory. An explanation isgiven in which cases this method will be used. The definition of residue and nonresidue of fixedpower for fixed modulus is given before describing this method. After describing this method theimplementation of the algorithm for solving diophantine equations and systems of them is describedin detail. Then the results of experimental studies of the time and quality of the geneticalgorithm are presented. Then the result of the algorithm is presented for an equation that has nosolutions and for a system of equations that also has no solutions, but in which the total number ofunknowns is too large for the proposed method to work. The algorithm running time is comparedwhen solving an equation and when solving a system of equations. The conclusion is made aboutthe usefulness of the proposed method in solving diophantine equations and systems of diophantineequations.