Despite ongoing efforts to convince users of the value of password security and to enforce password creation standards on them, in many information systems the human factor still plays a role. In addition, not only do most users’ password creation and management practices largely remain unchanged, but password cracking tools and more critically, computer hardware also continue to advance. In this paper we present a model in ethical hacking; the proposed model concentrated on brute force attacks for password cracking. The main novelty of our work is that it first presents a mathematical model that calculates the number of different password permutations of varying lengths. Then the brute force attack is modelled using the Markov chain model and a method is developed to formulate the conventional optimization problem, which is classified as a discrete nonlinear problem. The experiments’ results demonstrate and validate the method’s effectiveness and suitability.