In this paper, we present an algorithm for capacity optimization in intelligent reflecting surface (IRS)-based multiple-input multiple-output (MIMO) communication systems. To maximize the capacity of elements in IRS, we use augmented Lagrange method with the equivalent transformations on the covariance matrix and reflection matrix constraints. This results an adjustable phase shift on the incident signal. Furthermore, we reshape the complex-valued covariance matrix and reflection matrix to a vector for the ease of calculating partial derivatives to find the search direction. Then, the quasi-Newton updates and modified Broyden-Fletcher-Goldfarb-Shano (BFGS) method in the complex domain form are used to find the local minimum. Finally, numerical simulation results demonstrate that our proposed IRS-aided system using the algorithm performs better than the state-of-the-art and the conventional communication systems.