The Cholesky decomposition-block diagonalization (CD-BD) interference alignment (IA) for a multiuser multiple input multiple output (MU-MIMO) relay system is proposed, which designs precoders for the multiple access channel (MAC) by employing the singular value decomposition (SVD) as well as the mean square error (MSE) detector for the broadcast Hermitian channel (BHC) taken advantage of in our design. Also, in our proposed CD-BD IA algorithm, the relaying function is made use to restructure the quasieigenvalue decomposition (quasi-EVD) equivalent channel. This approach used for the design of BD precoding matrix can significantly reduce the computational complexity and proposed algorithm can address several optimization criteria, which is achieved by designing the precoding matrices in two steps. In the first step, we use Cholesky decomposition to maximize the sum-of-rate (SR) with the minimum mean square error (MMSE) detection. In the next step, we optimize the system BER performance with the overlap of the row spaces spanned by the effective channel matrices of different users. By iterating the closed form of the solution, we are able not only to maximize the achievable sum-of-rate (ASR), but also to minimize the BER performance at a high signal-to-noise ratio (SNR) region.