In this paper, we investigate joint source precoding matrices and relay processing matrix design for multiuser multiple-input-multiple-output (MIMO) nonregenerative relay networks in the presence of the direct source-destination links. We consider both capacity and mean-square-error (MSE) criteria subject to the distributed power constraints, which are nonconvex and apparently have no simple solutions. Therefore, we propose an optimal source precoding matrix structure based on the point-to-point MIMO channel technique and a new relay processing matrix structure under the modified power constraint at relay node, based on which a nested iterative algorithm of jointly optimizing sources precoding and relay processing is established. We show that the capacity-based optimal source precoding matrices share the same structure with the MSE-based matrices and so does the optimal relay processing matrix. Simulation results demonstrate that the proposed algorithm outperforms the existing results.