In this paper, we consider certain descending filtrations of the $\mathrm{SL}(m,\C)$-representation algebras of free groups and free abelian groups. By using it, we introduce analogs of the Johnson homomorphisms of the automorphism groups of free groups. We show that the first homomorphisms are extended to the automorphism groups of free groups as crossed homomorphisms. Furthermore we show that the extended crossed homomorphisms induce Kawazumi's cocycles and Morita's cocycles. This works are generalization of our previous results for the $\mathrm{SL}(2,\C)$-representation algebras.