It is a well-established fact in quantum information theory, that uniform averaging over the collective action of a unitary group on a multipartite quantum state projects the state to a form equivalent to a permutation operator of the subsystems. Hence states equivalent to permutation operators are untouched by collective unitary noise. A trivial observation shows that uniform averaging over permutation operators projects the state into a form with block-diagonal structure equivalent to the one of the collective action of the unitary group. We introduce a name for this property: duality of averaging. The mathematical reason behind this duality is the fact that the collective action of the unitary group on the tensor product state space of a multipartite quantum system and the action of the permutation operations are mutual commutants when treated as matrix algebras. Such pairs of matrix algebras are known as dual reductive pairs. In this work we show, that in the case of finite dimensional quantum systems such duality of averaging holds for any pairs of symmetry groups being dual reductive pairs, regardless of whether they are compact or not, as long as the averaging operation is defined via iterated integral over the Cartan decomposition of the group action. Although our result is very general, we focus much attention on the concrete example of a dual reductive pair consisting of collective action of special linear matrices and permutation operations, which physically corresponds to averaging multipartite quantum states over non-unitary SLOCC-type (Stochastic Local Operations and Classical Communication) operations. In this context we show, that noiseless subsystems known from collective unitary averaging persist in the case of SLOCC averaging in a conditional way: upon postselection to specific invariant subspaces.