A detailed analysis of the photoluminescence (PL) intensity distribution in singly Cr-doped CdTe/ZnTe quantum dots (QDs) is performed. First of all, we demonstrate that hole-Cr flip-flops induced by an interplay of the hole-Cr exchange interaction and the coupling with acoustic phonons are the main source of spin relaxation within the exciton-Cr complex. This spin flip mechanism appears in the excitation power dependence of the PL of the exciton as well as in the intensity distribution of the resonant PL. The resonant optical pumping of the Cr spin which was recently demonstrated can also be explained by these hole-Cr flip-flops. Despite the fast exciton-Cr spin dynamics, an analysis of the PL intensity under magnetic field shows that the hole-Cr exchange interaction in CdTe/ZnTe QDs is antiferromagnetic. In addition to the Cr spin dynamics induced by the interaction with carriers' spin, we finally demonstrate using time resolved optical pumping measurements that a Cr spin interacts with nonequilibrium acoustic phonons generated during the optical excitation inside or near the QD.