Mixing performance in the combustion chamber is important in achieving the efficiency of scramjet for its compact structure and the short residence time caused by high velocity. This study focuses on predicting the mixing performance in a viscous supersonic flow past the cavity flame holder in the scramjet, using the discontinuous spectral element method with direct numerical simulation. The arrangement and distribution of a series of cavities are calculated at various inlet velocities. By comparing the contours of Mach number and static temperature, as well as analyzing the airflow residence time with a new calculating formula and the drag based on the numerical results, it is concluded that higher inlet velocities result in faster stabilization. It also leads to longer airflow residence time when the supersonic flow passes through the dual-cavity with a tandem connection rather than the parallel one or the single-cavity structures. As for the shortened rear wall of cavities, these structures can decrease the drag quickly, but they also decrease the airflow residence time seriously, which destroys the mixing performance. In addition, these conclusions are applied to a practical case of the cavity flame holder, verifying the effectiveness of tandem dual-cavity structures in enhancing the mixing performance by increasing the airflow residence time and maintaining or reducing the drag. This study can provide valuable suggestions in further design of cavity flame holders for different flow conditions.