An attempt is made to clarify the ballistic nonlinear sigma model formalism recently proposed for quantum chaotic systems, by looking at the spectral determinant Z(s)=Det(1−sU) for quantized maps U∈U(N), and studying the correlator ωU(s)=∫dθ|Z(eiθs)|2. By identifying U(N) as one member of a dual pair acting in the spinor representation of Spin(4N), the expansion of ωU(s) in powers of s2 is shown to be a decomposition into irreducible characters of U(N). In close analogy with the ballistic nonlinear sigma model, a coherent-state integral representation of ωU(s) is developed. For generic U this integral has (N2N) saddle points and the leading-order saddle-point approximation turns out to reproduce ωU(s) exactly, up to a constant factor. This miracle is explained by interpreting ωU(s) as a character of U(2N), and arguing that the leading-order saddle-point result corresponds to the Weyl character formula. Unfortunately, the Weyl decomposition behaves nonsmoothly in the semiclassical limit N→∞, and to make further progress some additional averaging needs to be introduced. Several schemes are investigated, including averaging over basis states and an “isotropic” average. The saddle-point approximation applied in conjunction with these schemes is demonstrated to give incorrect results in general, one notable exception being a semiclassical averaging scheme, for which all loop corrections vanish identically. As a side product of the dual pair decomposition with isotropic averaging, the crossover between the Poisson and CUE limits is obtained.