For evaluating the performance of room acoustical simulations or numerical simulations in general, these are usually compared to corresponding measurements as a benchmark. Previous studies indicated that differences may result from neglecting wave effects (scattering, diffraction, attenuation at grazing incidence). However, it also proved to be a challenge to provide a precise representation of the primary and secondary structure (geometry, source and receiver characteristics, absorption and scattering coefficients) of the measured ground truth to be re-modeled in the simulation. The round robin on auralization aimed to overcome such shortcomings by generating a ground truth database of room acoustical environments provided to developers of room simulation software. The database includes a selection of acoustic scenes such as “single reflection,” or “coupled rooms” which isolate single acoustic phenomena, as well as three complex “real-world” environments of different size. Simulated monaural and binaural impulse responses were evaluated by comparing them to the corresponding measurements on the basis of acoustical parameters as well as perceptual qualities. We introduce the concept of the round robin along with the description of the acoustic scenes, the acquisition of monaural and binaural impulse responses, and the identification of the boundary conditions.