We present the robust selection of high-redshift quiescent galaxies (QG) and poststarburst (PSB) galaxies using ultra-deep NIRCam and MIRI imaging from the JWST Advanced Deep Extragalactic Survey (JADES). At 3 < z < 6, MIRI 7.7 μm imaging provides rest-frame J band, which is commonly used to break the degeneracy between old stellar populations and dust attenuation at lower redshifts. We identify 23 passively evolving galaxies in UVJ color space in a mass-limited (log M ⋆/M ⊙ ≥ 8.5) sample over 8.8 arcmin2. An evaluation of the contribution of the 7.7 μm shows that JADES-like NIRCam coverage (9+ photometric bands) can compensate for lacking the J band at these redshifts; however, more limited three-band selections perform better with MIRI. Our sample is characterized by rapid quenching timescales (∼100–600 Myr) with formation redshifts z f ≲ 9 and includes a potential record-holding massive QG at zphot=5.33−0.17+0.16 and two QGs with evidence for significant residual dust content (A V ∼ 1–2). In addition, we present a large sample of 12 log M ⋆/M ⊙ = 8.5–9.5 PSBs, demonstrating that UVJ selection can be extended to low mass. An analysis of the environment of our sample reveals that the group known as the Cosmic Rose contains a massive QG and a dust-obscured star-forming galaxy (a so-called Jekyll and Hyde pair) plus three additional QGs within ∼20 kpc. Moreover, the Cosmic Rose is part of a larger overdensity at z ∼ 3.7, which contains 7/12 of our low-mass PSBs. Another four low-mass PSBs are members of an overdensity at z ∼ 3.4; this result strongly indicates low-mass PSBs are preferentially associated with overdense environments at z > 3.