The large cavities observed in the dust and gas distributions of transition disks may be explained by planet-disk interactions. At ~145 pc, 2MASS J16042165-2130284 (J1604) is a 5-12 Myr old transitional disk with different gap sizes in the mm- and $\mu$m-sized dust distributions (outer edges at ~79 and at ~63 au, respectively). Its $^{12}$CO emission shows a ~30 au cavity. This radial structure suggests that giant planets are sculpting this disk. We aim to constrain the masses and locations of plausible giant planets around J1604. We observed J1604 with the Spectro-Polarimetric High-contrast Exoplanet REsearch (SPHERE) at the Very Large Telescope (VLT), in IRDIFS\_EXT, pupil-stabilized mode, obtaining YJH- band images with the integral field spectrograph (IFS) and K1K2-band images with the Infra-Red Dual-beam Imager and Spectrograph (IRDIS). The dataset was processed exploiting the angular differential imaging (ADI) technique with high-contrast algorithms. Our observations reach a contrast of $\Delta K, YH$ ~12 mag from 0.15" to 0.80" (~22 to 115 au), but no planet candidate is detected. The disk is directly imaged in scattered light at all bands from Y to K, and it shows a red color. This indicates that the dust particles in the disk surface are mainly $\gtrsim0.3\,\mu$m-sized grains. We confirm the sharp dip/decrement in scattered light in agreement with polarized light observations. Comparing our images with a radiative transfer model we argue that the southern side of the disk is most likely the nearest. This work represents the deepest search yet for companions around J1604. We reach a mass sensitivity of $\gtrsim 2-3M_{Jup}$ from ~22 to ~115 au according to a hot start scenario. We propose that a brown dwarf orbiting inside of ~15 au and additional Jovian planets at larger radii could account for the observed properties of J1604 while explaining our lack of detection.