Dynamic optical modulation in response to stimuli provides exciting opportunities for designing novel sensing, actuating, and authentication devices. Here, we demonstrate that the reversible swelling and deswelling of crosslinked polymer colloidal spheres in response to pH and temperature changes can be utilized to drive the assembly and disassembly of the embedded gold nanoparticles (AuNPs), inducing their plasmonic coupling and decoupling and, correspondingly, color changes. The multi-responsive colloids are created by depositing a monolayer of AuNPs on the surface of resorcinol-formaldehyde (RF) nanospheres, then overcoating them with an additional RF layer, followed by a seeded growth process to enlarge the AuNPs and reduce their interparticle separation to induce significant plasmonic coupling. This configuration facilitates dynamic modulation of plasmonic coupling through the reversible swelling/deswelling of the polymer spheres in response to pH and temperature changes. The rapid and repeatable transitions between coupled and decoupled plasmonic states of AuNPs enable reversible color switching when the polymer spheres are in colloidal form or embedded in hydrogel substrates. Furthermore, leveraging the photothermal effect and stimuli-responsive plasmonic coupling of the embedded AuNPs enables the construction of hybrid hydrogel films featuring switchable anticounterfeiting patterns, showcasing the versatility and potential of this multi-stimuli-responsive plasmonic system.