Noise within solid-state systems at low temperatures can typically be traced back to material defects. In amorphous materials, these defects are broadly described by the tunneling two-level systems (TLSs) model. TLS have recently taken on further relevance in quantum computing because they dominate the coherence limit of superconducting quantum circuits. Efforts to mitigate TLS impacts have thus far focused on circuit design, material selection, and surface treatments. Our work takes an approach that directly modifies TLS properties. This is achieved by creating an acoustic bandgap that suppresses all microwave-frequency phonons around the operating frequency of a transmon qubit. For embedded TLS strongly coupled to the transmon qubit, we measure a pronounced increase in relaxation time by two orders of magnitude, with the longest T1 time exceeding 5 milliseconds. Our work opens avenues for studying the physics of highly coherent TLS and methods for mitigating noise within solid-state quantum devices.