The Pliocene Yutengping Sandstone (depth 1642–1882 m) and its overlying caprock shale (depth 1395–1642 m) in Hsinchu City, central Taiwan, were intended for a storage site of CO2. Formation water was collected from a gas well located at a depth of 1827–1846 m. This study investigated changes in water chemistry and dissolution of trace elements from the sandstone and shale at 25 MPa and 90 °C in the presence and absence of supercritical CO2 (scCO2) over 7 days. The results showed substantial dissolution of V, Cr, Co, Cu, and Rb from the sandstone and shale into formation water in the presence of scCO2 fluid, while the release of Zn, Se, Mo, and Cd from the sandstone and shale was minimal. Desorption of V, Cr, Mn, Fe, Sr, and Ba was more pronounced from the sandstone than from shale, whereas Co, Ni, Cu, As, and Mo desorbed more from the shale. The concentration of As in formation water increased from 1.4 μg/L to 130 μg/L after in contact with scCO2. Such a high As concentration may present a significant threat to shallow groundwater quality in this region, particularly if leakage along faults and rock fractures in the region occurred.