The Sun is a bright gamma-ray source due to hadronic cosmic-ray interactions with solar gas. While it is known that incoming cosmic rays must generally first be reflected by solar magnetic fields to produce outgoing gamma rays, theoretical models have yet to reproduce the observed spectra. We introduce a simplified model of the solar magnetic fields that captures the main elements relevant to gamma-ray production. These are a flux tube, representing the network elements, and a flux sheet, representing the intergranular sheets. Both the tube and sheet have a horizontal size of order 100 km and serve as sites where cosmic rays are reflected and gamma rays are produced. While our simplified double-structure model does not capture all the complexities of the solar-surface magnetic fields, such as Alfvén turbulence from wave interactions or magnetic fluctuations from convection motions, it improves on previous models by reasonably producing both the hard spectrum seen by Fermi Large Area Telescope at 1–200 GeV and the considerably softer spectrum seen by the High Altitude Water Cherenkov Observatory (HAWC) at near 103 GeV. We show that lower-energy (≲10 GeV) gamma rays are primarily produced in the network elements and higher-energy (≳few × 10 GeV) gamma rays in the intergranular sheets. Notably, the spectrum softening observed by HAWC results from the limited effectiveness of capturing and reflecting ∼104 GeV cosmic rays by the finite-sized intergranular sheets. Our study is important for understanding cosmic-ray transport in the solar atmosphere and will lead to insights into small-scale magnetic fields at the photosphere.