Solar irradiance variations over solar rotational time-scales are largely determined by the passage of magnetic structures across the visible solar disk. Variations on solar cycle time scales are thought to be similarly due to changes in surface magnetism with activity. Understanding the contribution of magnetic structures to total solar irradiance and solar spectral irradiance requires assessing their contributions as a function of disk position. Since only relative photometry is possible from the ground, the contrasts of image pixels are measured with respect to a center-to-limb intensity profile. Using nine years of full-disk red and blue continuum images from the Precision Solar Photometric Telescope at the Mauna Loa Solar Observatory (PSPT/MLSO), we examine the sensitivity of continuum contrast measurements to the center-to-limb profile definition. Profiles which differ only by the amount of magnetic activity allowed in the pixels used to determine them yield oppositely signed solar cycle length continuum contrast trends; either agreeing with the result of Preminger et al. (2011) showing negative correlation with solar cycle or disagreeing and showing positive correlation with solar cycle. Changes in the center-to-limb profile shape over the solar cycle are responsible for the contradictory contrast results, and we demonstrate that the lowest contrast structures, internetwork and network, are most sensitive to these. Thus the strengths of the full-disk, internetwork, and network photometric trends depend critically on the magnetic flux density used in the quiet-sun definition. We conclude that the contributions of low contrast magnetic structures to variations in the solar continuum output, particularly to long-term variations, are difficult, if not impossible, to determine without the use of radiometric imaging.