We apply two reconstructed spectral solar forcing scenarios, one SIM (Spectral Irradiance Monitor) based, the other the SATIRE (Spectral And Total Irradiance REconstruction) modeled, as inputs to the GISS (Goddard Institute for Space Studies) GCMAM (Global Climate Middle Atmosphere Model) to examine climate responses on decadal to centennial time scales, focusing on quantifying the difference of climate response between the two solar forcing scenarios. We run the GCMAM for about 400 years with present day trace gas and aerosol for the two solar forcing inputs. We find that the SIM-based solar forcing induces much larger long-term response and 11-year variation in global averaged stratospheric temperature and column ozone. We find significant decreasing trends of planetary albedo for both forcing scenarios in the 400-year model runs. However the mechanisms for the decrease are very different. For SATIRE solar forcing, the decreasing trend of planetary albedo is associated with changes in cloud cover. For SIM-based solar forcing, without significant change in cloud cover on centennial and longer time scales, the apparent decreasing trend of planetary albedo is mainly due to out-of-phase variation in shortwave radiative forcing proxy (downwelling flux for wavelength >330 nm) and total solar irradiance (TSI). From the Maunder Minimum to present, global averaged annual mean surface air temperature has a response of ~0.1 °C to SATIRE solar forcing compared to ~0.04 °C to SIM-based solar forcing. For 11-year solar cycle, the global surface air temperature response has 3-year lagged response to either forcing scenario. The global surface air 11-year temperature response to SATIRE forcing is about 0.12 °C, similar to recent multi-model estimates, and comparable to the observational-based evidence. However, the global surface air temperature response to 11-year SIM-based solar forcing is insignificant and inconsistent with observation-based evidence.