We present a novel approach to photometric redshifts, one that merges the advantages of both the template-fitting and empirical-fitting algorithms without any of their disadvantages. This technique derives a set of templates describing the spectral energy distributions of galaxies from a catalog with both multicolor photometry and spectroscopic redshifts. The algorithm is essentially using the shapes of the templates as the fitting parameters. From simulated multicolor data we show that for a small training set of galaxies we can reconstruct robustly the underlying spectral energy distributions even in the presence of substantial errors in the photometric observations. We apply these techniques to the multicolor and spectroscopic observations of the Hubble Deep Field, building a set of template spectra that reproduced the observed galaxy colors to better than 10%. Finally, we demonstrate that these improved spectral energy distributions lead to a photometric redshift relation for the Hubble Deep Field that is more accurate than standard template-based approaches.