Nanophotonics is an emerging science dealing with the interaction of light and matter on a nanometer scale and holds promise to produce new generation nanophosphors with highly efficient frequency conversion of infrared (IR) light. Scientists can control the excitation dynamics by using nanochemistry to produce hierarchically built nanostructures and tailor their interfaces. These nanophosphors can either perform frequency up-conversion from IR to visible or ultraviolet (UV) or down-conversion, which results in the IR light being further red shifted. Nanophotonics and nanochemistry open up numerous opportunities for these photon converters, including in high contrast bioimaging, photodynamic therapy, drug release and gene delivery, nanothermometry, and solar cells. Applications of these nanophosphors in these directions derive from three main stimuli. Light excitation and emission within the near-infrared (NIR) "optical transparency window" of tissues is ideal for high contrast in vitro and in vivo imaging. This is due to low natural florescence, reduced scattering background, and deep penetration in tissues. Secondly, the naked eye is highly sensitive in the visible range, but it has no response to IR light. Therefore, many scientists have interest in the frequency up-conversion of IR wavelengths for security and display applications. Lastly, frequency up-conversion can convert IR photons to higher energy photons, which can then readily be absorbed by solar materials. Current solar devices do not use abundant IR light that comprises almost half of solar energy. In this Account, we present our recent work on nanophotonic control of frequency up- and down-conversion in fluoride nanophosphors, and their biophotonic and nanophotonic applications. Through nanoscopic control of phonon dynamics, electronic energy transfer, local crystal field, and surface-induced non-radiative processes, we were able to produce new generation nanophosphors with highly efficient frequency conversion of IR light. We show that nanochemistry plays a vital role in the design and interface engineering of nanophosphors, providing pathways to expand their range of applications. High contrast in vitro and in vivo NIR-to-NIR up- and down-conversion bioimaging were successfully demonstrated by our group, evoking wide interests along this line. We introduced trivalent gadolinium ions into the lattice of the nanophosphors or into the shell layer of nanophosphors in a core/shell configuration to produce novel nanophosphors for multimodal (MRI and optical) imaging. We also demonstrate the security and display applications using photopatternable NIR-to-NIR and NIR-to-visible frequency up-conversion nanophosphors with appropriately engineered surface chemistry. In addition, we present preliminary results on dye-sensitized solar cells using up-conversion in fluoride lattice-based nanophosphors for IR photon harvesting.