Establishment of a rapid, sensitive, visual, accurate and low-cost fluorescence detection system to detect multiple targets was of great significance in food safety evaluation, ecological environment monitoring and human health monitoring. In this work, a smartphone-assisted down/up-conversion dual-mode ratiometric fluorescence sensor was proposed based on metal-organic framework (NH2-MIL-101(Fe)) and CdTe quantum dots (CdTe QDs) for visual detection of mercury ions (Hg2+) and L-penicillamine (L-PA), in which NH2-MIL-101(Fe) was used as the reference signal and CdTe QDs was used as the response signal. The down-conversion fluorescence system at excitation wavelength of 300 nm (ex: 330 nm) was used to detect Hg2+ and L-PA, in which the detection limit of Hg2+ was 0.053 nM with the fluorescence color changed from green to blue, and the detection limit of L-PA was 1.10 nM with the fluorescence color changed from blue to green. Meanwhile, the up-conversion fluorescence system at excitation wavelength of 700 nm (ex: 700 nm) was used to detect Hg2+ and L-PA. The detection limits of Hg2+ and L-PA were 0.11 nM and 2.93 nM, respectively. The detection of Hg2+ and L-PA were also carried out based on the color extraction RGB values identified by the smartphone with a detection limit of 0.091 nM for Hg2+ and 8.97 nM for L-PA. In addition, the concentrations of Hg2+ and L-PA were evaluated by three-dimensional dynamic analysis in complex environments. The smartphone-assisted down/up-conversion dual-mode ratiometric fluorescence sensor system provides a new strategy for detection Hg2+ and L-PA in food safety evaluation, environmental monitoring and human health monitoring.