The growing interest in converting infrared light to visible light in many photovoltaic fields requires the development of upconversion systems with high luminescence quantum yields and stability without blinking. However, the weak and narrow absorption of rare earth ions has greatly restricted the luminescent efficiency of rare earth dopants and hampered their translation from experimental research to real world applications. In this work, horizontally aligned gold nanorod arrays with self-contained spacer and obvious polarization dependent plasmonic properties are prepared on the substrate of silicon wafers by a self-assembly method of droplet evaporation, and polarization-dependent enhancement on the upconversion luminescence of rare earth ions doped nanoparticles (NaYF4: Yb3+, Er3+) are investigated. The results show that the luminescence intensity of the nanoparticles is greatly enhanced by a factor of about 140, when the horizontally aligned gold nanorod arrays are available as a substrate. The effect of self-contained spacer of the arrays to prevent quenching of the luminescence is investigated with the introduction of an Al2O3 layer as an isolate and the adjustment of its thickness. It is found that the self-contained spacer can effectively block the non-radiative energy transfer without the need of additional spacer. In addition, the horizontally aligned gold nanorod arrays also exhibit excellent optical polarization anisotropy. As the polarization angle of the incident light gradually increases, the intensity of the fluorescent radiation decreases, which is interpreted and analyzed by simulating the near field distribution. This strategy not only provides a powerful solution to build high-efficiency surface enhanced fluorescence systems, but also suggests a new pathway to realize the polarization-selective excitation of upconversion luminescence.