Optical transition and near-infrared upconversion luminescence properties of Er3+ single-doped and Er3+, Yb3+ co-doped YNbO4 phosphors are reported. Based on the framework of Judd-Ofelt theory and modified Judd-Ofelt calculation method, the optical transition properties of Er3+ and Yb3+ ions in YNbO4: Er3+, Yb3+ sample were studied, respectively. The calculated results showed that the absorption cross-section of Er3+ ion at ∼1550 nm is larger than that at ∼980 nm in YNbO4: Er3+ phosphor. And, the absorption cross-section of 2F7/2 → 2F5/2 transition of Yb3+ in YNbO4: 15.0 mol% Er3+, 15.0 mol% Yb3+ sample was calculated to be 138.70 × 10−20 cm2. According to the emission cross section of Er3+ (4I11/2 → 4I15/2) and the absorption cross section of Yb3+ (2F7/2 → 2F5/2), the energy transfer rates from Er3+ to Yb3+ were quantitatively analyzed via the Föster-Dexter model. Furthermore, 1550 nm was selected as the excitation wavelength, the near-infrared upconversion emission spectra of YNbO4: Er3+, Yb3+ samples were measured. It was found that the sample has strong emission at ∼980 nm and the emission intensity was still enhanced with the incorporation of Yb3+.