Refractive index (RI) sensors are of great interest for label-free optical biosensing. A tapered optical fiber (TOF) RI sensor with micron-sized waist diameters can dramatically enhance sensor sensitivity by reducing the mode volume over a long distance. Here, a simple and fast method is used to fabricate highly sensitive refractive index sensors based on localized surface plasmon resonance (LSPR). Two TOFs (l = 5 mm) with waist diameters of 5 µm and 12 µm demonstrated sensitivity enhancement at λ = 1559 nm for glucose sensing (5–45 wt%) at room temperature. The optical power transmission decreased with increasing glucose concentration due to the interaction of the propagating light in the evanescent field with glucose molecules. The coating of the TOF with gold nanoparticles (AuNPs) as an active layer for glucose sensing generated LSPR through the interaction of the evanescent wave with AuNPs deposited at the tapered waist. The results indicated that the TOF (Ø = 5 µm) exhibited improved sensing performance with a sensitivity of 1265%/RIU compared to the TOF (Ø = 12 µm) at 560%/RIU towards glucose. The AuNPs were characterized using scanning electron microscopy and ultraviolent-visible spectroscopy. The AuNPs-decorated TOF (Ø = 12 µm) demonstrated a high sensitivity of 2032%/RIU toward glucose. The AuNPs-decorated TOF sensor showed a sensitivity enhancement of nearly 4 times over TOF (Ø = 12 µm) with RI ranging from 1.328 to 1.393. The fabricated TOF enabled ultrasensitive glucose detection with good stability and fast response that may lead to next-generation ultrasensitive biosensors for real-world applications, such as disease diagnosis.