The refractive index (RI), an important optical property of a material, is measured by commercial refractometers in the food, agricultural, chemical, and manufacturing industries. Most of these refractometers must be equipped with a prism for light dispersion, which drastically limits the design and size of the refractometer. Recently, there have been several reports on the development of a surface plasmon resonance (SPR)-based RI detector, which is characterized by its high sensitivity and simplicity. However, regardless of the prism, an expensive spectrometer is required to analyze the resonance wavelength or angle of incidence. This paper proposes a method that eliminates the need for the prism and other conventional spectrometer components. For this purpose, total internal reflection SPR technology was used on an Ag thin film, and RI analysis was combined with a lens-free CMOS image sensor or a smartphone camera. A finite-difference time-domain (FDTD) numerical simulation was performed to evaluate the relationship between the output power intensity and Ag film thickness for different RIs at three wavelengths of commercial light-emitting diodes (LEDs). The maximum sensitivity of −824.54 RIU−1 was achieved with AG20 at an incident wavelength of 559 nm. Due to its simple design and cost effectiveness, this prism-less, SPR-based refractometer combined with a lens-free CMOS image sensor or a smartphone could be a superior candidate for a point-of-care device that can determine the RIs of various analytes in the field of biological or chemical sensing.