The soluble solids content (SSC) is an important factor in the internal quality detection of apples. It is essential to have reliable and high-speed measurement of the SSC. However, almost all traditional equipment is inconvenient and expensive. We designed a handheld nondestructive SSC detector based on near-infrared (NIR) spectroscopy, which is composed of a portable NIR spectrometer, cloud server, smartphone app, and prediction model of SSC. We preprocessed the spectrum with multiplicative scatter correction (MSC), standard normal variable transformation (SNV), and Savitzky-Golay (S-G) smoothing algorithms. Besides, the linear weight reduction of the particle swarm optimization algorithm is carried out, and we establish the model of an extreme learning machine optimized with the improved particle swarm optimization (IPSO-ELM) algorithm. The R2, root mean square error of prediction (RMSEP), and residual prediction deviation (RPD) of the model are 0.993, 0.0155, and 11.6, respectively, which are better than the traditional model obviously. In addition, the number of wavelengths reduced from 228 to 70 as the model is simplified with the uninformative variable elimination (UVE) method. The time of training is reduced by 29.30% compared with the original spectrum. It can be verified that the IPSO-ELM model has good prediction performance, and the NIR diffuse reflectance spectroscopy is a reliable nondestructive measurement of SSC in apples.