Single-cell or particle characterization typically requires multiple technologies for simultaneous physical and chemical quantitative analysis. Despite the tremendous development of microscopy, multiparameter measurements are challenged by stains or labels, resolution limitations, and quantification of molecular contents. Here we report the Scattering Image-Spectro-Microscopy (SISM) with single-wavelength excitation that enables label-free, quantitative determination of both morphological and molecular contents of single cells or particles. The SISM offers simultaneous measurements of the side elastic light scattering image and the spontaneous Raman scattering spectra from label-free samples. Here elastic scattering imaging performs high-resolution imaging and refractive index quantification for the microscale and nanoscale particles by solving the challenging inverse scattering problem, while Raman scattering provides high molecular sensitivity. We demonstrate the applications of SISM for label-free differentiation of normal cells, clinical leukemic cells, leukemic cell lineage cells, and the nanoscale small extracellular vesicles (EVs) from the plasma of healthy and hepatoma cell-treated mice. Our SISM provides a label-free, multiscale, and quantitative description of single cells or small EVs, which is foreseen to have significant applications in many fields, such as cancer detection and nanoparticle analysis.