To meet the demand for the aerosol detection of single-angle and multi-band polarization instrument containing short-wave infrared bands, an inversion algorithm that makes full use of multi-band intensity and polarization information is proposed based on optimal estimation theory. This method uses the polarization information in the short-wave infrared band to perform surface and atmosphere decoupling without a prior information on the surface. This obtains the initial value of the aerosol, and then it uses the scalar information to obtain the final result. Moreover, the multi-band information of the instrument is used for decoupling the surface and atmospheric information, which avoids the inversion error caused by the untimely update of the surface reflectance database and the error of spatio-temporal matching. The measured data of the Particulate Observing Scanning Polarimeter (POSP) are used to test the proposed algorithm. Firstly, to verify the effectiveness of the algorithm under different surface conditions, four regions with large geographical differences (Beijing, Hefei, Baotou, and Taiwan) are selected for aerosol optical depth (AOD) inversion, and they are compared with the aerosol robotic network (AERONET) products of the nearby stations. The validation against the AERONET products produces high correlation coefficients of 0.982, 0.986, 0.718, and 0.989, respectively, which verifies the effectiveness of the algorithm in different regions. Further, we analyzed the effectiveness of the proposed algorithm under different pollution conditions. Regions with AOD >0.7 and AOD < 0.7 are screened by using the AOD products of the Moderate-Resolution Imaging Spectroradiomete (MODIS), and the AOD of the corresponding region is inverted using POSP data. It was found to be spatially consistent with the MODIS products. The correlation coefficient and root mean square error (RMSE) in the AOD high region were 0.802 and 0.217, respectively, and 0.944 and 0.022 in the AOD low region, respectively, which verified the effectiveness of the proposed algorithm under different pollution conditions.