The pedestrian level pollutant transport in street canyons with multiple aspect ratios (H/W) is numerically investigated in the present work, regarding of various unstable thermal stratification scenarios and plain surrounding. Non-isothermal turbulent wind flow, temperature field and pollutant spread within and above the street canyons are solved by the realizable k-ε turbulence model along with the enhanced wall treatment. One-vortex flow regime is observed for shallow canyons with H/W=0.5, whereas multi-vortex flow regime is observed for deep canyons with H/W=2.0. Both one-vortex and multi-vortex regimes could be observed for the street canyons with H/W=1.0, where the secondary vortex could be initiated by the flow separation and intensified by unstable thermal stratification. Air exchange rate (AER) and pollutant retention time are adopted to respectively evaluate the street canyon ventilation and pollutant removal performance. A second-order polynomial functional relationship is established between AER and Richardson number (Ri). Similar functional relationship could be established between retention time and Ri, and it is only valid for canyons with one-vortex flow regime. In addition, retention time could be prolonged abruptly for canyons with multi-vortex flow regime. Very weak secondary vortex is presented at the ground level of deep canyons with mild stratification, where pollutants are highly accumulated. However, with the decrease of Ri, pollutant concentration adjacent to the ground reduces accordingly. Present research could be applied to guide the urban design and city planning for enhancing pedestrian environment.