We demonstrate a low threading dislocation density (TDD) and smooth surface InAs layer epitaxially grown on Si by suppressing phase separation of InxAl1−xAs (x = 0 to 1) graded buffer and by inserting a tensile-strained In0.95Al0.05As dislocation filter layer. While keeping the total III–V layer below 2.7 μm to avoid thermal cracks, we have achieved a sixfold reduction of TDD in InAs on Si compared to the unoptimized structure. We found a strong correlation between the metamorphic InAs surface roughness and TDD as a function of InxAl1−xAs buffer thickness. An optimal thickness of 175 nm was obtained where both phase separation and 3D islanding growth were suppressed. Moreover, a tensile-strained In0.95Al0.05As dislocation filter layer and high growth temperature of the InAs cap layer further assisted the dislocation reduction process, which led to a TDD to 1.37 × 108 cm−2. Finally, an InAs p-i-n photodetector grown on the optimized InAs/Si template confirmed its high quality by showing an improved responsivity from 0.16 to 0.32 A/W at a 2 μm wavelength.