In recent years, the detection of gravitational waves by LIGO and PTA collaborationshave raised the intriguing possibility of excess matter power at small scales. Such anincrease can be achieved by ultra slow roll (USR) phase during inflationary epoch.We constrain excess power over small scales within the framework of such models usingcosmological datasets, particularly of CMB anisotropies and Lyman-α.We parameterize the USR phase in terms of thee-fold at the onset of USR (counted from the end of inflation) N̅1 andthe duration of USR phase Δ N.The former dictates the scale of enhancement in the primordial power spectrum, while the latter determines the amplitude of such anenhancement.From a joint dataset of CMB and galaxy surveys, we obtainN̅1 ≲ 45 with no bound on Δ N. This in turn implies thatthe scales over which the power spectrum can deviate significantly from thenearly scale invariant behavior of a typical slow-roll model isk ≳1 Mpc-1.On the other hand, the Lyman-α data is sensitive to baryonic powerspectrum along the line of sight. We consider a semi-analytic theoretical methodand high spectral-resolution Lyman-α data to constrain the model.The Lyman-α data limits both the USR parameters:N̅1 ≲ 41 and Δ N ≲ 0.4.This constrains the amplitude of the power spectrum enhancement to be less thana factor of hundred over scales 1 ≲ k/ Mpc-1≲ 100,thereby considerably improving the constraint on power over these scales ascompared to the bounds arrived at from CMB spectral distortion.