Here we propose the local polynomial method to solve the problem of estimating the frequency response function in the linear system. Compared with other nonparametric identification methods based on the windowing strategies, this new identification method can be remarkably efficient in reducing the effect caused by the leakage error when the discrete Fourier transform is used under a non-periodic input excited signal. Considering the constraints between the coefficients of the polynomials at neighbour frequencies, we modify the proposed local polynomial method to get one constrained local polynomial method. The modified local polynomial method reduces the mean square error of the frequency response function and the estimation of the frequency response function is identified by one multi-objective least squares criterion. Finally the simulation example results confirm the identification theoretical results.