For millimeter wave (mm-wave) communication systems, a lens antenna array with single-carrier transmission and path delay compensation is a promising technique for realizing cost-effective large multiple-input multiple-output communications with limited number of radio frequency chains. In this paper, we study the multi-user mm-wave downlink lens antenna array system for the general frequency-selective channels. By leveraging the angle-dependent energy focusing property of the lens antenna array and the angular sparsity of mm-wave channels, we investigate the low-complexity single-carrier transmission scheme with path delay pre-compensation applied at the base station (BS). The resulting signal-to-interference-plus-noise ratio (SINR) is derived by taking into account both the residual inter-symbol interference and inter-user interference. Based on the derived SINR expression, we propose an effective joint antenna selection and beamforming scheme by utilizing the group sparse optimization to accommodate for the limited number of RF chains at the BS. Thus, the proposed scheme can obtain the approximate performance with the fully digital case and has a better performance than the conventional orthogonal frequency-division multiplexing mode for the frequency-selectivity channels. Numerical results are provided to verify the effectiveness of the proposed schemes.