This paper presents a dedicated digital signal process (DSP) for four pulse amplitude modulation (PAM4) SerDes receivers. It is targeted to implement data recovery and adaptive equalization under ultra-high-speed and large channel attenuation with a small area and high power efficiency. The DSP consists of a clock data recovery (CDR), a 16-tap feed forward equalizer (FFE), a 1-tap decision feedback equalizer (DFE), and an automatic adaptation engine. An adaptive least mean square (LMS) algorithm is utilized to make the system more intelligent in calculating tap coefficients of the FFE and DFE. To address the timing limitation associated with traditional digital DFE that cannot handle large amounts of parallel data at a high speed, speculative techniques and a customized 4-to-1 multiplexer (MUX) unit are employed to remove the summation time and reduce the selection time, respectively. A first-order sigma-delta modulator is used to replace the traditional moving average to calculate average voltages, which could prominently save the hardware resources and power consumption. Additionally, the influence of input quantization resolution on the equalization ability is analyzed. Implemented in a 28-nm CMOS, the DSP could compensate for up to 33-dB loss at 100 Gb/s with a power consumption of 7.22 pJ/bit.