In this paper, we propose a low-complexity format-transparent digital signal processing (DSP) scheme for next generation flexible and energy-efficient transceiver. It employs QPSK symbols as the training and pilot symbols for the initialization and tracking stage of the receiver-side DSP, respectively, for various modulation formats. The performance is numerically and experimentally evaluated in a dual polarization (DP) 11 Gbaud 64QAM system. Employing the proposed DSP scheme, we conduct a system-level study of Tb/s bandwidth-adaptive superchannel transmissions with flexible modulation formats including QPSK, 8QAM and 16QAM. The spectrum bandwidth allocation is realized in the digital domain instead of turning on/off sub-channels, which improves the performance of higher order QAM. Various transmission distances ranging from 240 km to 6240 km are demonstrated with a colorless detection for hardware complexity reduction.