The concept of a multi-dimensional carrierless amplitude and phase modulation (CAP) passive optical network (PON) system based on non-orthogonal transceiver filters is presented for the first time, to the best of our knowledge, to improve the spectral efficiency (SE). To eliminate the interferences caused by the non-orthogonal transceiver filters, a novel inter-symbol interference (ISI) and cross-channel interference (CCI) cancellation algorithm (ICCA) is proposed. In our experiment, a 10 Gb/s four-dimensional with four levels/dimension and non-orthogonal CAP data transmission is successfully demonstrated over a 20 km standard single mode fiber. The results indicate that a SE gain from 3.85 to 4.35 bit/s/Hz is achieved by the combination of the proposed ICCA with the traditional modified cascaded multi-modulus and decision-directed least mean square algorithm.