We propose a novel direct detection (DD) scheme for polarization division multiplexed (PDM) single sideband (SSB) signals with two orthogonal carriers located at the opposite sides. Polarization diversity is realized with a pair of optical filters that are used to suppress the unwanted orthogonal carrier component. A PDM-SSB DD receiver is thus constructed without polarization de-rotation. The intra-polarization signal-signal beat interference (SSBI) can be mitigated by Kramers-Kronig detection or iterative SSBI cancellation. For inter-polarization SSBI mitigation, we propose a joint iterative SSBI cancellation method. The proposed PDM-SSB DD scheme is validated with a principle experiment of 40Gbaud PDM-SSB 16-ary quadrature amplitude modulation (16-QAM) signals. After 80km standard single-mode fiber (SSMF) transmission, the bit-error rates (BERs) achieve 20% hard-decision forward error correction (HD-FEC) threshold of 1.5 × 10-2. The performance of iterative SSBI cancellation, Kramers-Kronig detection, and joint iterative SSBI cancellation are evaluated for PDM-SSB signals with different carrier-to-signal ratios (CSPRs) through numerical simulations. Moreover, a multi-input-multi-output (MIMO) equalization scheme is proposed and validated with numerical simulation, which can suppress the linear inter-polarization crosstalk and relax the sharpness requirement of optical filter edges.