Carrier-assisted differential detection (CADD) is a promising solution for high-capacity and cost-sensitive short-reach application scenarios, in which the optical field of a complex-valued double-sideband (CV-DSB) signal is reconstructed without using a local oscillator laser. In this work, we propose a polarization division multiplexed asymmetric twin single-sideband CADD (PDM-ATSSB CADD) scheme to realize the optical field recovery of the PDM CV-DSB signals. The polarization fading is solved by using a pair of optical bandpass filters (OBPFs) to suppress the unwanted other polarized offset carrier and signal, and the dual-polarization optical field is recovered by the CADD receiver. An asymmetric twin-SSB signal is used to relax the sharpness requirement of optical filter edges. We also propose a joint signal-signal beat interference (SSBI) iterative mitigation algorithm, which can effectively mitigate intra- and inter-polarization SSBI. The proposed PDM-ATSSB CADD scheme is validated for 30 Gbaud PDM asymmetric twin-SSB 16-ary quadrature amplitude modulation signals. We illustrate the parameter optimization process for PDM-ATSSB CADD, including optical delay and the number of iterations. The impacts of phase noise and relative intensity noise for laser and polarization impairment on PDM-ATSSB CADD are evaluated through numerical simulation. Compared with the PDM-symmetric-TSSB CADD (PDM-STSSB CADD), the required frequency gap to reach the 7% FEC threshold can be reduced by 1 GHz, and the OSNR sensitivity is improved by about 3 dB. Moreover, two simplified PDM-ATSSB CADD schemes are proposed and discussed, which could be considered hardware-efficient and integrative candidates for metro and inter-data center interconnects.