A Polarimetric Synthetic Aperture Radar (PolSAR) sensor is able to collect images in different polarization states, making it a rich source of information for target characterization. PolSAR images are inherently affected by speckle. Therefore, before deriving ad hoc products from the data, the polarimetric covariance matrix needs to be estimated by reducing speckle. In recent years, deep learning based despeckling methods have started to evolve from single channel SAR images to PolSAR images. To this aim, deep learning based approaches separate the real and imaginary components of the complex-valued covariance matrix and use them as independent channels in a standard convolutional neural networks. However, this approach neglects the mathematical relationship that exists between the real and imaginary components, resulting in sub-optimal output. Here, we propose a multi-stream complex-valued fully convolutional network to reduce speckle and effectively estimate the PolSAR covariance matrix. To evaluate the performance of CV-deSpeckNet, we used Sentinel-1 dual polarimetric SAR images to compare against its real-valued counterpart, that separates the real and imaginary parts of the complex covariance matrix. CV-deSpeckNet was also compared against the state of the art PolSAR despeckling methods. The results show CV-deSpeckNet was able to be trained with a fewer number of samples, has a higher generalization capability and resulted in a higher accuracy than its real-valued counterpart and state-of-the-art PolSAR despeckling methods. These results showcase the potential of complex-valued deep learning for PolSAR despeckling.