Sound Source Localization (SSL) involves estimating the Direction of Arrival (DOA) of sound sources. Since the DOA estimation output space is continuous, regression might be more suitable for DOA, offering higher precision. However, in practice, classification often outperforms regression, exhibiting greater robustness. Conversely, classification’s drawback is inherent quantization error. Within the classification paradigm, the DOA output space is discretized into several intervals, each treated as a class. These classes show strong inter-class correlations, being inherently ordered, with higher similarity as intervals grow closer. Nevertheless, this characteristic has not been fully exploited. To address this, we propose Unbiased Label Distribution (ULD) to eliminate quantization error in training targets. Furthermore, we introduce Weighted Adjacent Decoding (WAD) to overcome quantization error during the decoding stage. Finally, we tailor two loss functions for the soft labels: Negative Log Absolute Error (NLAE) and Mean Squared Error without activation (MSE(wo)). Experimental results show our approach surpasses classification quantization limits, achieving state-of-the-art performance. Our code and supplementary material are available at https://github.com/linfeng-feng/ULD.