Neural joint source-channel coding via Bernoulli latent straight-through estimator

Abstract

Under infinite block length, from Shannon's separation theorem, it is well-known that by independent design of source coding and channel coding, the optimal throughput — the channel capacity — can be reached with careful design of the corresponding functionalities. However, when restricted to finite block length, the separation theorem does not necessarily hold, and hence joint source-channel coding (JSCC) theorem has been raised as an alternative strategy for achieving the capacity of the channel. However, as JSCC formulates highly non-convex problem which cannot be directly solved analytically, recently, deep learning has been proposed as a key enabler for JSCC. While most work on deep-learning-based JSCC focused on transmission of continuous signals through wireless channels, we focus in this paper the case of information-theoretic channel, namely binary symmetric channel, which can give useful insights on information-theoretic perspective on JSCC. Unlike recent work on deep-learning-based, or neural, JSCC for discrete channels that considered score function estimator to train the JSCC, in this paper, we improve the performance of neural JSCC by estimating the gradient more precisely as compared to the previous approach. Key idea is to consider soft codeword during training to enable path-wise gradient estimator which is proven to have lower variance than the score-function estimator. Experimental results on MNIST and CIFAR-10 datasets show that the proposed neural JSCC outperforms the previous work on JSCC for discrete channels, validating the effectiveness of the proposed gradient computation technique.