Stochastic Encoding Research Articles

K-Receiver Wiretap Channel: Optimal Encoding Order and Signaling Design

The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -receiver wiretap channel is a channel model where a transmitter broadcasts <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> independent messages to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> intended receivers while keeping them secret from an eavesdropper. The capacity region of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -receiver multiple-input multiple-output (MIMO) wiretap channel has been characterized by using dirty-paper coding and stochastic encoding. However, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> factorial encoding orders may need to be enumerated to evaluate the capacity region, which makes the problem intractable. In addition, even though the capacity region is known, the optimal signaling to achieve the capacity region is unknown. In this paper, we determine one optimal encoding order to achieve every point on the capacity region, and thus reduce the encoding complexity <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> factorial times. We prove that the optimal decoding order for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -receiver MIMO wiretap channel is the same as that for the MIMO broadcast channel without secrecy. To be specific, the descending weight ordering in the weighted sum-rate (WSR) maximization problem determines the optimal encoding order. Next, to reach the border of the secrecy capacity region, we form a WSR maximization problem and apply the block successive maximization method to solve this nonconvex problem and find the input covariance matrices corresponding to each message. Numerical results are used to verify the optimality of the encoding order and to demonstrate the efficacy of the proposed signaling design.

Quadratically Constrained Myopic Adversarial Channels

We study communication in the presence of a jamming adversary where quadratic power constraints are imposed on the transmitter and the jammer. The jamming signal is allowed to be a function of the codebook, and a noncausal but noisy observation of the transmitted codeword. For a certain range of the noise-to-signal ratios (NSRs) of the transmitter and the jammer, we are able to characterize the capacity of this channel under deterministic encoding or stochastic encoding, i.e., with no common randomness between the encoder/decoder pair. For the remaining NSR regimes, we determine the capacity under the assumption of a small amount of common randomness (at most <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\log (n)$ </tex-math></inline-formula> bits in one sub-regime, and at most <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\Omega ({n})$ </tex-math></inline-formula> bits in the other sub-regime) available to the encoder-decoder pair. Our proof techniques involve a novel myopic list-decoding result for achievability, and a Plotkin-type push attack for the converse in a subregion of the NSRs, both of which may be of independent interest. We also give bounds on the strong secrecy capacity of this channel assuming that the jammer is simultaneously eavesdropping.

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

In this work we address the problem of solving ill-posed inverse problems in imaging where the prior is a variational autoencoder (VAE). Specifically we consider the decoupled case where the prior is trained once and can be reused for many different log-concave degradation models without retraining. Whereas previous MAP-based approaches to this problem lead to highly non-convex optimization algorithms, our approach computes the joint (space-latent) MAP that naturally leads to alternate optimization algorithms and to the use of a stochastic encoder to accelerate computations. The resulting technique (JPMAP) performs Joint Posterior Maximization using an Autoencoding Prior. We show theoretical and experimental evidence that the proposed objective function is quite close to bi-convex. Indeed it satisfies a weak bi-convexity property which is sufficient to guarantee that our optimization scheme converges to a stationary point. We also highlight the importance of correctly training the VAE using a denoising criterion, in order to ensure that the encoder generalizes well to out-of-distribution images, without affecting the quality of the generative model. This simple modification is key to providing robustness to the whole procedure. Finally we show how our joint MAP methodology relates to more common MAP approaches, and we propose a continuation scheme that makes use of our JPMAP algorithm to provide more robust MAP estimates. Experimental results also show the higher quality of the solutions obtained by our JPMAP approach with respect to other non-convex MAP approaches which more often get stuck in spurious local optima.

Microbiome-based disease prediction with multimodal variational information bottlenecks.

Scientific research is shedding light on the interaction of the gut microbiome with the human host and on its role in human health. Existing machine learning methods have shown great potential in discriminating healthy from diseased microbiome states. Most of them leverage shotgun metagenomic sequencing to extract gut microbial species-relative abundances or strain-level markers. Each of these gut microbial profiling modalities showed diagnostic potential when tested separately; however, no existing approach combines them in a single predictive framework. Here, we propose the Multimodal Variational Information Bottleneck (MVIB), a novel deep learning model capable of learning a joint representation of multiple heterogeneous data modalities. MVIB achieves competitive classification performance while being faster than existing methods. Additionally, MVIB offers interpretable results. Our model adopts an information theoretic interpretation of deep neural networks and computes a joint stochastic encoding of different input data modalities. We use MVIB to predict whether human hosts are affected by a certain disease by jointly analysing gut microbial species-relative abundances and strain-level markers. MVIB is evaluated on human gut metagenomic samples from 11 publicly available disease cohorts covering 6 different diseases. We achieve high performance (0.80 < ROC AUC < 0.95) on 5 cohorts and at least medium performance on the remaining ones. We adopt a saliency technique to interpret the output of MVIB and identify the most relevant microbial species and strain-level markers to the model’s predictions. We also perform cross-study generalisation experiments, where we train and test MVIB on different cohorts of the same disease, and overall we achieve comparable results to the baseline approach, i.e. the Random Forest. Further, we evaluate our model by adding metabolomic data derived from mass spectrometry as a third input modality. Our method is scalable with respect to input data modalities and has an average training time of < 1.4 seconds. The source code and the datasets used in this work are publicly available.

Role of Shared Key for Secure Communication Over 2-User Gaussian Z-Interference Channel

In this paper, the role of secret key with finite rate is studied to enhance the secrecy performance of the system when users are operating in interference limited scenarios. To address this problem, a 2-user Gaussian Z-interference channel with secrecy constraint at the receiver is considered. The paper proposes novel achievable schemes, where the schemes differ from each other based on how the key has been used in the encoding process. The first achievable scheme uses a combination of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">key rate splitting</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">one-time pad</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic encoding</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">superposition coding</i> . In this scheme, one part of the key is used for one-time pad and the remaining part of the key is used for stochastic encoding. The encoding is performed such that the receiver experiencing interference can decode some part of the interference without violating the secrecy constraint. As a special case of the derived result, one can obtain the secrecy rate region when the key is completely used for one-time pad or part of the stochastic encoding. The second scheme uses the shared key to encrypt the message using one-time pad and in contrast to the previous case no interference is decoded at the receiver. The paper also derives outer bound on the sum rate and secrecy rate. The main novelty of deriving outer bound lies in the selection of side information provided to the receiver and using the secrecy constraint. The derived outer bounds are found to be tight for certain channel conditions and rate of the key. The scaling behaviour of key rate is also explored for different schemes using the notion of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">secure generalized degrees of freedom</i> . The optimality of different schemes are characterized for some specific cases. The developed results show the importance of key rate splitting in enhancing the secrecy performance of the system.

Single-Serving Quantum Broadcast Channel With Common, Individualized, and Confidential Messages

The two-receiver broadcast channel with primary and third party receivers is studied. The sender wishes to reliably communicate a common (or public) message to both receivers as well as individualized and confidential messages to the primary receiver only. The third party receiver must be kept completely ignorant of the confidential message but there are no secrecy requirements associated to the individualized message. A trade-off arises between the rates of the three messages: when one of the rates is high, the other rates may need to back off to guarantee the reliable transmission of all three messages. In addition, the confidentiality requirement implies availability of local randomness at the transmitter in order to implement a stochastic encoding. This article studies the trade-off between the rates of the common, individualized and confidential messages as well as that of the local randomness in the one-shot regime of a quantum broadcast channel. We provide an achievability region, by proving a conditional version of the convex-split lemma combined with the position-based decoding, as well as a (weak) converse region. We study the asymptotic behaviour of our bounds and recover several well-known asymptotic results in the literature, including simultaneous transmission of classical and quantum information.

Analysis and Design of Low-Complexity Stochastic DEM Encoder for Reduced-Distortion Multi-bit DAC in Sigma-Delta Modulators

A low-complexity stochastic dynamic element matching (DEM) technique to mitigate the distortions caused by the nonlinearity of multi-bit digital-to-analog converters (DACs) in sigma-delta modulator is proposed. Multi-bit DAC noise is analyzed in detail from a signal processing point-of-view in this research, and then implementation of the basic stochastic unit named swapper cell which includes bar state and cross state controlled by a 1-bit pseudo-random noise signal, is illustrated. To reduce the hardware complexity, a multi-layer scheme for multi-input multi-output stochastic encoder is presented and demonstrated based on rigorous mathematical derivation. For a n-input n-output version, the number of the switch elements employed in the proposed encoder is reduced to 2nlog2n from the conventional n!. Simulation results show that, with the use of the stochastic encoder, severe harmonic distortions resulted by the DAC elements mismatch errors are suppressed effectively. When the proposed encoders are applied to the internal multi-bit DACs in sigma-delta (ΣΔ) modulators, large improvements of the linearity performance of the modulators are achieved. Additionally, the analyses and explanations for this stochastic scheme provide a theoretical foundation for the future development of noise shaping DEM.

Optimal Parameter Design for Estimation Theoretic Secure Broadcast

In this letter, estimation theoretic secure broadcast of a random parameter is investigated. In the considered setting, each receiver device employs a fixed estimator and carries a certain security risk such that its decision can be available to a malicious third party with a certain probability. The encoder at the transmitter is allowed to use a random mapping to minimize the weighted sum of the conditional Bayes risks of the estimators under secrecy and average power constraints. After formulating the optimal parameter design problem, it is shown that the optimization problem can be solved individually for each parameter value and the optimal mapping at the transmitter involves a randomization among at most three different signal levels. Sufficient conditions for improvability and non-improvability of the deterministic design via stochastic encoding are obtained. Numerical examples are provided to corroborate the theoretical results.

Estimation Theoretic Secure Communication via Encoder Randomization

Estimation theoretic secure transmission of a scalar random parameter is investigated in the presence of an eavesdropper. The aim is to minimize the estimation error at the receiver under a secrecy constraint at the eavesdropper; or, alternatively, to maximize the estimation error at the eavesdropper for a given estimation accuracy limit at the receiver. In the considered setting, the encoder at the transmitter is allowed to use a randomized mapping between two one-to-one and continuous functions and the eavesdropper is fully aware of the encoding strategy at the transmitter. For small numbers of observations, both the eavesdropper and the receiver are modeled to employ linear minimum mean-squared error (LMMSE) estimators, and for large numbers of observations, the expectation of the conditional Cramer-Rao bound (ECRB) metric is employed for both the receiver and the eavesdropper. Optimization problems are formulated and various theoretical results are provided in order to obtain the optimal solutions and to analyze the effects of encoder randomization. In addition, numerical examples are presented to corroborate the theoretical results. It is observed that stochastic encoding can bring significant performance gains for estimation theoretic secrecy problems.

INCREASING THE SECRECY OF TRANSMISSION INFORMATION BASED ON COMBINED RANDOM CODING

Context. The task of increasing the secrecy of information transmission based on combined random coding is considered. Theobject of research is the integration processes of stochastic, block and timer coding.Objective. The aim of the article is to develop a method of increasing the secrecy transmission based on combined random coding.Method. The method of increasing the secrecy of information transmission based on combined random coding with three stepsof code converters: internal, intermediate and external is proposed. The internal converter is implemented based on stochastic coding with various numbers of random code combinations. The number of random combinations depends on the probability of the character of the symbol in the message text. The information secrecy of the transmission is ensured by equalizing the probabilities of the occurrence of code combinations from the output of the statistical coder. Intermediate coding is implemented using a noise immunity block code. Information secrecy and noise immunity leads to the increasing the length of the code block at each stage of coding, which reduces the code rate. To solve this problem, an external converter based on timer coding is used. At this stage, binary combinations are converted into timer signal structures, which allows for the implementation of a structural transmission secrecy, increase the code rate and improve noise immunity. When the stochastic encoding and a timer used codebooks.Results. The developed method of combined random coding allows to increase the main indicators of noise immunity: interferenceprotection, information and structural secrecy.Conclusions. The proposed method of combined random coding allows to eliminate the main disadvantages of stochastic coding,in which the choice of the number of random code combinations does not take into account the entropy of the discrete source of information. For solving this problem, it was proposed to use different number of random combinations, in view of the probability of an alphabet symbol appearing in the text. From the theoretical point of view, this will allow the output of the statistical coder to form a stream of combinations with equal probabilities of occurrence. Due to the usage of timer signal constructions, the problem of redundancy compensation is solved, which appears during stochastic and noise-immune coding.

Channel Code Using Constrained-Random-Number Generator Revisited

A construction of a channel code by using a source code with decoder side information is introduced. The encoder and decoder pair of any source code can be used for the construction. Constrained-random-number generators, which generate random numbers satisfying a condition specified by a function and its value, are used to construct stochastic encoders and decoders. The result suggests that we can divide the channel coding problem into the problems of channel encoding and source decoding with side information.

Stochastic learning in deep neural networks based on nanoscale PCMO device characteristics

Abstract Deep Neural Networks (DNN) have proven to be highly effective in extracting high level abstractions of input data using multiple neural network layers. However, the huge training times for DNNs in traditional von-Neumann machines have hindered their ubiquitous adoption in IoT and other mobile computing platforms. Recently, acceleration of DNN with a time complexity of O ( 1 ) was proposed using the idea of stochastic weight update with resistive processing units (RPU). However, it has been projected that RPU devices require more than 1000 reliable conductance levels, which is a stringent requirement to realize in memristive devices. Here, we study the optimization of stochastic learning for DNNs for the hand-written digit classification benchmark using the characteristics of non-filamentary Pr0.7Ca0.3MnO3 (PCMO) devices that are fabricated using a standard lithography process. The electrical characteristics of these devices exhibit a linear conductance response with an on-off ratio of 1.8 with 26 levels and significant programming variability. We captured these non-ideal behaviors of experimental PCMO device in the simulations to demonstrate stochastic learning with O ( 1 ) time complexity, achieving a test accuracy of 88.1% for the hand-written digit recognition benchmark. While the linearity, dynamic range, bit resolution, programming variability and the reset rate of the device conductance to account for its limited dynamic range have to be co-optimized for improving the training efficiency, we show that programming variability has the paramount role in determining the network performance. We also show that if devices with reduced programming variability (5x smaller compared to our experimental device) can be developed keeping all other parameters constant, it is possible to boost the network performance as high as 95%. We also observe that the performance of stochastic DNNs with memristive synapses is independent of the on-off ratio of the devices for a fixed programming variability. Thus, programming variability represents a new optimization corner for on-chip learning of stochastic DNNs. Further, we also evaluate the performance of stochastic inference engines to noise corrupted input test data as a function of the variability in the memristive devices. We demonstrate that noise-resilient inference engines can be achieved if 100 bits are used for stochastic encoding during inference even though the expensive network training can be done with as few as 10 bits. Thus, our studies emphasize the need for optimization of learning strategies for DNNs based on the non-ideal characteristics of experimental nanoscale devices.

Collective Secrecy Over the &lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$K$ &lt;/tex-math&gt; &lt;/inline-formula&gt;-Transmitter Multiple Access Channel

This paper studies the problem of secure communication over a $K$ -transmitter multiple access channel (MAC) in the presence of an external eavesdropper, subject to a collective secrecy constraint (i.e., information leakage rate to an eavesdropper on a collection of messages that are from a pre-specified subset of the $K$ transmitters, say $\mathcal {S}\subseteq \mathcal {K}=\{1,2,\ldots, K\}$ , is made vanishing). Since secrecy is of concern only to transmitters $\{i|i\in \mathcal {S}\}$ but not to transmitters $\{i|i\in \mathcal {S}^{c}\}$ , where $\mathcal {S}^{c}=\mathcal {K}\backslash \mathcal {S}$ , different transmission strategies could be employed at transmitters $\{i|i\in \mathcal {S}^{c}\}$ . Consider the following two scenarios: 1) transmitters $\{i|i\in \mathcal {S}^{c}\}$ use deterministic encoders (which are conventionally used for MAC without secrecy), competing for the channel resource (i.e., being competitive) and 2) transmitters $\{i|i\in \mathcal {S}^{c}\}$ use stochastic encoders, helping to hide other transmitters’ messages from the eavesdropper (i.e., being cooperative). As a result, we establish the respective $\mathcal {S}$ -collective secrecy achievable rate regions and demonstrate the advantage of being cooperative theoretically and numerically. To this end, in addition to the standard techniques, our results build upon two techniques. The first is a generalization of Chia-El Gamal’s lemma on entropy bound for a set of codewords given partial information. The second is to utilize a compact representation of a list of sets that, together with submodular properties of mutual information functions involved, leads to an efficient Fourier-Motzkin elimination. These two approaches allow us to derive achievable regions in this work, and could also be of independent interest in other context.

A Universal Coding Scheme for Remote Generation of Continuous Random Variables

Alice selects an arbitrary pdf $f$ and uses a stochastic encoder to generate a prefix-free codeword $M$ , which is sent to Bob so that he can generate a single instance of the random variable $X\sim f$ . We describe a universal coding scheme for this setup which works for any $f$ , and establish an upper bound on its expected codeword length when the pdf $f$ is bounded, orthogonally concave (which includes quasiconcave pdf), and has a finite first absolute moment. A dyadic decomposition scheme is used to express the pdf as a mixture of uniform pdfs over hypercubes. Alice randomly selects a hypercube according to its weight, encodes its position and size into $M$ , and sends it to Bob who generates $X$ uniformly over the hypercube. Compared to previous results on channel simulation, our coding scheme applies to any continuous distribution and does not require two-way communication or shared randomness. Applying our coding scheme to classical simulation of quantum entanglement, we obtain a tighter bound on the average codeword length than previously known.

The Deterministic Information Bottleneck.

Lossy compression and clustering fundamentally involve a decision about which features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek ( 1999 ) formalized this notion as an information-theoretic optimization problem and proposed an optimal trade-off between throwing away as many bits as possible and selectively keeping those that are most important. In the IB, compression is measured by mutual information. Here, we introduce an alternative formulation that replaces mutual information with entropy, which we call the deterministic information bottleneck (DIB) and argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem turns out to be a deterministic encoder, or hard clustering, as opposed to the stochastic encoder, or soft clustering, that is optimal under the IB. We compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB significantly outperforms the IB in terms of the DIB cost function. We also empirically find that the DIB offers a considerable gain in computational efficiency over the IB, over a range of convergence parameters. Our derivation of the DIB also suggests a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.

Polar Coding for the Broadcast Channel With Confidential Messages: A Random Binning Analogy

We develop a low-complexity polar coding scheme for the discrete memoryless broadcast channel with confidential messages under strong secrecy and randomness constraints. Our scheme extends previous work by using an optimal rate of uniform randomness in the stochastic encoder, and avoiding assumptions regarding the symmetry or degraded nature of the channels. The price paid for these extensions is that the encoder and the decoders are required to share a secret seed of negligible size and to increase the block length through chaining. We also highlight a close conceptual connection between the proposed polar coding scheme and a random binning proof of the secrecy capacity region.

Variable-Length Lossy Source Code Using a Constrained-Random-Number Generator

A variable-length lossy source code is introduced with a rate-distortion pair close to the rate-distortion function. Random numbers that satisfy a condition specified by a function and its value are used to construct a stochastic encoder. The proof of the theorem is based on the balanced-coloring property of an ensemble of functions. Since an ensemble of systematic sparse matrices has this property, we can construct a tractable code for a memoryless source. Some algorithms for implementing the code are introduced and compared by simulation.

The Optimal Use of Rate-Limited Randomness in Broadcast Channels With Confidential Messages

In coding schemes for the wire-tap channel or for broadcast channels with confidential messages, it is well-known that the sender needs to use stochastic encoding to avoid information about the transmitted confidential message from being leaked to an eavesdropper. In this paper, we investigate the tradeoff between the rate of random numbers needed to realize the stochastic encoding and the rates of common, private, and confidential messages. For the direct theorem, we use the superposition coding scheme for the wire-tap channel, recently proposed by Chia and El Gamal, and its strong security is proved. The matching converse theorem is also established. Our result clarifies that a combination of ordinary stochastic encoding and channel prefixing by channel simulation is suboptimal.

Imperfect Secrecy in Wiretap Channel II

In a point-to-point communication system, which consists of a sender, a receiver, and a set of noiseless channels, the sender wishes to transmit a private message to the receiver through the channels, which may be eavesdropped by a wiretapper. The set of wiretap sets is arbitrary. The wiretapper can access any one but not more than one wiretap set. From each wiretap set, the wiretapper can obtain some partial information about the private message, which is measured by the equivocation of the message given the symbols obtained by the wiretapper. The security strategy is to encode the message with some random key at the sender. Only the message is required to be recovered at the receiver. Under this setting, we define an achievable rate tuple consisting of the size of the message, the size of the key, and the equivocation for each wiretap set. We first prove a tight rate region when both the message and the key are required to be recovered at the receiver. Then, we extend the result to the general case when only the message is required to be recovered at the receiver. Moreover, we show that even if stochastic encoding is employed at the sender, the message rate cannot be increased.

Cognitive Interference Channels With Confidential Messages Under Randomness Constraint

The cognitive interference channel with confidential messages (CICC) proposed by Liang et al. is investigated. When the security is considered in coding systems, it is well-known that the sender needs to use a stochastic encoding to avoid the information about the transmitted confidential message to be leaked to an eavesdropper. For the CICC, the tradeoff between the rate of the random number to realize the stochastic encoding and the communication rates is investigated, and the optimal tradeoff is completely characterized.