Average Codeword Length Research Articles

Novel Generalized Entropy Measure's Characteristics Associated with Code-Word Length

In this article, we introduce a novel, all-purpose entropy measure and derive a new, all-purpose average codeword length in accordance with it. After that, we determined new average generalized codeword length constraints in terms of additional metrics of generalized entropy. In addition, we demonstrate that the metrics employed in this study are generalizations of other metrics typically employed in the coding and information theory communities.

Federated Learning in Massive MIMO 6G Networks: Convergence Analysis and Communication-Efficient Design

In federated learning (FL), model weights must be updated at local users and the base station (BS). These weights are subjected to uplink (UL) and downlink (DL) transmission errors due to the limited reliability of wireless channels. In this paper, we investigate the impact of imperfections in both UL and DL links. First, for a multi-user massive multi-input-multi-output (mMIMO) 6G network, employing zero-forcing (ZF) and minimum mean-squared-error (MMSE) schemes, we analyze the estimation errors of weights for each round. A tighter convergence bound on the modelling error for the communication efficient FL algorithm is derived of the order of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal{O}\left(T^{-1}\sigma_{z}^{2}\right)$</tex-math></inline-formula> , where… <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sigma_z^{2}$</tex-math></inline-formula> denotes the variance of overall communication error including the quantization noise. The analysis shows that the reliability of DL links is more critical than that of UL links; and the transmit power can be varied in training process to reduce energy consumption. We also vary the number of local training steps, average codeword length after quantization and scheduling policy to improve the communication efficiency. Simulations with image classification problems on MNIST, EMNIST and FMNIST datasets verify the derived bound and are useful to infer the minimum SNR required for successful convergence of the FL algorithm.

An entropy minimization histogram mergence scheme and its application in image compression

In this paper, we propose an entropy minimization histogram mergence (EMHM) scheme that can significantly reduce the number of grayscales with nonzero pixel populations (GSNPP) without visible loss to image quality. We proved in theory that the entropy of an image is reduced after histogram mergence and that the reduction in entropy is maximized using our EMHM. The reduction in image entropy is good for entropy encoding considering that the minimum average code word length per source symbol is the entropy of the source signal according to Shannon’s first theorem. Extensive experimental results show that our EMHM can significantly reduce the code length of entropy coding, such as Huffman, Shannon, and arithmetic coding, by over 20% while preserving the image subjective and objective quality very well. Moreover, the performance of some classic lossy image compression techniques, such as the Joint Photographic Experts Group (JPEG), JPEG2000, and Better Portable Graphics (BPG), can be improved by preprocessing images using our EMHM.

Efficient Coding of the Embedded Signal in Steganographic Systems with Multiple Access

Today, steganographic systems with multiple access are of considerable importance. In such sys-tems, the orthogonal Walsh-Hadamard transform is most often used for multiplexing and divid-ing channels, which leads to the need for efficient coding of the Walsh-Hadamard transform coefficients for the convenience of their subsequent embedding. The purpose of the research is to develop a theoretical basis for efficient coding of the embedded signal in steganographic sys-tems with multiple access with an arbitrary number of users N, based on MC-CDMA technology. This purpose was fulfilled by forming the theoretical basis for constructing effective codes de-signed to encode the embedded signal in steganographic systems with multiple access. The most important results obtained are the proposed and proven relations that determine both the possible values of the Walsh-Hadamard transform coefficients, for a given value of the number of divid-ed channels, and the probability of occurrence of the given values of the Walsh-Hadamard transform coefficients, which allow the construction of effective codes to represent the embed-ded signal. In the case of the number of divided channels N=4, we propose to use a constant amplitude code that provides a smaller value of the average codeword length in comparison with the Huffman code, while the constructed code has correcting capabilities. The significance of the obtained results is determined by the possibility of using the developed theoretical basis when constructing effective codes for encoding the embedded signal in steganographic systems with multiple access at an arbitrary value of the number of divided channels N.

Notice of Removal: A Novel Representation for Permutations

In this paper, we present a variable-length binary code for permutations of degree n, where n is a power of 2. The Lehmer code and its variants provide a bijection between permutations and the indexes in lexicographic ordering. They provide compression of permutations approaching Shannon bound at the expense of structural information. The variable-length code proposed in this paper has asymptotically optimal compression capability while preserving structural information of permutations. In other words, the code encodes the way a permutation is organized, and is optimal in the sense that the ratio of average codeword length and the entropy of permutations approaches 1 as n tends to infinity. The encoding and decoding are efficient. Like the Lehmer code and other enumerative codes, it is not necessary to construct look-up tables. The code is complete and satisfies the prefix condition. Furthermore, the codeword length indicates how “well shuffled” a permutation is. Permutations with longer codewords seem more “random.” An enumeration method of the variable-length code is also given by using T. Cover's enumerative coding scheme and Schalkwijk code. This gives a different indexing from the Lehmer code. This work can be extended to the case of arbitrary n in a straightforward way.

On the Design of Near-Optimal Variable-Length Error-Correcting Codes for Large Source Alphabets

In this paper, we focus on near-optimal designs for variable-length error-correcting (VLEC) codes, while considering the performance in terms of both the average codeword length (ACL) and the symbol-error rate (SER). The criteria for narrowing the search space and enhancing the SER performance are investigated. An efficient code construction algorithm is then devised based on a random search process. Taking advantage of the significantly reduced search complexity, we are able to construct near-optimal VLEC codes for large source alphabets. Numerical results obtained for various free distance values and alphabet sizes show that constructed VLEC codes have both reduced ACL values and enhanced SER performances. Performance improvement is more notable for large source alphabets.

Variable-Length Source Dispersions Differ Under Maximum and Average Error Criteria

Variable-length compression without prefix-free constraints and with side-information available at both encoder and decoder is considered. Instead of requiring the code to be error-free, we allow for it to have a non-vanishing error probability. We derive one-shot bounds on the optimal average codeword length by proposing two new information quantities; namely, the conditional and unconditional $\varepsilon$-cutoff entropies. Using these one-shot bounds, we obtain the second-order asymptotics of the problem under two different formalisms---the average and maximum probabilities of error over the realization of the side-information. While the first-order terms in the asymptotic expansions for both formalisms are identical, we find that the source dispersion under the average error formalism is, in most cases, strictly smaller than its maximum error counterpart. Applications to a certain class of guessing problems, previously studied by Kuzuoka [\emph{{IEEE} Trans.\ Inf.\ Theory}, vol.~66, no.~3, pp.~1674--1690, 2020], are also discussed.

An Importance Aware Weighted Coding Theorem Using Message Importance Measure

There are numerous scenarios in source coding where not only the code length but the importance of each value should also be taken into account. Different from the traditional coding theorems, by adding the importance weights for the length of the codes, we define the average cost of the weighted codeword length as an importance-aware measure of the codes. This novel information theoretical measure generalizes the average codeword length by assigning importance weights for each symbol according to users' concerns through focusing on user's selections. With such definitions, coding theorems of the bounds are derived and the outcomes are shown to be extensions of traditional coding theorems.

A New Generalized Entropy Measure and its Properties

 In this research article, a new two parametric measure of Entropy  and its corresponding code word length  has been developed. The developed measures are the generalizations to some well known existing measures. Besides, some noiseless coding theorems for discrete noiseless channel have been developed, and the results thus obtained have been verified with the support of an numerical example. Also, at the end of this research article, a comparative study in terms of  monotonic behavior  among the proposed entropy  , Matahi’s entropy  and Tsallis entropy  together  with their respective average code word length measures have been made and graphically displayed.

Optimality in quantum data compression using dynamical entropy

In this article we study lossless compression of strings of pure quantum states of indeterminate-length quantum codes which were introduced by Schumacher and Westmoreland. Past work has assumed that the strings of quantum data are prepared to be encoded in an independent and identically distributed way. We discuss the notion of quantum stochastic ensembles, allowing us to consider strings of quantum states prepared in a general way. For any quantum stochastic ensemble we define an associated quantum dynamical system and prove that the optimal average codeword length via lossless coding is equal to the quantum dynamical entropy of the associated quantum dynamical system.

Variable-Length Coding With Shared Incremental Redundancy: Design Methods and Examples

Variable-length (VL) coding with feedback is a commonly used technique that can approach point-to-point Shannon channel capacity with a significantly shorter average codeword length than fixed-length coding without feedback. This paper uses the inter-frame coding of Zeineddine and Mansour, originally introduced to address varying channel-state conditions in broadcast wireless communication, to approach capacity on point-to-point channels using VL codes without feedback. The per-symbol complexity is comparable to decoding the VL code with feedback (plus the additional complexity of a small peeling decoder amortized over many VL codes) and presents the opportunity for encoders and decoders that utilize massive parallel processing, where each VL decoder can process simultaneously. This paper provides an analytical framework and a design process for the degree distribution of the inter-frame code that allows the feedback-free system to achieve 96% or more of the throughput of the original VL code with feedback. As examples of VL codes, we consider non-binary (NB) low-density parity-check (LDPC), binary LDPC, and convolutional VL codes. The NB-LDPC VL code with an 8-bit CRC and an average codeword length of 336 bits achieves 85% of capacity with four rounds of ACK/NACK feedback. The proposed scheme using shared incremental redundancy without feedback achieves 97% of that performance or 83% of the channel capacity.

A Low-Latency Transmission Method for Massive MIMO Systems With Low Control Signaling Overhead

In this paper, we consider the uplink of massive MIMO systems where the channel capacity is almost deterministic and varies slowly over time. We exploit this fact, by appropriately choosing the length of the codeword only once based on an initial estimate of the uplink channel capacity fed back by the base station (BS). The user terminal (UT) then transmits the codeword over consecutive coherence time intervals without waiting for any acknowledgement (ACK/NACK) from the BS. The BS sends only a single ACK/NACK after it completes decoding. Therefore, the proposed method has significantly less latency and signaling overhead when compared with stop-and-wait type hybrid automatic repeat request (HARQ) methods (as in LTE uplink) where a lot of delay is introduced as the UT needs to wait for the BS to signal the ACK/NACK for every retransmission. Analysis reveals that for sufficiently large number of antennas at the BS, the average codeword length of the proposed method (in number of coherence intervals) is the same as the average number of retransmissions (in number of coherence intervals) required by incremental redundancy (IR)-based HARQ and also that the energy efficiency of the proposed method and IR-HARQ is the same.

An Efficient Construction Strategy for Near-Optimal Variable-Length Error-Correcting Codes

In this letter, we present an efficient cross-entropy (CE)-based algorithm for the design of variable-length error-correcting (VLEC) codes under the joint source and channel coding (JSCC) framework. The algorithm enables us to construct the near-optimal VLEC codes that have the minimum average codeword length (ACL) with low search complexity. The efficiency of the proposed CE-based algorithm makes it possible to construct the VLEC codes that have small ACL values under various free-distance constraints, especially for large-sized signal alphabets.

A Universal Coding Scheme for Remote Generation of Continuous Random Variables

Alice selects an arbitrary pdf $f$ and uses a stochastic encoder to generate a prefix-free codeword $M$ , which is sent to Bob so that he can generate a single instance of the random variable $X\sim f$ . We describe a universal coding scheme for this setup which works for any $f$ , and establish an upper bound on its expected codeword length when the pdf $f$ is bounded, orthogonally concave (which includes quasiconcave pdf), and has a finite first absolute moment. A dyadic decomposition scheme is used to express the pdf as a mixture of uniform pdfs over hypercubes. Alice randomly selects a hypercube according to its weight, encodes its position and size into $M$ , and sends it to Bob who generates $X$ uniformly over the hypercube. Compared to previous results on channel simulation, our coding scheme applies to any continuous distribution and does not require two-way communication or shared randomness. Applying our coding scheme to classical simulation of quantum entanglement, we obtain a tighter bound on the average codeword length than previously known.

A New Approach to Video Coding Using 3D Directional Filter Bank

The high resolution 3-D and higher dimensional volumetric data are increasingly used in a wide variety of applications including biomedical imaging, video processing and compression. Multidimensional filter banks and its associated constructions provide a framework and serve as an efficient computational tool in the formation, representation, and processing of multidimensional data sets. This paper focuses on the construction of 3-D directional filter banks (DFB) and its signal representations for scalable video coding. In this paper we consider five such DFBs and analyse the performance to find the best suitable one for scalable video coding. The five different filters includes: Maxflat filter, Vk filter, kos filter, Sinc filter and Haar filter. The performance evaluation has been done which quantifies the factors like Peak Signal to Noise Ratio (PSNR) and Mean Square Error (MSE). In order to achieve better compression, a comparative study between RLC and Huffman coding schemes has been performed and the average codeword length is measured and tabulated. Among the five filters and two compression scheme it is observed that Kos filters with Huffman coding provide better compression and low Average codeword length.

The Most Likely Optimal Symmetric RVLC

For each alphabet size up to 30, a symmetric reversible variable length code (RVLC) is determined which is most likely to be optimal in average codeword length. It is shown that these codes are optimal for more than half of all the possible sources.

Efficient and Compact Representations of Prefix Codes

Most of the attention in statistical compression is given to the space used by the compressed sequence, a problem completely solved with optimal prefix codes. However, in many applications, the storage space used to represent the prefix code itself can be an issue. In this paper, we introduce and compare several techniques to store prefix codes. Let N be the sequence length and n be the alphabet size. Then, a naive storage of an optimal prefix code uses O(n log n) bits. Our first technique shows how to use O(n log log(N/n)) bits to store the optimal prefix code. Then, we introduce an approximate technique that, for any 0 <; ε <; 1/2, takes O(n log log(1/E)) bits to store a prefix code with an average codeword length within an additive ε of the minimum. Finally, a second approximation takes, for any constant c > 1, O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/c</sup> log n) bits to store a prefix code with an average codeword length at most c times the minimum. In all cases, our data structures allow encoding and decoding of any symbol in O(1) time. We experimentally compare our new techniques with the state of the art, showing that we achieve sixfold-to-eightfold space reductions, at the price of a slower encoding (2.5-8 times slower) and decoding (12-24 times slower). The approximations further reduce this space and improve the time significantly, up to recovering the speed of classical implementations, for a moderate penalty in the average code length. As a byproduct, we compare various heuristic, approximate, and optimal algorithms to generate length-restricted codes, showing that the optimal ones are clearly superior and practical enough to be implemented.

Comparative Study of Generalized Quantitative-Qualitative Inaccuracy Fuzzy Measures for Noiseless Coding Theorem and 1:1 Codes

In coding theory, we study various properties of codes for application in data compression, cryptography, error correction, and network coding. The study of codes is introduced in Information Theory, electrical engineering, mathematics, and computer sciences for the transmission of data through reliable and efficient methods. We have to consider how coding of messages can be done efficiently so that the maximum number of messages can be sent over a noiseless channel in a given time. Thus, the minimum value of mean codeword length subject to a given constraint on codeword lengths has to be founded. In this paper, we have introduced mean codeword length of orderαand typeβfor 1:1 codes and analyzed the relationship between average codeword length and fuzzy information measures for binary 1:1 codes. Further, noiseless coding theorem associated with fuzzy information measure has been established.

Energy-Efficient Prefix-Free Codes for Wireless Nano-Sensor Networks Using OOK Modulation

Wireless nano-sensor networks (WNSNs), which consist of nano-sensors a few hundred nanometers in size with the capability to detect and sense new types of events in nano-scale, are promising for many unique applications such as air pollution surveillance. The nano-sensors of WNSNs are highly energy-constrained, which makes it essential to develop energy-efficient communication techniques in such networks. In this paper, we focus on WNSNs employing on-off keying (OOK) modulation, whereby transmission energy minimization corresponds to the minimization of average codeword weight (ACW). We formulate an integer nonlinear programming problem to construct prefix-free codes with minimum ACW under the constraint of average codeword length (ACL) so as to minimize the transmission energy consumption while guaranteeing the throughput larger than a preset desired value. In addition, two efficient algorithms, called binary tree based weight decreasing (BT-WD) algorithm and binary tree based length decreasing (BT-LD) algorithm, are presented for constructing low-ACW prefix-free codes. The effectiveness of the proposed algorithms is verified through simulations and comparisons with the exhaustive search method. Compared with the available fixed-length low-weight codes, the designed prefix-free variable-length codes allow us to not only control the throughput more flexibly but also achieve lower transmission energy consumption in the scenarios with low or medium bit error rates.

Learning olfactory codes using matrix factorization on 2DG uptake patterns from rats

Recent studies based on both the analysis of odor descriptors [1] and psychophysical studies [2] have suggested that olfactory perception is mainly dominated by odor pleasantness. In contrast, findings on the sensory side [3] as well as on the perceptual side [4] indicate an odor perception space that is spanned by about 30 dimensions. On the way to understand the basic principles in the decoding of olfactory information from the single molecular compounds, we are focusing in this study on the first level of information processing in the sense of smell: The olfactory bulb (OB). We know that the olfactory epithelium is separated into four regions [5] of olfactory receptor protein expression and we expect a similar organization in the olfactory bulb [6]. With 2-DG-uptake-images, Johnson et al. [7] measured odorant-specific activity patterns, covering 98% of the bulbar surface with a resolution of one glomerulus per pixel. Using a database of 143 such bulbar activity fingerprints, we propose the so called non-negative matrix factorization [8] to decompose the data into a set of common features. These features can be best described as the coding alphabet while the uptake images of single molecular compounds resemble natural occurring code words. Interestingly, we can achieve three meaningful alphabets with a different size: 4, 6, and 9 digits. Only the last one is suitable for actually coding the compounds, as a single compound is evoking activity of only about 1/8 of the bulbar surface, with an average code word length of two to three digits. The other two decompositions might reflect the superglomerular organization of the receptor neurons in the bulb, like e.g. the above mentioned expression patterns and neuronal subnetworks that directly result from the targeting mechanisms guiding the neurons from the OE to the OB. We argue that the proposed decompositions are not only reflecting the chemotopic organization of the bulbus but rather giving a first hint on the coding scheme underlying olfactory perception.