External Memory Research Articles

MoViT: Memorizing Vision Transformers for Medical Image Analysis.

The synergy of long-range dependencies from transformers and local representations of image content from convolutional neural networks (CNNs) has led to advanced architectures and increased performance for various medical image analysis tasks due to their complementary benefits. However, compared with CNNs, transformers require considerably more training data, due to a larger number of parameters and an absence of inductive bias. The need for increasingly large datasets continues to be problematic, particularly in the context of medical imaging, where both annotation efforts and data protection result in limited data availability. In this work, inspired by the human decision-making process of correlating new "evidence" with previously memorized "experience", we propose a Memorizing Vision Transformer (MoViT) to alleviate the need for large-scale datasets to successfully train and deploy transformer-based architectures. MoViT leverages an external memory structure to cache history attention snapshots during the training stage. To prevent overfitting, we incorporate an innovative memory update scheme, attention temporal moving average, to update the stored external memories with the historical moving average. For inference speedup, we design a prototypical attention learning method to distill the external memory into smaller representative subsets. We evaluate our method on a public histology image dataset and an in-house MRI dataset, demonstrating that MoViT applied to varied medical image analysis tasks, can outperform vanilla transformer models across varied data regimes, especially in cases where only a small amount of annotated data is available. More importantly, MoViT can reach a competitive performance of ViT with only 3.0% of the training data. In conclusion, MoViT provides a simple plug-in for transformer architectures which may contribute to reducing the training data needed to achieve acceptable models for a broad range of medical image analysis tasks.

Utilizing a feature-aware external memory network for helpfulness prediction in e-commerce reviews

With the explosive growth of reviews on e-commerce websites, consumers usually have trouble finding helpful information in massive reviews. Therefore, predicting review helpfulness has drawn the attention of researchers in the research community. Previous works have focused on identifying the helpfulness of reviews based on content, but we found that the helpfulness of a review is determined by the content of reviews and the matching degree between items and reviews. To meet this challenge, this paper proposes a deep matching model based on a feature-aware external memory network (DMMN) to predict review helpfulness. Specifically, the DMMN contains a memory matrix that can store typical features of items via memory reading/writing operations. We then derive item features by properly reading the memory matrix. Meanwhile, a self-attention mechanism is adopted to obtain review features. Finally, the helpfulness score of the review is acquired by the matching degree between item features and review features. Experiments conducted on Amazon datasets, compared with the baseline models, the DMMN obtains an about 1–2% performance improvement on the eight datasets. The experimental results demonstrate that our efforts to predict helpful reviews are feasible and effective.

SmartDeal: Remodeling Deep Network Weights for Efficient Inference and Training.

The record-breaking performance of deep neural networks (DNNs) comes with heavy parameter budgets, which leads to external dynamic random access memory (DRAM) for storage. The prohibitive energy of DRAM accesses makes it nontrivial for DNN deployment on resource-constrained devices, calling for minimizing the movements of weights and data in order to improve the energy efficiency. Driven by this critical bottleneck, we present SmartDeal, a hardware-friendly algorithm framework to trade higher-cost memory storage/access for lower-cost computation, in order to aggressively boost the storage and energy efficiency, for both DNN inference and training. The core technique of SmartDeal is a novel DNN weight matrix decomposition framework with respective structural constraints on each matrix factor, carefully crafted to unleash the hardware-aware efficiency potential. Specifically, we decompose each weight tensor as the product of a small basis matrix and a large structurally sparse coefficient matrix whose nonzero elements are readily quantized to the power-of-2. The resulting sparse and readily quantized DNNs enjoy greatly reduced energy consumption in data movement as well as weight storage, while incurring minimal overhead to recover the original weights thanks to the required sparse bit-operations and cost-favorable computations. Beyond inference, we take another leap to embrace energy-efficient training, by introducing several customized techniques to address the unique roadblocks arising in training while preserving the SmartDeal structures. We also design a dedicated hardware accelerator to fully utilize the new weight structure to improve the real energy efficiency and latency performance. We conduct experiments on both vision and language tasks, with nine models, four datasets, and three settings (inference-only, adaptation, and fine-tuning). Our extensive results show that 1) being applied to inference, SmartDeal achieves up to 2.44x improvement in energy efficiency as evaluated using real hardware implementations and 2) being applied to training, SmartDeal can lead to 10.56x and 4.48x reduction in the storage and the training energy cost, respectively, with usually negligible accuracy loss, compared to state-of-the-art training baselines. Our source codes are available at: https://github.com/VITA-Group/SmartDeal.

Ferroelectric-defined reconfigurable homojunctions for in-memory sensing and computing.

Recently, the increasing demand for data-centric applications is driving the elimination of image sensing, memory and computing unit interface, thus promising for latency- and energy-strict applications. Although dedicated electronic hardware has inspired the development of in-memory computing and in-sensor computing, folding the entire signal chain into one device remains challenging. Here an in-memory sensing and computing architecture is demonstrated using ferroelectric-defined reconfigurable two-dimensional photodiode arrays. High-level cognitive computing is realized based on the multiplications of light power and photoresponsivity through the photocurrent generation process and Kirchhoff's law. The weight is stored and programmed locally by the ferroelectric domains, enabling 51 (>5 bit) distinguishable weight states with linear, symmetric and reversible manipulation characteristics. Image recognition can be performed without any external memory and computing units. The three-in-one paradigm, integrating high-level computing, weight memorization and high-performance sensing, paves the way for a computing architecture with low energy consumption, low latency and reduced hardware overhead.

Memory capacity of recurrent neural networks with matrix representation

It is well known that canonical recurrent neural networks (RNNs) face limitations in learning long-term dependencies, which have been addressed by memory structures in long short-term memory (LSTM) networks. Neural Turing machines (NTMs) are a variant of RNNs that implement the notion of programmable memory with neural network controllers that can learn simple algorithmic tasks. Matrix neural networks, on the other hand, feature matrix representations which inherently has the potential to preserve the spatial structure of data when compared to canonical neural networks that use only vector-based representation. One may then argue that neural networks with matrix representations may have the potential to provide better memory capacity. In this paper, we define and study a probabilistic notion of memory capacity based on Fisher information for matrix-based RNNs. We find bounds on memory capacity for such networks under various hypotheses and compare them with their conventional (vector) counterparts. In particular, we show that the memory capacity of such networks is bounded by N2, for N×N state matrix which generalizes and provides similar results for vector networks. We also show and analyze the increase in memory capacity for such networks which is introduced when one exhibits an external state memory, such as NTMs. Consequently, we construct NTMs with RNN controllers with matrix-based representation of external memory, leading us to introduce Matrix NTMs. We demonstrate the performance of this class of memory networks under certain algorithmic learning tasks such as copying and recall and compare it with Matrix RNNs. We find an improvement in the performance of Matrix NTMs by the addition of external memory, in comparison to Matrix RNNs.

AoCStream: All-on-Chip CNN Accelerator with Stream-Based Line-Buffer Architecture and Accelerator-Aware Pruning.

Convolutional neural networks (CNNs) play a crucial role in many EdgeAI and TinyML applications, but their implementation usually requires external memory, which degrades the feasibility of such resource-hungry environments. To solve this problem, this paper proposes memory-reduction methods at the algorithm and architecture level, implementing a reasonable-performance CNN with the on-chip memory of a practical device. At the algorithm level, accelerator-aware pruning is adopted to reduce the weight memory amount. For activation memory reduction, a stream-based line-buffer architecture is proposed. In the proposed architecture, each layer is implemented by a dedicated block, and the layer blocks operate in a pipelined way. Each block has a line buffer to store a few rows of input data instead of a frame buffer to store the whole feature map, reducing intermediate data-storage size. The experimental results show that the object-detection CNNs of MobileNetV1/V2 and an SSDLite variant, widely used in TinyML applications, can be implemented even on a low-end FPGA without external memory.

Polyomino: A 3D-SRAM-Centric Accelerator for Randomly Pruned Matrix Multiplication With Simple Reordering Algorithm and Efficient Compression Format in 180-nm CMOS

We have developed a sparse matrix reordering algorithm with a novel 3D-SRAM-centric Polyomino accelerator that enables efficient processing of the reordered matrix for parameter compression. By reordering randomly pruned, irregularly structured sparse matrices into regularly structured matrices, both the compression ratio of the data and the efficiency of the hardware processing increase. The reordering algorithm can be implemented simply by attributing it to the widely known k-sum problem. We also developed a compression format for storing the reordered matrices and show that the reordered regular structure can reduce the amount of required memory by 63% compared with the conventional method. The proposed Polyomino accelerator can efficiently process reordered matrices by using a 3D stacked SRAM, which is an external memory with random accessibility and low latency. The measurement results using a test chip fabricated in a 180-nm CMOS process demonstrate that the proposed accelerator can achieve high area-efficiency and high energy-efficiency and scales well with the pruning rate.

A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware

Spiking neural networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANNs): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate BP using spike-time dependent plasticity with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based back-prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts BP weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly resistive processing units. Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a winner take all rule. Our SNN with a two-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

A 22-pJ/spike 73-Mspikes/s 130k-compartment neural array transceiver with conductance-based synaptic and membrane dynamics.

Neuromorphic cognitive computing offers a bio-inspired means to approach the natural intelligence of biological neural systems in silicon integrated circuits. Typically, such circuits either reproduce biophysical neuronal dynamics in great detail as tools for computational neuroscience, or abstract away the biology by simplifying the functional forms of neural computation in large-scale systems for machine intelligence with high integration density and energy efficiency. Here we report a hybrid which offers biophysical realism in the emulation of multi-compartmental neuronal network dynamics at very large scale with high implementation efficiency, and yet with high flexibility in configuring the functional form and the network topology. The integrate-and-fire array transceiver (IFAT) chip emulates the continuous-time analog membrane dynamics of 65 k two-compartment neurons with conductance-based synapses. Fired action potentials are registered as address-event encoded output spikes, while the four types of synapses coupling to each neuron are activated by address-event decoded input spikes for fully reconfigurable synaptic connectivity, facilitating virtual wiring as implemented by routing address-event spikes externally through synaptic routing table. Peak conductance strength of synapse activation specified by the address-event input spans three decades of dynamic range, digitally controlled by pulse width and amplitude modulation (PWAM) of the drive voltage activating the log-domain linear synapse circuit. Two nested levels of micro-pipelining in the IFAT architecture improve both throughput and efficiency of synaptic input. This two-tier micro-pipelining results in a measured sustained peak throughput of 73 Mspikes/s and overall chip-level energy efficiency of 22 pJ/spike. Non-uniformity in digitally encoded synapse strength due to analog mismatch is mitigated through single-point digital offset calibration. Combined with the flexibly layered and recurrent synaptic connectivity provided by hierarchical address-event routing of registered spike events through external memory, the IFAT lends itself to efficient large-scale emulation of general biophysical spiking neural networks, as well as rate-based mapping of rectified linear unit (ReLU) neural activations.

Improving Structured Grid-Based Sparse Matrix-Vector Multiplication and Gauss–Seidel Iteration on GPDSP

Structured grid-based sparse matrix-vector multiplication and Gauss–Seidel iterations are very important kernel functions in scientific and engineering computations, both of which are memory intensive and bandwidth-limited. GPDSP is a general purpose digital signal processor, which is a very significant embedded processor that has been introduced into high-performance computing. In this paper, we designed various optimization methods, which included a blocking method to improve data locality and increase memory access efficiency, a multicolor reordering method to develop Gauss–Seidel fine-grained parallelism, a data partitioning method designed for GPDSP memory structures, and a double buffering method to overlap computation and access memory on structured grid-based SpMV and Gauss–Seidel iterations for GPDSP. At last, we combined the above optimization methods to design a multicore vectorization algorithm. We tested the matrices generated with structured grids of different sizes on the GPDSP platform and obtained speedups of up to 41× and 47× compared to the unoptimized SpMV and Gauss–Seidel iterations, with maximum bandwidth efficiencies of 72% and 81%, respectively. The experiment results show that our algorithms could fully utilize the external memory bandwidth. We also implemented the commonly used mixed precision algorithm on the GPDSP and obtained speedups of 1.60× and 1.45× for the SpMV and Gauss–Seidel iterations, respectively.

External Slot Relationship Memory for Multi-Domain Dialogue State Tracking

Dialogue state tracking is an essential component in multi-domain dialogue systems that aims to accurately determine the current dialogue state based on the dialogue history. Existing research has addressed the issue of multiple mappings in dialogues by employing slot self-attention as a data-driven approach. However, learning the relationships between slots from a single sample often has limitations and may introduce noise and have high time complexity issues. In this paper, we propose an external slot relation memory-based dialogue state tracking model (ER-DST). By utilizing external memory storage, we learn the relationships between slots as a dictionary of multi-domain slot relations. Additionally, we employ a small filter to discard slot information irrelevant to the current dialogue state. Our method is evaluated on MultiWOZ 2.0 and MultiWOZ 2.1, achieving improvements of 0.23% and 0.39% over the baseline models, respectively, while reducing the complexity of the slot relationship learning component to O(n).

Age-related differences in memory when offloading important information.

People can choose to use external memory aids and offload information to help them remember it, but it is unclear how objective and subjective value or importance influence offloading decisions in younger and older adults. We presented younger adults (n = 99; age range: 18-31) and older adults (n = 93; age range: 60-96) with items to remember for a later test and allowed them to offload a subset of the presented items. In Experiment 1, the to-be-remembered information was lists of associated words paired with point values counting toward participants' scores if recalled. In Experiment 2, the to-be-remembered information was lists of items along a theme, such as packing for vacation, which differed in subjective value. Results revealed that when words were paired with objective point values, younger adults were more selective in their offloading decisions and subsequent recall than older adults (i.e., younger adults were more likely to offload and recall high-value items than low-value items relative to older adults). When the to-be-remembered items instead differed in subjective value, older adults were more selective in their offloading decisions than younger adults. Specifically, older adults were more likely to offload words they rated as important relative to items they rated as less important while younger adults displayed the opposite pattern-younger adults were more likely to offload words they rated as less important compared with items they rated as more important. This difference in offloading tendencies when to-be-remembered information varies in subjective value may be indicative of older adults engaging in a form of metacognitive control that can help ensure the use of responsible remembering. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Generating synthetic clinical data that capture class imbalanced distributions with generative adversarial networks: Example using antiretroviral therapy for HIV

Objective:Clinical data’s confidential nature often limits the development of machine learning models in healthcare. Generative adversarial networks (GANs) can synthesise realistic datasets, but suffer from mode collapse, resulting in low diversity and bias towards majority demographics and common clinical practices. This work proposes an extension to the classic GAN framework that includes a variational autoencoder (VAE) and an external memory mechanism to overcome these limitations and generate synthetic data accurately describing imbalanced class distributions commonly found in clinical variables. Methods:The proposed method generated a synthetic dataset related to antiretroviral therapy for human immunodeficiency virus (ART for HIV). We evaluated it based on five metrics: (1) accurately representing imbalanced class distribution; (2) the realism of the individual variables; (3) the realism among variables; (4) patient disclosure risk; and (5) the utility of the generated dataset for developing downstream machine learning models. Results:The proposed method overcomes the issue of mode collapse and generates a synthetic dataset that accurately describes imbalanced class distributions commonly found in clinical variables. The generated data has a patient disclosure risk of 0.095%, lower than the 9% threshold stated by Health Canada and the European Medicines Agency, making it suitable for distribution to the research community with high security. The generated data also has high utility, indicating the potential of the proposed method to enable the development of downstream machine learning algorithms for healthcare applications using synthetic data. Conclusion:Our proposed extension to the classic GAN framework, which includes a VAE and an external memory mechanism, represents a promising approach towards generating synthetic data that accurately describe imbalanced class distributions commonly found in clinical variables. This method overcomes the limitations of GANs and creates more realistic datasets with higher patient cohort diversity, facilitating the development of downstream machine learning algorithms for healthcare applications.

Preliminary Evidence for Dementia Collaborative Coaching.

The primary purpose of this study was to obtain preliminary evidence for a communication coaching intervention, Dementia Collaborative Coaching. The secondary aim of this study was to assess the acceptability, appropriateness, and feasibility of the intervention according to routine care providers. In a pre-/posttest design, speech-language pathologists (SLPs) delivered Dementia Collaborative Coaching to certified nursing assistants (CNAs) and people living with dementia (PLWD) in six different skilled nursing facilities over a period of 6 weeks. A self-perceived knowledge and efficacy measure regarding the use of external memory aids to support communication in PLWD was administered to CNA and SLP participants. The Cohen-Mansfield Agitation Inventory was administered to PLWD participants. The Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure were administered post-intervention. For CNAs, self-perceived knowledge and efficacy increased from pre-intervention (M = 3.73, SD = 0.69) to post-intervention (M = 4.07, SD = 0.44), t(11) = -1.97, one-sided p = .037. There was a significant improvement (e.g., reduction) in scores on the Cohen-Mansfield Agitation Inventory for PLWD (n = 10) from pre-intervention (M = 73.10, SD = 29.98) to post-intervention (M = 58.6, SD = 18.82), t(9) = 2.83, p = .01. CNA participants (n = 12) rated the intervention as acceptable (M = 4.48, SD = 0.48), appropriate (M = 4.33, SD = 0.61), and feasible (M = 4.19, SD = 0.48). SLPs rated the intervention as slightly more acceptable, appropriate, and feasible than CNAs with scores of M = 4.54, SD = 0.51; M = 4.54, SD = 0.51; and M = 4.46, SD = 0.51, respectively. Dementia Collaborative Coaching showed preliminary positive outcomes for CNAs and PLWD. The intervention was acceptable, appropriate, and feasible for routine providers and warrants further study.

ELECTRONIC EDUCATIONAL CONTENT FORMATION WITH THE APPLICATION OF CLOUD TECHNOLOGIES

Many experts and scientists focus on cloud computing, mobile Internet technologies, and the rapid development of the Internet of Things in the field of e-learning which caused innovation and revolution. The development of the capabilities of cloud technologies has been applied to solve problems in various fields of human activity, including the educational process. Cloud technologies enable students to actively participate in the preparation of educational content, modify learning methods, increase access to information resources (external memory, applications, educational materials, etc.). Clouds unify physical or virtualized resources in data centers and require less software and hardware. The goal of this research is to expand and use the possibilities of applying cloud technologies in the teaching process in higher education institutions. As a result of the conducted research, we determine didactic and information opportunities by using cloud services in the educational process. This article reviews scientific research conducted in the field of data mining and cloud technologies in the formation of educational content during the last decade. Moreover, the methods of solving educational content management problems are proposed using data mining methods.

LL‐FMC: Low‐latency frame memory compression scheme with high reconstructed quality

AbstractFrame memory compression (FMC) saves power by reducing the data bandwidth for accessing reference frames of video coding standards in the external memory. However, since the continuously accumulated data loss degrades the performance of video coding standards, the quality of the reconstructed reference frame must be improved. Previous lossy frame memory compression methods focused on processing speed or simplicity, but their performance remains limited in terms of the quality of the reconstructed frame. In this study, a Huffman coding with a fixed codebook designed to considerably reduce the latency caused by arithmetic coding in lossless compression schemes is proposed. A novel lossy frame memory compression technique based on the Hadamard transform that does not require additional memory access using a lookup table is also proposed. The result of experiments on several reconstructed videos preprocessed with HEVC show that the proposed method improves BDPSNR by 10.7 dB and BDBR by 40.4% compared with the DWT+SPIHT‐based compression technique which is widely used for lossy frame memory compression.

Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues

Maintaining engagement and consistency is particularly important in dialogue systems. Existing works have improved the performance of dialogue systems by intentionally learning interlocutor personas with sophisticated network structures. One issue with this approach is that it requires more personal corpora with annotations. Additionally, these models typically perform the next utterance prediction to generate a response but neglect the discourse coherence in the entire conversation. To address these issues, this study proposes a method of learning to memorize entailment and discourse relations for persona-consistent dialogue tasks. Entailment text pairs in natural language inference dataset were applied to learn latent entailment relations as external memories by premise-to-hypothesis generation task. Furthermore, an internal memory with a similar architecture was applied to the discourse information in the dialogue. Placing orthogonality restrictions on these two memory spaces ensures that the latent entailment relations remain dialogue-independent. Both memories collaborate to obtain entailment and discourse representation for the generation, allowing a deeper understanding of both consistency and coherence. Experiments on two large public datasets, PersonaChat and DSTC7-AVSD, demonstrated the effectiveness of the proposed method. Both automatic and human evaluations indicate that the proposed model outperforms several strong baselines in terms of both persona consistency and response coherence. Our source code is availabled at https://github.com/Chenrj233/LMEDR.

Proof-of-Concept Real-Time Implementation of Interleavers for Optical Satellite Links

Ground-satellite optical links are a promising technology for increasing capacity and security over radio frequency links, while reducing weight and power consumption. However, atmospheric turbulence generates power fadings at the reception terminal. This effect can be mitigated by combining adaptive optics, forward error correction and data interleaving. While adaptive optics has been extensively studied, forward error correction together with long interleaving (e.g., multiple 10 ms) has not yet been demonstrated in this context. In this article, we present a proof of concept of an optical satellite uplink, precompensated by adaptive optics, which leverages a long interleaver in an external memory and operates at 10 Gbps with SFP+ transceivers. We account for the issues associated with real-time implementation and interfacing to a high-speed memory. To validate our proof of concept, we generate time-series for a typical ground to geostationary satellite communication link, using a field-validated simulator. Using these time series, we calculate the required interleaver time as a function of the launch power. Finally, we show the impact of different transmitter pupil diameters and of different wind speed profiles, on the interleaver duration.

Does memory rehabilitation improve health outcomes in people with multiple sclerosis? A Cochrane Review summary with commentary.

Memory disturbance is common in people with multiple sclerosis (pwMS). Currently, a range of memory rehabilitation approaches alone or as a component of cognitive rehabilitation is utilized clinically. To evaluate the effectiveness of memory rehabilitation in improving health outcomes (memory, cognitive function, functional ability, quality of life) in pwMS. A summary of the Cochrane Review "Memory rehabilitation for people with multiple sclerosis" by Taylor etal from a rehabilitation perspective. The review included 44 studies (with 2714 participants). The memory rehabilitation approaches varied amongst the included primary studies for memory retraining techniques (computerized programs, training using internal and external memory aids, etc.). Overall, the risk of bias amongst the included trials was low. The findings suggest high-certainty evidence for a beneficial effect of memory rehabilitation in improving subjective memory at intermediate- (1-6 months) and longer-term (> 6 months); and moderate-certainty evidence at immediate post-intervention. The evidence of the effect of memory rehabilitation on other outcomes showed mixed results. The evidence suggests some beneficial effects of memory rehabilitation in improving subjective memory and quality of life in pwMS. However, further evidence is required for the evaluation of memory strategies for other outcomes.

Compressing fully connected layers of deep neural networks using permuted features

AbstractModern deep neural networks typically have some fully connected layers at the final classification stages. These stages have large memory requirements that can be expensive on resource‐constrained embedded devices and also consume significant energy just to read the parameters from external memory into the processing chip. The authors show that the weights in such layers can be modelled as permutations of a common sequence with minimal impact on recognition accuracy. This allows the storage requirements of FC layer(s) to be significantly reduced, which reflects in the reduction of total network parameters from 1.3× to 36× with a median of 4.45× on several benchmark networks. The authors compare the results with existing pruning, bitwidth reduction, and deep compression techniques and show the superior compression that can be achieved with this method. The authors also showed 7× reduction of parameters on VGG16 architecture with ImageNet dataset. The authors also showed that the proposed method can be used in the classification stage of the transfer learning networks.