Abstract
Since its invention, generative adversarial networks (GANs) have shown outstanding results in many applications. GANs are powerful, yet resource-hungry deep learning models. The main difference between GANs and ordinary deep learning models is the nature of their output and training instability. For example, GANs output can be a whole image versus other models detecting objects or classifying images. Thus, the architecture and numeric precision of the network affect the quality and speed of the solution. Hence, accelerating GANs is pivotal. Data transfer is considered the main source of energy consumption, that is why memory compression is a very efficient technique to accelerate and optimize GANs. Two main types of memory compression exist: lossless and lossy ones. Lossless compression techniques are general among all models; thus, we will focus in this paper on lossy techniques. Lossy compression techniques are further classified into (a) pruning, (b) knowledge distillation, (c) low-rank factorization, (d) lowering numeric precision, and (e) encoding. In this paper, we survey lossy compression techniques for CNN-based GANs. Our findings showed the superiority of knowledge distillation over pruning alone and the gaps in the research field that needs to be explored like encoding and different combination of compression techniques.
Highlights
Nowadays, deep learning (DL) applications are getting unprecedented popularity
This failure is attributed to the following reasons: (1) the high resolution of the generator output compared to discriminator models makes it more sensitive to noise, (2) The generator evaluation metrics are more subjective than objective, and (3) the training of generative adversarial networks (GANs) is unstable and care should be taken to avoid discriminator over-powering the generator
No doubt that minimizing the memory footprint enhances storage, speed, and power efficiency of running GANs, there are some areas such as “encoding” that is not explored at all in GANs. Another opportunity exists in optimizing GANs is to fill in the gaps and mix between different compression techniques
Summary
Deep learning (DL) applications are getting unprecedented popularity. Many deep learning applications are used daily such as face recognition, voice recognition, weather predictions, and image super-resolution. We divided the lossy techniques to five categories: (a) pruning, (b) knowledge distillation, (c) low-rank factorization, (d) lowering numeric precision, and (e) encoding Those five techniques are explained thoroughly in this survey, focusing only on methods applied to GANs. Several survey papers exist about deep learning compression like [9–12]. The remainder of this paper is organized as follows: the “Memory compression techniques for GANs” section presents a brief background on how GANs work followed by reviewing different efforts to optimize GANs using compression techniques and eventually, it provides open research questions that need to be further studied. Memory compression techniques for GANs will start by background explanation for GANs and its performance metrics in the “Background” section, followed by reviewing latest work on lossy compression techniques in the “Memory compression techniques”. In the last section “Results and discussion”, we will provide summary of findings and open research questions that need to be further studied
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
