Device-to-device (D2D) communication, which ena- bles a direct connection between users while bypassing the cellular channels to base stations (BSs), is a promising way to offload the traffic from conventional cellular networks. In D2D communication, optimizing the resource allocation requires the knowledge of D2D channel gains. However, such knowledge is hard to obtain at reasonable signaling costs. In this paper, we show this problem can be circumvented by tapping into the information provided by the estimated cellular channels between the users and surrounding BSs as these channels are estimated anyway for a normal operation of the network. While the cellular and D2D channel gains exhibit independent fast fading behavior, we show that average gains of the cellular and D2D channels share a non-explicit relation, which is rooted into the network topology, terrain, and buildings setup. We propose a deep learning approach to predict the D2D channel gains from seemingly independent cellular channels. Our results show a high degree of convergence between the true and predicted D2D channel gains. Moreover, we demonstrate the robustness of the proposed scheme against environment changes and inaccuracies during the offline training. The predicted gains allow to reach a near-optimal capacity in many radio resource management algorithms.