This paper presents end-to-end learning from spectrum data-an umbrella term for new sophisticated wireless signal identification approaches in spectrum monitoring applications based on deep neural networks. End-to-end learning allows to: 1) automatically learn features directly from simple wireless signal representations, without requiring design of hand-crafted expert features like higher order cyclic moments and 2) train wireless signal classifiers in one end-to-end step which eliminates the need for complex multi-stage machine learning processing pipelines. The purpose of this paper is to present the conceptual framework of end-to-end learning for spectrum monitoring and systematically introduce a generic methodology to easily design and implement wireless signal classifiers. Furthermore, we investigate the importance of the choice of wireless data representation to various spectrum monitoring tasks. In particular, two case studies are elaborated: 1) modulation recognition and 2) wireless technology interference detection. For each case study three convolutional neural networks are evaluated for the following wireless signal representations: temporal IQ data, the amplitude/phase representation, and the frequency domain representation. From our analysis, we prove that the wireless data representation impacts the accuracy depending on the specifics and similarities of the wireless signals that need to be differentiated, with different data representations resulting in accuracy variations of up to 29%. Experimental results show that using the amplitude/phase representation for recognizing modulation formats can lead to performance improvements up to 2% and 12% for medium to high SNR compared to IQ and frequency domain data, respectively. For the task of detecting interference, frequency domain representation outperformed amplitude/phase and IQ data representation up to 20%.