Ultrawideband (UWB) microwave detection technology, which is low cost and harmless, has been intensively studied and developed for breast cancer detection. In this study, a composite end-to-end framework that consists of convolutional neural network (CNN) and long-short-term memory (LSTM) is proposed, which can realize the tasks of detecting and quadrant locating the breast tumor simultaneously without any complicated microwave imaging processing. In order to verify the proposed network, three datasets are constructed. First, the microwave signals are solved by the auxiliary differential equation-finite-difference time-domain (ADE-FDTD) solver and the magnetic resonance imaging (MRI)-derived breast models with varied breast densities. Then, Simulation datasets 1 and 2 with varied dielectric properties are constructed. Furthermore, an experimental dataset is constructed through an experimental system consisting of the UWB pulse generation module, UWB antennas, the realistic breast phantom, and sampling oscilloscope. With the proposed network, the overall prediction accuracies of the three datasets reach 99.56%, 98.94%, and 89.50%. These promising results demonstrate the effectiveness and accuracy of the proposed deep learning framework for microwave breast cancer detection.