The purpose of this paper is to present an end-to-end deep convolutional neural network to improve the dual-energy CT (DECT) material decompositionperformance. In this study, we proposes a unified mutual-domain (sinogram domain and CT domain) material decomposition network (DIRECT-Net) for DECT imaging. By design, the DIRECT-Net has immediate access to mutual-domain data, and utilizes stacked convolution neural network layers for noise reduction and material decomposition. The training data are numerically generated following the fundamental DECT imaging physics. Numerical simulation of the XCAT digital phantom, experiments of a biological specimen, a calcium chloride phantom and an iodine solution phantom are carried out to evaluate the performance of DIRECT-Net. Comparisons are performed with different DECT decompositionalgorithms. Results demonstrate that the proposed DIRECT-Net can generate water and bone basis images with less artifacts compared to the other decomposition methods. Additionally, the quantification errors of the calcium chloride (75-375 mg/cm3 ) and the iodine (2-20 mg/cm3 ) are less than 4%. An end-to-end material decomposition network is proposed for quantitative DECT imaging. The qualitative and quantitative results demonstrate that this new DIRECT-Net has promising benefits in improving the DECT imagequality.