As an emerging technique, sparse imaging from three-dimensional (3-D) and non-uniform samples provides an attractive approach to obtain high resolution 3-D images along with great convenience in data acquisition, especially in the case of targets consisting of strong isolated scatterers. Although data interpolation in k-space and fast Fourier transform have been employed in the existing 3-D sparse imaging methods to reduce the computational complexity, the data-gridding errors induced by local interpolation may usually result in poor imaging performance. In this paper, we directly regard the imaging problem as a joint sparse reconstruction problem from non-uniform data without interpolation in 3-D space. Combining dictionary reduction and Gauss iterative method with the optimized signal processing scheme, a sparse imaging algorithm is proposed to address the difficulty of large-scale computation involved in direct 3-D sparse reconstruction. Benefited from the optimized signal processing scheme and the avoidance of data interpolation, the direct 3-D sparse imaging (DTDSI) method proposed in this paper is of low computation scale and high imaging performance. Experiments of electromagnetic simulation data demonstrate the DTDSI method outperforms baseline methods in terms of resolving ability, lower side-lobes and higher accuracy.