Because optical properties of colloidal quantum dots (QDs) greatly depend on their layer thicknesses at the sub-nanometer scale, accurately predicting optical properties based on these structural parameters can help streamline the development process. However, most of traditional inverse design approaches require time-consuming numerical simulations and optimization processes. In this simulation-only paper, we propose a deep-learning-based inverse design model for colloidal InP/ZnSe/ZnS QDs based on the structure parameters of layer thicknesses and optical properties of the emission and absorption spectrum. Four different deep neural network (DNN) structures are implemented and compared. A tandem DNN structure with the simultaneous use of both emission and absorption spectra as input data shows the best accuracy due to its ability to alleviate the non-uniqueness problem in the inverse design of colloidal QDs. Our inverse design model enables the prediction of layer thicknesses for QDs that exhibit on-demand optical emission/absorption spectra, which are particularly enhanced at the specific target wavelength region. Moreover, the weights and biases of the well-trained tandem DNN for InP/ZnSe/ZnS QDs can be transferred and tuned to implement a transfer-learning-assisted inverse design model for CdSe/ZnSe/ZnS QDs, where the amount of training data to achieve the average prediction accuracy of 99.3 % is reduced by a maximum of 71.4 %.