Tensor data is emerging in many scientific applications, such as multi-tissue transcriptomics. In such cases, the covariates for each individual are no longer a vector. To apply traditional vector-based methods to this type of data, we need to either do the vectorization or analyze data marginally, which suffers a significant information loss. We propose a novel parsimonious tensor dimension reduction (pTDR) approach to directly link the response and tensor covariate through an unknown function g. In pTDR, the response variable, continuous or discrete, depends on K rank-one projections of the covariates, with the projections estimated via a sequential iterative dimension reduction algorithm. We further propose an asymptotic sequential statistical test to select the correct number of rank-one tensors. In contrast to the classic low-rank tensor regression, pTDR model is not restricted to the linear relationship between response and covariates. We apply pTDR to two modern genomic studies. We find that the gene expression of multiple tissues has a stronger association with aging and obesity than was apparent using previous approaches. Numerical results demonstrate the advantages of pTDR over competitors in terms of prediction accuracy and computing efficiency. Our software is publicly available on GitHub (https://github.com/BioAlgs/pTDR).