Text generation is a vital natural language processing task that generates a sequence given an input text. Most recent text generation models have mainly utilized the power of pre-trained language models (PLMs) and achieved promising results. However, while working on this problem, we observed that important indicators like salient words and sentences from downstream generation tasks can still be useful and beneficial for enhancing the performance. This paper aims to fill this gap by exploring and investigating the contribution of auxiliary supervised learning tasks to improve the performance of PLMs for text generation. To this end, we observe the nature of text generation under three scenarios: incomplete utterance restoration, question generation, and summarization. The observation suggests that the indicators can be beneficial for text generation. We reflect this nature by defining a set of auxiliary tasks. Specifically, we design a model that considers both auxiliary tasks and the main text generation task under the joint training manner. The efficiency of the model is validated in the three generation scenarios. The results of the extensive experiments on 10 datasets for the three scenarios and languages show that the auxiliary tasks significantly contribute to improving the performance of PLMs for the main generation task. The ablation study facilitates the next studies of text generation by showing the behavior of the model in different settings. The model is simple, effective, and flexible for other text generation tasks.