Bolts are frequently subjected to loosening due to time varying external loads during service. The electromechanical impedance (EMI) technique based on piezoelectric ceramic wafers (PZT) is sensitive to the initial bolt preload looseness. However, the change in environmental temperature has a great effect on EMI monitoring. Deep convolutional neural network (CNN) is a promising technique for EMI monitoring. Nevertheless, it is difficult to train a deep CNN with limited training data to accurately identify damages under a wide range of temperature variations. To this end, this study proposes a multitask CNN for identifying bolts loosening. The network consists of a temperature compensation subnetwork to compensate for the temperature effect, and a lightweight damage identification subnetwork to identify bolt loosening states. The temperature compensation subnetwork is a modified Unet, and both the impedance and temperature are used as its input. The damage identification subnetwork is connected in series behind the temperature compensation subnetwork. A multiloss function is proposed in which a TV regularizer is used. Experimental results show that the validation accuracy of the multitask network is 97.71% when the network is trained by only about 30 samples from each loosening state. Moreover, the generalization abilities of the proposed multitask model to unexpected temperatures and bolt torques are investigated. The model is interpreted by the integrated gradients method, and is also compared with single-task damage identification CNNs. It is proved that the multitask network trained by limited samples can achieve accurate damage identification in temperature varying environments.