Accurate tropical cyclone (TC) intensity and size estimation are key in disaster management and prevention. While great breakthroughs have been made in TC intensity estimation research, there is currently a lack of research on TC size reflecting TC influence radius. Therefore, we propose a lightweight multi-task learning model (TC-MTLNet) with adaptive loss balance to simultaneously estimate TC intensity and size. Adaptive loss balance is utilized to solve the problem of inconsistent convergence speed of TC intensity and size estimation tasks. The model based on four 2-D convolutions, four 3-D convolutions and three fully connected layers takes up less computational and storage space and improves the accuracy of TC intensity and size estimation by sharing knowledge among multiple tasks. In addition, due to the imbalanced distribution of TC samples, with significantly few low-intensity and high-intensity TC satellite data, this phenomenon poses a great challenge to TC intensity and size estimation. So, we utilize the influence of nearby samples to calibrate the sample density to weight the loss function to enable the model to be generalized to all samples. The result shows that the root-mean-square error (RMSE) of TC intensity estimation is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{8.40}\,\text{kts}$</tex-math></inline-formula> , which is 33.5% lower than that of the Advanced Dvorak Technique (ADT) and 11.4% lower than that of the deep learning method (3DAttentionTCNet). The mean absolute error (MAE) of the TC size estimation is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{20.89}\,\text{nmi}$</tex-math></inline-formula> , which is a 16% reduction compared to the Multi-Platform Tropical Cyclone Surface Winds Analysis (MTCSWA).