The primary objective of conducting deviation analysis in the context of assembly process is to ascertain and guarantee the quality and precision of the final product. The deviation analysis is important for components that are prone to deformation. The conventional analysis approach primarily relies on the identification of local feature points on individual components to establish relationship between deviations before and after assembly. However, the method does not sufficiently represent the spatial distribution of deviations across the parts. The conventional approach also fails to account for shape inaccuracies and the influence of multiple sources of deviation coupling to certain degree. Therefore, this paper proposes a novel deviation framework for analyzing the sheet metal assembly process based on the skin model shapes and a conditional Generative Adversarial Network (cGAN). The cross-scale shape errors of the critical feature surfaces are modeled by taking the statistical parameters into account. Additionally, the contour maps of the established surface models and the multiple-source deviations are combined using images in order to construct the data for the flexible deviation network models. Then, the finite element method is used to simulate the assembly process, yielding the final component deviations. The contour maps of the assembly deviations and the images of the part deviations are utilized as the input conditions and ground truth for the dataset. The simulations and experiments conducted in this study provide evidence to support the effectiveness of the proposed method in predicting field-to-field deviation deformation. The results indicate that the proposed method outperforms traditional approaches in terms of accuracy. Furthermore, this approach enables end-to-end deviation prediction. The well-trained model is capable of directly outputting the corresponding predicted deviations given input of deviation factors. By employing such a deviation analysis framework, it enables achieve an accurate and high efficiency analysis in the assembly process.