The rigid and flexible contact of the refrigerator seal has a significant impact on its heat transfer characteristics, but existing research has ignored this important factor. In this paper, the assembly physical model at the seal region is established by static structural analysis. Then a three-dimensional heat transfer numerical simulation is carried out. The assembly distance affects the contact state of the seal by changing the contact position between the seal and the door or the cabinet. The calculation results show that the calculated temperature at the seal region considering the actual assembly deformation is closer to the measured temperatures. In the case study, the heat leakage load decreases by 18.57% and 19.99% at the compressed state, while it increases by 4.45% under the stretching state. The proportion of heat transfer load in the path of rubber strip-cold air and rubber strip-outer shell of the cabinet is the largest, reaching over 30.0%. The heat transfer load of the cold bridge also accounts for over 20.0%. When optimizing the seal structure, reducing the contact interface length of rubber strip-cold air, reducing thermal conductivity, and increasing the contact thermal resistance of cold bridge should be focused on.