SIP (Session Initiation Protocol) has been widely adopted as a signaling protocol to establish, modify and terminate media sessions between end-users in the Internet. SIP introduces a retransmission mechanism to ensure the reliability of its real-time message delivery. However, retransmission can make server overload worse, leading to server crashes in SIP-based carrier networks (e.g. Skype). In order to study the impact of retransmission mechanism on SIP overload, in this paper, we create a discrete time fluid model to describe the queuing dynamics of an overloaded SIP server. Then we derive a sufficient stability condition that a SIP server can handle the overload effectively under the retransmission mechanism. Fluid model allows us to run fluid-based Matlab simulation directly to evaluate the overload performance. Event-driven OPNET simulation was also conducted to validate our fluid model. Our simulation results demonstrate that: (1) the sufficient stability bound is quite tight. The bound indicates that effective CPU utilization as low as 20% can still lead to an unstable system after a short period of demand burst or a temporary server slowdown. Resource over-provisioning is not a viable solution to the server crash problem; (2) by satisfying the stability condition, the initial queue size introduced by a transient overload can avoid a system crash. Such stability condition can help the operator to determine whether and when to activate overload control mechanism in case of heavy load. A simple overload control solution is also proposed.