Spike timing-dependent plasticity (STDP) and other conventional Hebbian-type plasticity rules are prone to produce runaway dynamics of synaptic weights. Once potentiated, a synapse would have higher probability to lead to spikes and thus to be further potentiated, but once depressed, a synapse would tend to be further depressed. The runaway synaptic dynamics can be prevented by precisely balancing STDP rules for potentiation and depression; however, experimental evidence shows a great variety of potentiation and depression windows and magnitudes. Here we show that modifications of synapses to layer 2/3 pyramidal neurons from rat visual and auditory cortices in slices can be induced by intracellular tetanization: bursts of postsynaptic spikes without presynaptic stimulation. Induction of these heterosynaptic changes depended on the rise of intracellular calcium, and their direction and magnitude correlated with initial state of release mechanisms. We suggest that this type of plasticity serves as a mechanism that stabilizes the distribution of synaptic weights and prevents their runaway dynamics. To test this hypothesis, we develop a cortical neuron model implementing both homosynaptic (STDP) and heterosynaptic plasticity with properties matching the experimental data. We find that heterosynaptic plasticity effectively prevented runaway dynamics for the tested range of STDP and input parameters. Synaptic weights, although shifted from the original, remained normally distributed and nonsaturated. Our study presents a biophysically constrained model of how the interaction of different forms of plasticity--Hebbian and heterosynaptic--may prevent runaway synaptic dynamics and keep synaptic weights unsaturated and thus capable of further plastic changes and formation of new memories.