Activation of gamma-aminobutyric acid B (GABA(B)) and 5-hydroxytryptamine (5-HT) receptors produces presynaptic inhibition at glutamatergic terminals in the rat neocortex. To evaluate interactions between these metabotropic receptors, field potentials were recorded in layer 2/3 of somatosensory cortex. In addition, the paired pulse (PP) protocol was used to measure changes in the ratio of the second/first extracellular synaptic potentials (S(2)/S(1) ratio) as an index of glutamate release probability in the area. Lowering extracellular [Ca(2+)](o) to 0.5 mM, increased the S(2)/S(1) ratio by 318 +/- 134%. 5-HT (1 microM) increased the S(2)/S(1) ratio by 61 +/- 15%. In presence of the GABA(A) antagonist bicuculline (10 microM), 5-HT increased the S(2)/S(1) ratio by 98 +/- 15%. This effect did not desensitize after two consecutive applications of the amine, and was dose dependent in the concentration range between 0.03-1 microM (EC(50) = 2.36 x 10(-7) mol/L). The increase of S(2)/S(1) ratio induced by 5-HT (1 microM) was blocked reversibly by the 5-HT(1A) antagonist NAN-190 (10-30 microM), but was unaffected by the selective GABA(B) antagonist CGP 52432 (1 microM). The action of 5-HT was mimicked by the 5-HT(1A/7) agonist 8OH-DPAT (10 microM), increasing the S(2)/S(1) ratio by 84 +/- 2%, a response that was unaffected by the 5-HT(2/7) antagonist ritanserin (2 microM). The 5-HT(1B) agonist CP93129 (10 microM) had no effect. The GABA(B) agonist baclofen (1 microM) increased the S(2)/S(1) ratio up to 308 +/- 33%, which is similar to that produced by low [Ca(2+)](o). When the effect of baclofen was maximal, application of 5-HT (1 microM) reversed the S(2)/S(1) ratio back to 78 +/- 27%, a result that was blocked by the 5-HT(2/7) antagonist ritanserin (100 nM). Notably, the interaction between the GABA(B) agonist and 5-HT was order dependent, because enhancement of the S(2)/S(1) ratio elicited by baclofen was not inhibited if 5-HT was applied first. These results suggest a complex interaction between GABA(B), 5-HT(1A), and 5-HT(2) receptors in layer 2/3 of rat somatosensory cortex. Activation of GABA(B) receptors induces PP facilitation (inhibits glutamate release) more efficiently than does activation of 5-HT(1A) receptors. When the effect of GABA(B) receptor activation is maximal, however, the influence of 5-HT changes to the opposite direction, inhibiting PP facilitation (increasing glutamate release) through activation of 5-HT(2) receptors.