Phosphorus (P) accumulation is a common phenomenon in greenhouse soil for vegetables. Excessive P accumulation in soil usually decreases the yield and quality of vegetables as well as potentially polluting water environments. Ninety-eight tomato and 48 cucumber greenhouses were investigated in the eight main vegetable production areas of Hebei Province, China. Soil Olsen-P, the electrical conductivity (EC), the pH value, the organic matter of the soil, and the cropping years of these greenhouses were investigated and analyzed in order to better understand the status of soil P accumulation and positively find effective ways to solve the excessive phosphate accumulation problem. The investigation showed that the ratio was above 70% for all of the greenhouses where the soil Olsen-P exceeded 90 mg·kg−1 (upper bound of soil Olsen-P optimum value in greenhouse) in the 0–20 cm surface soil in the investigated greenhouses. There was a significant positive correlation between the soil Olsen-P content and the soil EC, between the soil Olsen-P and the cropping years, and the Olsen-P had a significant negative correlation with the soil pH value. It is concluded that supplying phosphate fertilizer excessively induced the soil EC to ascend and the pH value to descend, which increases the possibility of the soil secondary salinization and soil degeneration. The significant positive correlation between the soil organic content and the soil Olsen-P contents suggests that supplying organic fertilizer might mobilize soil residual phosphate. This also provides a good way to solve the problem of soil P accumulation. In order to further explore the threshold content of soil Olsen-P demanded by tomato and cucumber under the high soil Olsen-P condition, two tomato greenhouses (T1, T2) in Dingzhou and two cucumber greenhouses (C1, C2) in Wuqiang were researched. All of the greenhouses had ranges of soil Olsen-P content that were between 150 and 300 mg·kg−1, which far exceeded the 90 mg·kg−1 ideal. The P fertilizer application rates showed positive correlations with the soil Olsen-P contents and EC values in cucumber and tomato greenhouses in the current season. Analyzing T1 and T2 results showed that tomato was sensitive to the high soil Olsen-P contents ranging from 230.64 to 729.42 mg kg−1 at the seedling stage (15 days after transplanting; DAT) and from 199.41 to 531.42 mg kg−1 at the fruiting stage (90 DAT), because the yields correlated negatively with soil Olsen-P contents at each growth stage. It is suggested that the maximum soil Olsen-P threshold content for tomato should be lower than 230 mg·kg−1 at the seedling stage and lower than 199 mg·kg−1 at the fruiting stage. But cucumber yield did not change significantly as soil Olsen-P content rose from 248.75 to 927.62 mg kg−1, 212.40 to 554.07 mg kg−1, 184.48 to 455.90 mg kg−1, and 128.42 to 400.96 mg kg−1 at the seedling stage (15 DAT), early fruiting stage (50 DAT), middle fruiting stage (140 DAT), and late fruiting stage (235 DAT), respectively, suggesting that the maximal soil Olsen-P threshold content was lower than 249, 212, 185, and 128 mg·kg−1 at each growth stage, respectively. The relationship between fruit qualities and soil Olsen-P contents at each growth stage was not evident. Activities of soil alkaline phosphatase (ALP) decreased as soil Olsen-P supply was raised in T1, T2, and C1 at the seedling stage. It is concluded that in an excess soil Olsen-P condition tomato yield decreases strongly as soil ALP activity decreases, whereas ALP activity has little direct effect on cucumber yield.