Pollinating insects are decreasing worldwide due to various environmental stresses (so-called pollinator crisis), raising concerns that plant productivity could be undermined in natural and agricultural ecosystems. To date, however, few studies have reported a concurrent decline in both pollinators and plants, and little is known about when a “plant crisis” occurs. Here, we propose that anthropogenic environmental stresses on pollinating insects (e.g. climate change, habitat loss, and pesticide usage) can negatively affect herbivorous insects (e.g., pollinator larvae and crop pests) as well, and effects of pollinator declines may be masked by positive effects of herbivore declines. To test the idea, we theoretically investigated plant population dynamics mediated by two insect groups: one representing a pollinator that is mutualistic at the adult stage but antagonistic at the larval stage, and the other representing a non-structured pest herbivore. Our model revealed that environmental stresses (increasing insect mortality) can have counterintuitive effects on plants. Nonetheless, plant abundance generally decreases with decreasing pollinator abundance, especially when plant populations grow slowly without pollinators, when pollinators are effective mutualists, or when pollinators are susceptible to environmental stresses. These findings offer a theoretical basis for assessing the pollinator crisis for biodiversity conservation and agricultural management.