Learning vector representations (a.k.a., embeddings) of users and items lies at the heart of building a modern recommender system. Typically, graph neural networks (GNN) have been a prevalent way to learn user and item embeddings with implicit feedback. Nevertheless, since item popularity in many recommendation scenarios follows a long-tail distribution, we argue that recent GNN-based recommenders are not sufficient to yield satisfactory embeddings, especially for tail items. More importantly, most of GNN-based methods depend on negative sampling in model optimization, which makes them inefficient in the GPU era and highly sensitive to the design of negative sampler. In light of these limitations, we propose a novel recommendation method named ECGN, short for efficient complementary graph convolutional network. Specifically, by means of the complementary relationships amongst items, we introduce a complementary-aware graph convolution module, which better alleviates the long-tail problem. By regarding user behaviors and item relationships as two bipartite graphs, such module could explicitly explore the propagation process of user interest and complementary influence. Furthermore, an item-specific attention mechanism is designed to fuse item representations from different bipartite graphs. To make our solution attractive in large-scale industrial scenarios, we devise an efficient non-sampling learning paradigm to optimize ECGN’s parameters. Experiments on six real-world datasets demonstrate that our method consistently and significantly outperforms several state-of-the-art methods in model effectiveness and training efficiency. Our implementations and datasets are available at https://github.com/wubinzzu/ECGN.