Brain tumors are one of the most lethal diseases in the world. The segmentation of brain tumor is of great significance for physician in formulating appropriate diagnostic and treatment plans, not only accurate but also efficient 3D segmentation algorithms are urgently demanded in clinical practice. Nowadays, several 3D convolution neural networks have achieved impressive segmentation performance. However, these architectures come with extremely high computational overheads due to the extra depth dimensionality in 3D convolution, which may make these models prohibitive from practical large-scale clinic application. In this work, we aim at designing a more efficient and lightweight network without accuracy reduction for real-time segmentation of magnetic resonance images. To this end, we propose a multi-branch sharing network which consists of novel multi-branch sharing units. Different from other works, our proposed multi-branch sharing units focus the information sharing and communication between grouped layers by leveraging a Multiplexer operation, which can reduce the computational cost significantly while maintaining decent performance. Extensive experimental results on the BraTS2018 challenge dataset show that the proposed architecture achieve real-time inference while maintaining high accuracy for 3D brain magnetic resonance image segmentation.