This paper deals with the k-way normalized cut problem in complex networks. It presents a methodology that uses mathematical optimization to provide mixed-integer linear programming formulations for the problem. The paper also develops a branch-and-price algorithm for the above-mentioned problem which scales better than the compact formulations. Additionally, a heuristic algorithm which is able to approximate large-scale image problems in those cases where the exact methods are not applicable is presented. Extensive computational experiments assess the usefulness of these methods to solve the k-way normalized cut problem. Finally, we have applied the minimum normalized cut objective function to the segmentation of actual images, showing the applicability of the introduced methodology.