Continuous traffic growth and crunched profit margins are leading network operators to deploying high-capacity backbone infrastructures with minimal capital investment. However, the cost-effective design of optical transport networks (OTNs) remains a complex challenge subjected to multiple constraints, e.g., maximum bit rate per channel, electrical multiplexing capabilities, wavelength count per fiber, optical interface cost, maximum transparent reach, etc. To efficiently solve this multi-constrained dimensioning problem in presence of heterogeneous client demands and optical channel rates of 40 Gb/s and 100 Gb/s, this paper presents a novel hybrid optimization framework. The proposed approach is based on an iterative combination of linear programming and rounding algorithms for the demand routing, with graph coloring heuristics for the wavelength assignment. The performance of this framework is assessed and compared with a similar approach that resorts to an integer linear programming (ILP) model to route the demands. The results obtained show that our proposal is able to reach the same low network expenditures as the ILP while requesting less computation time. We also confirm that the most cost-effective network solutions are attained when optical line rate heterogeneity is jointly applied with diverse multiplexing capabilities at the OTN electrical layer, such as grooming and inverse-multiplexing.