Natural gas is increasingly becoming the preferred choice of fuel for flexible electricity generation globally resulting in an electricity system whose reliability is progressively dependent on that of the natural gas transportation system. The cascaded relationship between the reliabilities of these system necessitates an integrated approach to planning both systems. This paper presents a chance constrained programing approach to minimize the investment cost of integrating new natural gas-fired generators, natural gas pipeline, compressors, and storage required to ensure desired confidence levels of meeting future stochastic power and natural gas demands. The proposed model also highlights the role of natural gas storage in managing short-time uncertainties in developing a long-term expansion plan for both the electric and natural gas systems. A two-stage chance constrained solution algorithm is employed in solving the mix-integer nonlinear programing optimization problem and illustrated on a standard IEEE 30 bus test system superimposes on the Belgian high-calorific gas network.