Designing thermal conductivity efficiently is one of the most important study fields for taking the advantages of woven composites. This paper presents an inverse method for the thermal conductivity design of woven composite materials. Based on the multi-scale structure characteristics of woven composites, a multi-scale model of inversing heat conduction coefficient of fibers is established, including a macroscale composite model, mesoscale fiber yarn model, microscale fiber and matrix model. In order to improve computational efficiency, the particle swarm optimization (PSO) algorithm and locally exact homogenization theory (LEHT) are utilized. LEHT is an efficient analytical method for heat conduction analysis. It does not require meshing and preprocessing but obtains analytical expressions of internal temperature and heat flow of materials by solving heat differential equations and combined with Fourier's formula, relevant thermal conductivity parameters can be obtained. The proposed method is based on the idea of optimum design ideology of material parameters from top to bottom. The optimized parameters of components need to be designed hierarchically, including: (1) combing theoretical model with the particle swarm optimization algorithm at the macroscale to inverse parameters of yarn; (2) combining LEHT with the particle swarm optimization algorithm at the mesoscale to inverse original fiber parameters. To identify the validation of the proposed method, the present results are compared with given definite value, which can be seen that they have a good agreement with errors less than 1%. The proposed optimization method could effectively design thermal conductivity parameters and volume fraction for all components of woven composites.