Vacuum Insulation Panels (VIPs) provide significant adiabatic performance for heat/cooling systems to reduce energy consumption. The application of fibrous porous material (FPM) as the ideal core of VIPs has gained global attention in recent decades. The microstructure and physical properties of FPMs, filled as novel VIPs’ core material, and holding superior thermal performance, affected effective thermal conductivity (ETC) greatly. Aiming to deeply understand heat transfer mechanisms, a holistic simulation method that combined with a developed 3D FPM structure generation method and a D3Q15-Lattice Boltzmann method (LBM) is proposed to simulate the heat transfer in FPM and to illuminate the influence factors of ETC on the microstructure of FPM in a vacuum. The improved and modified mesoscopic 3D fibrous random micro-structure generation approach involved five structural parameters: generation probability of nucleus growth, fiber length, diameter, coincidence rate, and orientation angle. The calculation model of ETC is established, and the discrete velocity, distribution, evolution, and boundary conditions of D3Q15-LBM are invested in detail. The model is validated with influences of different microstructure parameters. It indicated that FPM with finer diameter, smaller average pore size, and bigger orientation angle easily gain the lower ETC in a vacuum. The ETC was also affected by the orientation angles of fibers. The more the heat transfer direction is inconsistent with the length direction of the fiber, the better the adiabatic performance is. The reliability of the model is verified by comparison, and this work is a reference to optimize the fibrous core of VIPs.