The recently proposed boundary condition-enforced immersed boundary-lattice Boltzmann method (IB-LBM) [14] is improved in this work to simulate three-dimensional incompressible viscous flows. In the conventional IB-LBM, the restoring force is pre-calculated, and the non-slip boundary condition is not enforced as compared to body-fitted solvers. As a result, there is a flow penetration to the solid boundary. This drawback was removed by the new version of IB-LBM [14], in which the restoring force is considered as unknown and is determined in such a way that the non-slip boundary condition is enforced. Since Eulerian points are also defined inside the solid boundary, the computational domain is usually regular and the Cartesian mesh is used. On the other hand, to well capture the boundary layer and in the meantime, to save the computational effort, we often use non-uniform mesh in IB-LBM applications. In our previous two-dimensional simulations [14], the Taylor series expansion and least squares-based lattice Boltzmann method (TLLBM) was used on the non-uniform Cartesian mesh to get the flow field. The final expression of TLLBM is an algebraic formulation with some weighting coefficients. These coefficients could be computed in advance and stored for the following computations. However, this way may become impractical for 3D cases as the memory requirement often exceeds the machine capacity. The other way is to calculate the coefficients at every time step. As a result, extra time is consumed significantly. To overcome this drawback, in this study, we propose a more efficient approach to solve lattice Boltzmann equation on the non-uniform Cartesian mesh. As compared to TLLBM, the proposed approach needs much less computational time and virtual storage. Its good accuracy and efficiency are well demonstrated by its application to simulate the 3D lid-driven cubic cavity flow. To valid the combination of proposed approach with the new version of IBM [14] for 3D flows with curved boundaries, the flows over a sphere and torus are simulated. The obtained numerical results compare very well with available data in the literature.