Using the Dirac-mode expansion method, which keeps the gauge invariance, we analyze the Polyakov loop in terms of the Dirac modes in SU(3) quenched lattice QCD in both confined and deconfined phases. First, to investigate the direct correspondence between confinement and chiral symmetry breaking, we remove low-lying Dirac-modes from the confined vacuum generated by lattice QCD. In this system without low-lying Dirac modes, while the chiral condensate $\langle \bar{q} q\rangle$ is extremely reduced, we find that the Polyakov loop is almost zero and $Z_3$-center symmetry is unbroken, which indicates quark confinement. We also investigate the removal of ultraviolet (UV) Dirac-modes, and find that the Polyakov loop is almost zero. Second, we deal with the deconfined phase above $T_c$, and find that the behaviors of the Polyakov loop and $Z_3$-symmetry are not changed without low-lying or UV Dirac-modes. Finally, we develop a new method to remove low-lying Dirac modes from the Polyakov loop for a larger lattice of $12^3 \times 4$ at finite temperature, and find almost the same results. These results suggest that each eigenmode has the information of confinement, i.e., the "seed" of confinement is distributed in a wider region of the Dirac eigenmodes unlike chiral symmetry breaking, and there is no direct correspondence between confinement and chiral symmetry breaking through Dirac-eigenmodes.