Eigenstates of fully many-body localized (FMBL) systems are described by quasilocal operators $\tau_i^z$ (l-bits), which are conserved exactly under Hamiltonian time evolution. The algebra of the operators $\tau_i^z$ and $\tau_i^x$ associated with l-bits ($\boldsymbol{\tau}_i$) completely defines the eigenstates and the matrix elements of local operators between eigenstates at all energies. We develop a non-perturbative construction of the full set of l-bit algebras in the many-body localized phase for the canonical model of MBL. Our algorithm to construct the Pauli-algebra of l-bits combines exact diagonalization and a tensor network algorithm developed for efficient diagonalization of large FMBL Hamiltonians. The distribution of localization lengths of the l-bits is evaluated in the MBL phase and used to characterize the MBL-to-thermal transition.