The Verwey transition in magnetite (Fe3O4 ) is the prototypical metal-insulator transition and has eluded a comprehensive explanation for decades. A major element of the challenge is the complex interplay between charge order and lattice distortions. Here we use ultrafast electron diffraction (UED) to disentangle the roles of charge order and lattice distortions by tracking the transient structural evolution after charge order is melted via ultrafast photoexcitation. A dual stage response is observed in which X3, X1, and Delta5 type structural distortions occur on markedly different timescales of 0.7 to 3.2 ps and longer than 3.2 ps. We propose that these distinct timescales arise because X3 type distortions strongly couple to the trimeron charge order, whereas the Delta5-distortions are more strongly associated with monoclinic to cubic distortions of the overall lattice. Our work aids in clarifying the charge lattice interplay using UED method and illustrates the disentanglement of the complex phases in magnetite.