Pentacene is an important model organic semiconductor in both the singlet exciton fission (SF) and organic electronics communities. We have investigated the effect of changing crystal structure on the SF process, generating multiple triplet excitons from an initial singlet exciton, and subsequent triplet recombination. Unlike for similar organic semiconductors that have strong SF sensitive to polymorphism, we find almost no quantitative difference between the kinetics of triplet pair (TT) formation in the two dominant polymorphs of pentacene. Both pairwise dimer coupling and momentum-space crystal models predict much faster TT formation from the bright singlet excited state of the Bulk vs ThinFilm polymorph, contrasting with the experiment. GW and Bethe-Salpeter equation calculations, including exciton-phonon coupling, reveal that ultrafast phonon-driven transitions in the ThinFilm polymorph compensate the intrinsically slower purely Coulomb-mediated TT formation channel, rationalizing the similarity in observed rates. Taking into account the influence of subtle structural distinctions on both the direct and phonon-mediated SF pathways reveals a predictive capability to these methods, expected to be applicable to a wide variety of molecular crystals.