The effect of structure difference between a crystal and its liquid on the interfacial roughening has been investigated. Jackson's theory on the interfacial roughening is modified by taking the mismatch between the two phases and the dangling bonds at the interface into consideration. The mismatch yields a tendency of decreasing the degree of the interfacial roughness. This tendency is strong, if the ratio of the latent heat of vaporization to that of fusion, δ h v/δ h f, or the anisotropy of the crystal structure is l When a dense random packing of hard spheres is assumed for the liquid structure, crystals of P 4, Sn, In, Hg, Zn, Cd and Al growing from their own liquids have intrinsically smooth surfaces, in contrast with the conclusion drawn from the Jackson theory. The present theoretical result for P 4 is consistent with the experimental observations of growing into a faceted form.