This study aims at providing a new insights into the impact toughness scatter from the aspect of crystallographic structure. It demonstrated that the large impact toughness scatter associated much to the microstructure diversity. The crystallographic structure with evident scatter will display obvious discrepancy. In this work, three groups of samples simulated coarse-grained heat affected zone (CGHAZ) of an offshore engineering steel were obtained at different cooling rates. The Charpy test results showed that the toughness decreases dramatically with the decrease of cooling rate. However, the largest scatter in impact toughness occurred in the sample with medium cooling rate (15 °C/s), which was attributed to the heterogeneity in crystallographic structures. The visualization of crystallographic features showed that the prior austenite grain size has a significant effect on bainitic variant selection, which governed the effective grain size and crack propagation mechanism. CP (close-packed plane) grouping of variants is more likely to take place in large austenite grain, indicating that the size of CP region is larger than Bain zone, and the crack is short and flexural. On the contrary, in smaller austenite grain, Bain grouping of variants that always forms low angle grain boundary and favors crack propagation dominates the transformation, and it will promote the crack to propagate through the entire Bain zone and then yield large long crack. However, these two cases can co-exist in the same sample at medium cooling rate, indicating that the cleavage fracture is controlled by the effective grain size (Bain-zone size) and the scatter in impact toughness is associated much to the proportion and relative location between fine and coarse Bain zones.