The fractal dimension and the surface roughness of the fracture surface was estimated using the height data generated by the stereo matching method on Charpy V-notch specimens of the SS400 steel impact-fractured in the temperature range from 194 K (−79°C) to 273 K (0°C). The representative fracture surfaces were quasi-cleavage fracture surface with river patterns and steps between facets at 194 K and ductile fracture surface with dimples and shear steps at 273 K. On the region close to the notch root (fracture initiation site), a definite correlation was not found between the absorbed energy and the fractal dimension of the fracture surface, but both absorbed energy and surface roughness increased with temperature because of increased proportion of ductile fracture surface with the larger surface roughness. There was essentially no difference in the fractal dimension between quasi-cleavage fracture surface and ductile fracture surface. Fracture process including fracture initiation, local crack growth or finally fractured part could be examined by the detection of characteristic fracture patterns using the fractal dimension map (FDM) and the surface roughness map (SRM). Detected steps of various sizes were associated with river patterns or fracture at ferrite grain boundaries on the quasi-cleavage fracture surface, and were associated with dimple walls or shear steps on the ductile fracture surface. Steps on both fracture surfaces were formed by common mechanism, namely, by plastic deformation (shear deformation). The density of steps was larger on the quasi-cleavage fracture surface than on the ductile fracture surface. These steps led to a large fractal dimension of the quasi-cleavage fracture surface comparable with the ductile fracture surface with dimples.