The Lamb wave technique has been widely used for damage detection in plate-like structures. However, for two-dimensional (2D) defects like corrosions or one-dimensional (1D) defects like cracks, conventional Lamb wave-based methodology usually selects the detection algorithm in terms of the damage type, which is restricted if the defect remains unknown. To address the issues, this paper proposes a damage assessment approach based on the Lamb wave factorization method, which is a general method that can adapt to the detection of both 2D and 1D defects. In the process, both the forward-scattered wave across the inspection area and the wave back-scattered from the defect are used for detection, so that the enclosed rich information can be fully exploited. To measure the wave field of defect, a transducer array is firstly set around the inspection area, whose transducers would take turns to excite the Lamb wave while others remain listeners. On the basis, the scattered waves of all directions are then extracted and transformed from the time domain to frequency domain. Under a certain frequency, the inspection area is next visualized by sampling the area and solving the inverse scattering problem using the spectral data at each sampling point. By fusing the results of different frequency together, the final image can be eventually obtained. The method is then numerically investigated and also validated through experiments. The results demonstrate that it can give out the outline description of 2D defect and the size evaluation of 1D defect accurately, thus being an effective tool for damage assessment.