This study presents a novel method for estimating the surface integrity of end-milled workpieces. It is well known that the mechanical properties of machined surfaces in cutting affect the quality of the final product. In particular, hardness and residual stress often require strict control; however, nondestructive inspection remains a challenge. This study proposes a method to estimate the hardness and residual stress of end-milled surfaces by analyzing cutting forces and images of the tool during machining to obtain approximate temperature and stress distributions. These state quantities are highly correlated with the dislocation density and its distribution on the machined surface, which in turn is strongly correlated with residual stress and surface hardness. Despite this strong correlation, few research studies have been conducted on the topic. In the proposed method, cutting forces, measured by a dynamometer, are analyzed to estimate the specific cutting forces and edge force coefficients. Simultaneously, the flank wear width is recorded using an image-based on-machine measuring device installed in the machine tool. From this information, the average stresses at the primary and tertiary cutting zones are estimated, while the cutting temperature in the primary cutting zone is roughly estimated by considering the traditional shear-angle prediction theory. Using these estimations, hardness and residual stress are calculated based on a simple linear regression model. Parameter identification for the model is performed based on measured hardness and residual stress in end-milling experiments. The model was validated against experimental measurements, which showed that the proposed method can accurately estimate hardness and residual stress, although it was observed that the selection of explanatory variables has a significant effect on accuracy.