The mechanical behaviors of crystalline materials, in particular the atomic scale deformation and failure processes, are strongly influenced by the local atomic stress near crystalline defects such as vacancies, dislocations and grain boundaries (GBs). Modern electron microscopy techniques have achieved unprecedented capabilities to image the internal microstructure of crystalline materials with atomic resolution. In comparison, our ability to map the corresponding atomic stress field, especially at grain boundaries with complex atomic structures, remains lagging behind due to the lack of direct experimental methods for stress measurement. Here, we propose a machine learning-based framework to realize atomic structure-stress (Atom-S2) mapping based on in situ transmission electron microscopy images. We demonstrate that, with optimized anti-noise performance and generalization ability, the Atom-S2 framework enables quantitative mapping of atomic stress in symmetrical tilt GBs, twin boundaries (TBs) as well as general GBs with changing curvature in crystalline metals. Moreover, the Atom-S2 can help understand, interpret and predict grain boundary stress evolution and plastic deformation mechanisms based on high-resolution transmission electron microscopy images, which provides a powerful tool to study atomic scale deformation behaviors and holds promises for accelerating materials design through defect engineering.