In this study, a noncontact vision-based sensing method is proposed to measure surface displacements and curvatures and to detect cracks in a reinforced concrete slab. The proposed method includes five independent modules for structure boundary identification by a dynamic programming algorithm, boundary movement tracking by a contour tracking algorithm, distinguishable surface feature detection by speed-up-robust features, feature (visual mark) tracking by a three-stage data association algorithm, and displacement interpolation from those at visual marks by a Delaunay triangulation algorithm. The displacement field was used to evaluate the slab curvature that functioned as a crack indicator. The proposed data association algorithm for visual mark translation, linking, and connection was successfully applied for visual mark tracking of concrete slab images. The proposed algorithms used in five modules are computationally efficient, making them viable tools for real-time structural health monitoring. By persistently tracking the features and positions of spatially distributed visual marks in time-lapse videos, the displacement time histories at mark locations are successfully evaluated. The relative error of displacement measurements for the tested concrete slab is approximately 1.24%. The proposed method was applied to successfully detect cracks of a full-scale reinforced concrete slab from image analysis. Unlike contact measurements, the proposed noncontact measurement is not affected by concrete cracking.