Inspection and documentation of masonry structures is a time-consuming and expensive process that heavily relies on an engineer's expertise. This paper introduces a computer vision-based method that automates the creation of classified point clouds from 3D reconstruction models obtained through photogrammetry and/or laser scanning. By leveraging Convolutional Neural Networks (CNN) on 3D renders, the proposed methodology can be used to classify structural features (i.e., blocks, mortar, cracks) with greater accuracy and less effort than conventional methods. Moreover, this approach is flexible and can include further classifications by incorporating additional CNN models, allowing broader applications across various materials (i.e., concrete, steel, timber) and defects. Additionally, a precise methodology for accurate crack quantification featuring a manual annotation tool was introduced which was validated using outputs from a full-scale masonry arch bridge test carried out in the laboratory. The results emphasize the robustness of the classification approach and highlight the useful geometric information that can be gained from full-scale masonry structures. The proposed approach has the potential to use image data from UAVs/static cameras and revolutionize the way we document and inspect structures in the future.