A three-dimensional numerical analysis is performed of the deformation-induced roughening in polycrystalline specimens with and without surface-hardened layers. Three-dimensional microstructure-based constitutive models are developed, using crystal plasticity, and employed in finite element calculations of uniaxial tension. Grain structure is shown to be responsible for free surface roughening under uniaxial loading. Microscale stresses acting normally to the free surface in the bulk of the material are associated with normal displacements. The surface-hardened layer moves the grain structure away from the free surface, smoothing out the microscale folds formed due to displacements of individual grains, while the mesoscale surface undulations remain clearly visible.