We present an experimental study of density and order fluctuations in the vicinity of the solid-liquid-like transition that occurs in a vibrated quasi-two-dimensional granular system. The two-dimensional projected static and dynamic correlation functions are studied. We show that density fluctuations, characterized through the structure factor, increase in size and intensity as the transition is approached, but they do not change significantly at the transition itself. The dense, metastable clusters, which present square symmetry, also increase their local order in the vicinity of the transition. This is characterized through the bond-orientational order parameter Q4, which in Fourier space shows an Ornstein-Zernike-like behavior. Depending on the filling density and vertical height, the transition can be of first- or second-order type. In the latter case, the associated correlation length ξ4, the relaxation time τ4, the zero k limit of Q4 fluctuations (static susceptibility), the pair correlation function of Q4, and the amplitude of the order parameter obey critical power laws, with saturations due to finite size effects. Their respective critical exponents are ν(perpendicular))=1, ν(parallel)=2, γ=1, η=0.67, and β=1/2, whereas the dynamical critical exponent z=ν(parallel)/ν(perpendicular)=2. These results are consistent with model C of dynamical critical phenomena, valid for a nonconserved critical order parameter (bond-orientation order) coupled to a conserved field (density).