A linear and a non-linear analysis are carried out for the instability of the free surface of a liquid layer contained in a Hele-Shaw cell subjected to periodic vertical oscillation. The linear stability analysis shows that for certain ranges of the oscillation frequency, the depth of the liquid layer and the surface tension can have a substantial effect on the selection of the wavenumbers and on the critical forcing amplitude. This results in a new dispersion relation, relating the critical wavenumber and the frequency of oscillation, which is in excellent agreement with recent experimental results by Li et al (2018 Phys. Fluids 30 102103). On the other hand, for low frequencies, the thresholds can be either harmonic or subharmonic with the existence of a series of bicritical points where these two types of thresholds can coexist. Weakly nonlinear analysis is performed in the vicinity of the first subharmonic resonance that occurs in the high frequency limit. Thus, using the multiscale technique, for low dissipation and forcing, we derive a free surface amplitude equation, involving a new nonlinear term coefficient, χ, that includes finite depth and surface tension. For infinite depth, Rajchenbach et al (2011 Phys. Rev. Lett. 107 024502), and Li et al (2019 J. Fluid Mech. 871 694–716) showed that hysteresis can only occur if the response frequency is lower than the natural frequency. However in the present work, it turns out that the coefficient χ can be either positive or negative depending on the depth and surface tension of the fluid. Thus, if χ is positive, hysteresis is found when the response frequency is greater than the natural frequency. Furthermore, the infinite depth approximation, where the short wavelengths dominate, is valid when the depth and wavenumber satisfy kh > 5, whereas for kh < 5, where long wavelengths dominate, the finite depth should be considered.