In this paper, we propose a new multiuser nonlinear optical cryptosystem using fractional-order vortex speckle (FOVS) patterns as security keys. In conventional optical cryptosystems, mostly random phase masks are used as the security keys which are prone to various attacks such as brute force attack. In the current study, the FOVSs are generated optically by the scattering of the fractional-order vortex beam, known for azimuthal phase and helical wavefronts, through a ground glass diffuser. FOVSs have a remarkable property that makes them almost impossible to replicate. In the input plane, the amplitude image is first phase encoded and then modulated with the FOVS phase mask to obtain the complex image. This complex image is further processed to obtain the encrypted image using the proposed method. Two private security keys are obtained through polar decomposition which enables the multi-user capability in the cryptosystem. The robustness of the proposed method is tested against existing attacks such as the contamination attack and known-plaintext attack. Numerical simulations confirm the validity and feasibility of the proposed method.