We have carried out a study of optical image encryption in the Fresnel transform (FrT) domain, using a random phase mask (RPM) in the input plane and a phase mask based on devil’s vortex toroidal lens (DVTL) in the frequency plane. The original images are recovered from their corresponding encrypted images by using the correct parameters of the FrT and the parameters of DVTL. The use of a DVTL‐based structured mask enhances security by increasing the key space for encryption and also aids in overcoming the problem of axis alignment associated with an optical setup. The proposed encryption scheme is a lensless optical system and its digital implementation has been performed using MATLAB 7.6.0 (R2008a). The scheme has been validated for a grayscale and a binary image. The efficacy of the proposed scheme is verified by computing mean‐squared‐error (MSE) between the recovered and the original images. We have also investigated the scheme’s sensitivity to the encryption parameters and examined its robustness against occlusion and noise attacks.