Lane detection is necessary for developing intelligent Autonomous Vehicles (AVs). Using vision-based lane detection is more cost-effective, requiring less operational power. Images captured by the moving vehicle include varying brightness, blur, and occlusion caused due to diverse locations. We propose a Vision-based Ingenious Lane Departure Warning System (VILDS) for AV to address these challenges. The Generative Adversarial Networks (GAN) of the VILDS choose the most precise features to create images that are identical to the original but have better clarity. The system also uses Long Short-Term Memory (LSTM) to learn the average behavior of the samples to forecast lanes based on a live feed of processed images, which predicts incomplete lanes and increases the reliability of the AV’s trajectory. Further, we devise a strategy to improve the Lane Departure Warning System (LDWS) by determining the angle and direction of deviation to predict the AV’s Lane crossover. An extensive evaluation of the proposed VILDS system demonstrated the effective working of the lane detection and departure warning system modules with an accuracy of 98.2% and 96.5%, respectively.