Calories play an essential role in health aspects that lead to diseases like coronary heart disease, liver disease, cancer, and cholesterol. A study from 2020 reported that globally, overweight adults outnumber underweight individuals by more than 1.9 billion, while obese adults outnumber underweight ones by 650 million. Statistics from India show that abdominal obesity is the most significant risk factor, and it varies from 16.9% to 36.3%. Deep learning is an advanced image processing technology that solves problems and ensures food challenges because deeper networks have a better ability to process many features in an image. In our study, we propose a hybrid framework to predict the calorie content of food items on a plate. This includes three main parts: segmentation to segment the food from the image, image classification for classifying the food items, and calculating the calories present in those food items. A generative adversarial network is used for the segmentation, while a convolutional neural network is used for the classification and calorie estimation. The above models trained on the food images from the UNIMIB 2016 dataset have correctly recognized and estimated the calories of a food item with an accuracy of 95.21%.