Water is an all-important need of all living beings. With the exponential growth of the human population, the need for conservation of water resources is gaining greater importance. Many water management systems have been proposed in the past using different technologies to address the issue which are high in cost and energy consumption. With the advent of the Internet of Things (IoT), the pursuit of the smart water management system is gaining momentum. This study first discusses the architecture and various components of IoT based water management system in detail followed by in-depth survey of all existing IoT based water management systems. Various measurement parameters such as water level, pH level, turbidity, salinity, etc. used in different water management systems proposed in the literature have also been identified and a comparison of various systems based on these parameters has also been presented. Finally, based on the survey, list of various essential attributes of these systems are framed which must be incorporated in future designs. In addition to this, an architecture of a smart water management system based on IoT and Machine learning has also been proposed as future scope which addresses all these essential attributes and also uses machine learning based predictions which can increase the efficiency of the smart management system.