Water distribution networks (WDNs) are critical to provide safe, clean drinking water around the globe. However, they are susceptible to accidental or deliberate contamination, potentially resulting in poisoned water, many fatalities and large economic consequences. In order to protect against such intrusions, an efficient sensor network should be placed in a WDN. Finding the optimal placement for water quality sensors is a challenging problem. Several sensor placement strategies have been proposed, but the vast majority of these strategies rely on the assumption that the sensors are perfect. In this paper we provide evidence for the imperfection of water quality sensors, by conducting measurements in an operational environment. We investigate the imperfection of four types of water quality sensors being employed in actual WDNs for the purpose of contamination detection. We describe experiments conducted at the WaDi testbed, a realistic water distribution facility at the Singapore University of Technology and Design. Through these experiments we study the imperfection, sensitivity and degradation of the water quality sensors, under normal conditions (water flow without contaminants present) as well as under attack conditions. It is shown that several aspects of sensor imperfection do occur, including missing values, inexplicable jumps and drifting.