Open set segmentation is a relatively new and unexplored task, with just a handful of methods proposed to model such tasks. We propose a novel method called CoReSeg that tackles the issue using class conditioned reconstruction of the input images according to their pixelwise mask. Our method conditions each input pixel to all known classes, expecting higher errors for pixels of unknown classes. It was observed that the proposed method produces better semantic consistency in its predictions than the baselines, resulting in cleaner segmentation maps that better fit object boundaries. CoReSeg outperforms state-of-the-art methods on the Vaihingen and Potsdam ISPRS datasets, while also being competitive on the Houston 2018 IEEE GRSS Data Fusion dataset. Our official implementation for CoReSeg is available at: https://github.com/iannunes/CoReSeg.