In this paper we consider an online publisher that sells advertisement space and propose a method for learning optimal reserve prices in second-price auctions. We study a limited information setting where the values of the bids are not revealed and no historical information about the values of the bids is available. Our proposed method is based on the principle of Thompson sampling combined with a particle filter to approximate and sample from the posterior distribution. Our method is suitable for non-stationary environments, and we show that, when the distribution of the winning bid suffers from estimation uncertainty, taking the gap between the winning bid and second highest bid into account leads to better decisions for the reserve prices. Experiments using real-life ad auction data show that the proposed method outperforms popular bandit algorithms.