Influential work on human thinking suggests that our judgment is often biased because we minimize cognitive effort and intuitively substitute hard questions by easier ones. A key question is whether or not people realize that they are doing this and notice their mistake. Here, we test this claim with one of the most publicized examples of the substitution bias, the bat-and-ball problem. We designed an isomorphic control version in which reasoners experience no intuitive pull to substitute. Results show that people are less confident in their substituted, erroneous bat-and-ball answer than in their answer on the control version that does not give rise to the substitution. Contrary to popular belief, this basic finding indicates that biased reasoners are not completely oblivious to the substitution and sense that their answer is questionable. This calls into question the characterization of the human reasoner as a happy fool who blindly answers erroneous questions without realizing it.