Reproducibility is fundamental to science and a recent matter of discussion. We report challenges for conceptual replications when employing different readout measures to target the same theoretical construct, particularly those requiring probed reactions. This was addressed in a fear conditioning paradigm, a prototype of emotional learning, in three experimental groups (N_total = 57). We demonstrate that the inclusion of startle probes (95 dB burst of white noise) to elicit a startle reflex delays the acquisition of fear as reflected by skin conductance responses and intermittent fear ratings. The inclusion of fear ratings in turn did not significantly affect fear acquisition. Hence, subtle differences in experimental design, such as the inclusion of a single additional outcome measure, may substantially impact on study results and interpretations. Thus, small effects, as are common in between‐group comparisons, may be particularly susceptible to this, which has strong implications for the replication of findings across studies employing slightly different experimental designs.