Quantum interference of two independent particles in pure quantum states is fully described by the particles’ distinguishability: the closer the particles are to being identical, the higher the degree of quantum interference. When more than two particles are involved, the situation becomes more complex and interference capability extends beyond pairwise distinguishability, taking on a surprisingly rich character. Here, we study many-particle interference using three photons. We show that the distinguishability between pairs of photons is not sufficient to fully describe the photons’ behavior in a scattering process, but that a collective phase, the triad phase, plays a role. We are able to explore the full parameter space of three-photon interference by generating heralded single photons and interfering them in a fiber tritter. Using multiple degrees of freedom—temporal delays and polarization—we isolate three-photon interference from two-photon interference. Our experiment disproves the view that pairwise two-photon distinguishability uniquely determines the degree of nonclassical many-particle interference.