The photonic crystal fiber (PCF) is one of the excellent media for generating photon pairs via spontaneous four-wave mixing. Here we study how the inhomogeneity of PCFs affect the spectral properties of photon pairs from both the theoretical and experimental aspects. The theoretical model shows that the photon pairs born in different places of the inhomogeneous PCF are coherently superposed, and a modulation in the broadened spectrum of phase-matching function will appear, which prevents the realization of spectral factorable photon pairs. In particular, the inhomogeneity induced modulation can be examined by measuring the spectrum of the individual signal or idler field when the asymmetric group-velocity matching condition is approximately fulfilled. Our experiments are performed by tailoring the spectrum of pulsed pump to satisfy the specified phase-matching condition. The observed spectra of individual signal photons, which are produced from different segments of the 1.9 m inhomogeneous PCF, agree with the theoretical predictions. The investigations are not only useful for fiber-based quantum state engineering, but also provide a dependable method to test the homogeneity of PCF.