Direct growth of orthorhombic Ta₃N₅-type Ta-O-N compound thin films, specifically Ta_3-xN_5-yO_y, on Si and sapphire substrates with various atomic fractions is realized by unbalanced magnetron sputtering. Low-degree fiber-textural Ta_3-xN_5-yO_y films were grown through reactive sputtering of Ta in a gas mixture of N₂, Ar, and O₂ with keeping a partial pressure ratio of 3:2:0.1 in a total working pressure range of 5–30 mTorr. With increasing total pressure from 5 to 30 mTorr, the atomic fraction of O in the as-grown Ta_3-xN_5-yO_y films was found to increase from 0.02 to 0.15 while that of N and Ta decrease from 0.66 to 0.54 and 0.33 to 0.31, respectively, leading to a decrease in b lattice constant up to around 1.3%. Metallic TaN_x phases were formed without oxygen. For a working pressure of 40 mTorr, an amorphous, O-rich Ta-N-O compound film with a high O fraction of ~0.48, was formed, mixed with non-stoichiometric TaON and Ta₂O₅. By analyzing the plasma discharge, the increasing O incorporation is associated with oxide formation on top of the Ta target due to a higher reactivity of Ta with O than with N. The increase of O incorporation in the films also leads to a bandgap widening from ~2.22 to ~2.96 eV, which is in agreement with the structural change from a crystalline Ta_3-xN_5-yO_y to an amorphous O-rich Ta-O-N compound.