A strong motivation for one individual to aggregate with others is to reduce their vigilance because other group members provide coverage and warning of approaching predators. This collective vigilance means that a focal individual is usually less susceptible to predation than when alone. However, individuals differ in their vigilance levels depending on status and context. They may also differ in how they adjust their vigilance levels as group size changes. This flexibility in response means that the collective vigilance of a group, and hence its optimal size, is not intuitive. We demonstrate, in both natural and experimental systems, that male and female pheasants, Phasianus colchicus, in harems differentially adjusted their vigilance levels as harem size changed. Females became less vigilant as harems became larger, and benefited by increasing their foraging time. Conversely, males became more vigilant as harems became larger. We calculated the collective probability that a harem would detect a predator. Within natural harem sizes, a male and two females exhibited the highest probability of collective detection, with decreases as more females joined. This optimal harem size matched the average harem size observed at our study site. Females may join harems for benefits of collective vigilance. Despite both sexes having a shared interest in larger harems for mating benefits, optimal harem size is influenced by trade-offs in a nonsexual behaviour, vigilance. This results in males with relatively small harems, females associating with less preferred males and each male being surrounded by fewer females than he could mate with.