Symmetry-driven phenomena arising in nonlocal metasurfaces supporting quasi-bound states in the continuum (q-BICs) have been opening new avenues to tailor enhanced light-matter interactions via perturbative design principles. Geometric phase concepts - observed in many physical systems - are particularly useful in nonlocal metasurfaces, as they enable to locally pattern the q-BIC scattering rate and phase across the metasurface aperture without affecting the delocalized nature of the q-BIC resonance. However, this control typically comes with stringent limitations in terms of efficiency and/or of polarization operation. Here, we unveil a new form of geometric phase control, accumulated along a continuous contour of geometric perturbations that parametrically encircle a singularity associated with a bound state. This response is obtained regardless of the chosen polarization state, which can be rationally specified by a judicious choice of the perturbation geometry. Our findings extend geometric phase concepts to arbitrary polarizations, including linear and elliptical states, with near-unity scattering efficiency. This polarization-agnostic geometric phase offers new opportunities for wavefront manipulation, multifunctional operation and light emission, with applications in augmented reality, secure communications, wireless systems, imaging, lasing, nonlinear and quantum optics.