We study the differential polynomial rings which are defined using the special geometry of the moduli spaces of Calabi-Yau threefolds. The higher genus topological string amplitudes are expressed as polynomials in the generators of these rings, giving them a global description in the moduli space. At particular loci, the amplitudes yield the generating functions of Gromov-Witten invariants. We show that these rings are isomorphic to the rings of quasi modular forms for threefolds with duality groups for which these are known. For the other cases, they provide generalizations thereof. We furthermore study an involution which acts on the quasi modular forms. We interpret it as a duality which exchanges two distinguished expansion loci of the topological string amplitudes in the moduli space. We construct these special polynomial rings and match them with known quasi modular forms for non-compact Calabi-Yau geometries and their mirrors including local and local del Pezzo geometries with , , and type singularities. We provide the analogous special polynomial ring for the quintic.