In these lecture notes, partly based on a course taught at the Karpacz Winter School in March 2014, we explore the close connections between non-adiabatic response of a system with respect to macroscopic parameters and the geometry of quantum and classical states. We center our discussion around adiabatic gauge potentials, which are the generators of unitary basis transformations in quantum systems and generators of special canonical transformations in classical systems. In quantum systems, eigenstate expectation values of these potentials are the Berry connections and the covariance matrix of these gauge potentials is the geometric tensor, whose antisymmetric part defines the Berry curvature and whose symmetric part is the Fubini-Study metric tensor. In classical systems one simply replaces the eigenstate expectation value by an average over the micro-canonical shell. For complicated interacting systems, we show that a variational principle may be used to derive approximate gauge potentials. We then express the non-adiabatic response of the physical observables of the system through these gauge potentials, specifically demonstrating the close connection of the geometric tensor to the notions of Lorentz force and renormalized mass. We highlight applications of this formalism to deriving counter-diabatic (dissipationless) driving protocols in various systems, as well as to finding equations of motion for slow macroscopic parameters coupled to fast microscopic degrees of freedom that go beyond macroscopic Hamiltonian dynamics. Finally, we illustrate these ideas with a number of simple examples and highlight a few more complicated ones drawn from recent literature.