Pinning at local defects is a significant road block for the successful implementation of technological paradigms which rely on the dynamic properties of non-trivial magnetic textures. Here, a comprehensive study of the influence of local pinning sites for non-homogeneous magnetic layers integrated as the free layer of a magnetic tunnel junction is presented, both experimentally and with corresponding micromagnetic simulations. The pinning sites are found to be extremely detrimental to the frequency controllability of the devices, a key requirement for their use as synapses in a frequency multiplexed artificial neural networks. In addition to describing the impact of the local pinning sites in the more conventional NiFe, a vortex-based magnetic tunnel junction with an amorphous free layer is presented which shows significantly improved frequency selectivity, marking a clear direction for the design of future low power devices.