We report that the aqueous self-assembly behavior of citrate based gold nanoparticles decorated with the temperature responsive RAFT-based polymer poly(N-isopropylacrylamide) critically depends on the presence of salt in the medium. Both for temperature induced reversible agglomeration and for hydrogen bonding based layer-by-layer assembly with tannic acid, the presence of salt dramatically promotes the assembly behavior. We attribute this to a combination of ionic screening of the remaining citrate groups on the nanoparticle surface and a salting out effect which increases the contribution of hydrophobic interactions in the self-assembly process. These findings provide new insights into an attractive class of polymer/gold hybrid nanomaterials that can find application in biotechnology, catalysis, and biomedicine.