The Internet of Things (IoT) exemplifies a large network of sensing and actuating devices that have penetrated into the physical world enabling new applications like smart homes, intelligent transportation, smart healthcare and smart cities. Through IoT, these applications have consolidated in the modern world to generate, share, aggregate and analyze large amount of security-critical and privacy sensitive data. As this consolidation gets stronger, the need for security in IoT increases. With first line of defense strategies like cryptography being unsuited due to the resource constrained nature, second line of defense mechanisms are crucial to ensure security in IoT networks. This paper presents a comprehensive study of existing second line of defense mechanisms for standardized protocols in IoT networks. The paper analyzes existing mechanisms in three aspects: Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS) and Intrusion Response Systems (IRS). We begin by providing an overview of standardized protocol stack, its layers and defensive security systems in IoT. From there, we build our narrative by presenting an extended taxonomy of IDS, IPS and IRS classifying them on their techniques, deployment, attacks, datasets, evaluation metrics and data pre-processing methods. We then thoroughly review, compare and analyze the research proposals in this context, considering the unique characteristics involved in these systems. Based on the extensive analysis of the existing defensive security systems, the paper also identifies open research challenges and directions for effective design of such systems for IoT networks, which could guide future research in the area.