Internet of Things (IoT) devices have become ubiquitous, with applications in many domains, including industry, transportation, and healthcare; these devices also have many household applications. The proliferation of IoT devices has raised security and privacy concerns, however many manufacturers neglect these aspects, focusing solely on the core functionality of their products due to the short time to market and the need to reduce product costs. Consequently, vulnerable IoT devices are left unpatched, allowing attackers to exploit them for various purposes, which include compromising the device users’ privacy or recruiting the devices to an IoT botnet. We present a practical and effective host-based anomaly detection system for IoT devices (HADES-IoT) as a novel last line of defense. HADES-IoT has proactive detection capabilities that enable the execution of any malicious process to be stopped before it even starts. HADES-IoT provides tamper-proof protection and can be deployed on a wide range of Linux-based IoT devices. HADES-IoT’s main advantage is its low overhead, making it suitable for Linux-based IoT devices where state-of-the-art security solutions are infeasible due to their high-performance demands. We deployed HADES-IoT on seven IoT devices, where it demonstrated 100% effectiveness in the detection of IoT malware, including VPNFilter, IoT Reaper, and Mirai malware, while requiring only 5.5% (on average) of the available memory and consuming just negligible CPU resources.