As an important application of the Internet of Things, many remote monitoring systems adopt a device-to-cloud network paradigm. In a remote patient monitoring case, various resource-constrained devices are used to measure the health conditions of a target patient in a distant non-clinical environment and the collected data are sent to the cloud backend of an authorized health care service for processing and decision making. As the measurements involve private patient information, access control and trustworthy processing of the confidential data become very important. Software-based solutions that adopt advanced cryptographic tools, such as attribute-based encryption and fully homomorphic encryption, can address the problem, but they also impose substantial computation overhead on both client and server sides. In this paper, we deviate from the conventional software-based solutions and propose a secure and efficient remote monitoring framework, called SRM, using the latest hardware-based trustworthy computing technology, such as Intel SGX. In addition, we present a robust and lightweight “heartbeat” protocol to handle notoriously difficult key revocation problem. We implemented a prototype of the framework for SRM and show that SRM can protect user data privacy against unauthorized parties, with minimum performance cost compared to existing software-based solutions.