Wearable devices such as smartwatches do not have enough power and computation capability to process computationally intensive tasks. One viable solution is to offload these tasks to the connected smartphone. Existing Android smartphones allocate CPU resources to a task according to its performance requirement, which is determined by the context of the task. However, due to lack of context information, smartphones cannot properly allocate resources to tasks offloaded from wearable devices. Allocating too few resources to urgent tasks (related to user interaction) may cause high interaction latency on wearable devices, while allocating too many resources to unimportant tasks (unrelated to user interaction) may lead to energy waste on the smartphone. To solve this problem, we propose a context-aware task offloading (CATO) framework, in which offloaded tasks can be properly executed on the smartphone or further offloaded to the cloud based on their context, aiming to achieve a balance between good user experience on wearable devices and energy saving on the smartphone. To validate our design, we have implemented CATO on the Android platform and developed two applications on top of it. Experimental results show that CATO can significantly reduce latency for urgent tasks and save energy for other unimportant tasks.