Dependable computer systems hosting critical commerce, transportation, and military applications, among others, must satisfy stringent quality-of-service (QoS) requirements. However, as these systems become increasingly complex, maintaining the desired QoS by manually tuning the numerous performance-related parameters are very difficult. This paper develops a generic online control framework to design self-managing computer systems. The proposed approach explores a limited region of the system state-space at each time step and decides the best control action accordingly. We present two case studies to demonstrate the practicality of the proposed control framework.