Dielectric loss from two-level states is shown to be a dominant decoherence source in superconducting quantum bits. Depending on the qubit design, dielectric loss from insulating materials or the tunnel junction can lead to short coherence times. We show that a variety of microwave and qubit measurements are well modeled by loss from resonant absorption of two-level defects. Our results demonstrate that this loss can be significantly reduced by using better dielectrics and fabricating junctions of small area . With a redesigned phase qubit employing low-loss dielectrics, the energy relaxation rate has been improved by a factor of 20, opening up the possibility of multiqubit gates and algorithms.