Androgens play a key role in skeletal growth and bone maintenance; however, their mechanism of action remains unclear. To address this, we selectively deleted the androgen receptor (AR) in terminally differentiated, mineralizing osteoblasts using the Cre/loxP system in mice (osteocalcin‐Cre AR knockouts [mOBL‐ARKOs]). Male mOBL‐ARKOs had decreased femoral trabecular bone volume compared with littermate controls because of a reduction in trabecular number at 6, 12, and 24 wk of age, indicative of increased bone resorption. The effects of AR inactivation in mineralizing osteoblasts was most marked in the young mutant mice at 6 wk of age when rates of bone turnover are high, with a 35% reduction in trabecular bone volume, decreased cortical thickness, and abnormalities in the mineralization of bone matrix, characterized by increased unmineralized bone matrix and a decrease in the amount of mineralizing surface. This impairment in bone architecture in the mOBL‐ARKOs persisted throughout adulthood despite an unexpected compensatory increase in osteoblast activity. Our findings show that androgens act through the AR in mineralizing osteoblasts to maintain bone by regulating bone resorption and the coordination of bone matrix synthesis and mineralization, and that this action is most important during times of bone accrual and high rates of bone remodeling.