Zn_0.94Mg_0.01TM_0.05O (TM=Ni, Co, Mn, and Cr) solutions were synthesized by the sol–gel technique using Zn, Mg, Ni, Co, Mn, and Cr based alkoxide. The effects of doping transition metals and annealing temperature on structure and mechanical properties of the ZnMgO nanoparticles were systematically investigated. The phases, crystal structures, sizes and microstructures of the samples were characterized using X-ray diffraction and scanning electron microscope. Microhardness values were measured using a digital Vickers microhardness tester. The experimental results were analyzed using Kick's Law, Proportional Specimen Resistance (PSR), Elastic/Plastic Deformation (EPD) models, and Hays–Kendall (HK) approach. The microhardness values of Ni, Co, Mn, and Cr doped ZnMgO nanoparticles gradually decreased, respectively. The Indentation Size Effect behavior was observed. The grain sizes of ZnMgNiO, ZnMgCoO, ZnMgMnO, and ZnMgCrO nanoparticles were calculated approximately to be 26.36, 26.22, 22.89, and 19.24nm using the Sherrer equation, respectively.