Using magnetic nanoparticles (MNPs) for emulsified oil separation from wastewater is becoming increasingly widespread. This study aims to synthesize MNPs using amphiphilic coatings to stabilize the MNPs and prevent their agglomeration for efficiently breaking oil-in-water nanoemulsions. We coat two different sizes of Fe₃O₄ nanoparticles (15–20 and 50–100 nm) using cetyltrimethylammonium bromide (CTAB) and sodium dodecyl sulfate (SDS) with surfactant-to-MNP mass ratios of 0.4 and 0.8. We study the effect of various variables on the demulsification performance, including the MNP size and concentration, coating materials, and MNP loading. Based on the oil–water separation analysis, the smaller size MNPs (MNP-S) show a better demulsification performance than the larger ones (MNP-L ) for a 1000 ppm dodecane-in-water emulsion containing nanosized oil droplets (250–300 nm). For smaller MNPs (MNP-S) and at low dosage level of 0.5 g/L, functionalizing with surfactant-to-MNP mass ratio of 0.4, the functionalization increases the separation efficiency (SE) from 57.5% for bare MNP-S to 86.1% and 99.8 for the SDS and CTAB coatings, respectively. The highest SE for MNP-S@CTAB and the zeta potential measurements imply that electrostatic attraction between negatively charged oil droplets (−55.9 ± 2.44 mV) and positively charged MNP-S@CTAB (+35.8 ± 0.34 mV) is the major contributor to a high SE. Furthermore, the reusability tests for MNP-S@CTAB reveal that after 10 cycles, the amount of oil adsorption capacity decreases slightly, from 20 to 19 mg/g, indicating an excellent stability of synthesized nanoparticles. In conclusion, functionalized MNPs with tailored functional groups feature a high oil SE that could be effectively used for oil separation from emulsified oily wastewater streams.