We present a technique for optimizing advanced optical imaging methods for nanoscale structures, such as those encountered in the inspection of cutting-edge semiconductor devices. The optimization flow is divided to two parts: simulating light-structure interaction using the finite-difference time-domain (FDTD) method and simulating the optical imaging system by means of its optical transfer function. As a case study, FDTD is used to simulate 10-nm silicon line-space and static random-access memory patterns, with irregular structural protrusions and silicon-oxide particles as defects of interest. An ultraviolet scanning-spot optical microscope is used to detect these defects, and the optimization flow is used to find the optimal imaging mode for detection.