Lithium niobate-on-insulator (LNOI) is an emerging photonic platform that exhibits favorable material properties (such as low optical loss, strong nonlinearities, and stability) and enables large-scale integration with stronger optical confinement, showing promise for future optical networks, quantum processors, and nonlinear optical systems. However, while photonics engineering has entered the era of automated “inverse design” via optimization in recent years, the design of LNOI integrated photonic devices still mostly relies on intuitive models and inefficient parameter sweeps, limiting the accessible parameter space, performance, and functionality. Here, we implement a 3D gradient-based inverse-design model tailored for topology optimization based on the LNOI platform, which not only could efficiently search a large parameter space, but also takes into account practical fabrication constraints, including minimum feature sizes and etched sidewall angles. We experimentally demonstrate a spatial-mode multiplexer, a waveguide crossing, and a compact waveguide bend, all with low insertion losses, tiny footprints, and excellent agreement between simulation and experimental results. The devices, together with the design methodology, represent a crucial step toward the variety of advanced device functionalities needed in future LNOI photonics and could provide compact and cost-effective solutions for future optical links, quantum technologies, and nonlinear optics.