The hybrid system is increasingly important in waterborne transportation due to an increase in fuel prices and people’s awareness of climate change. Hybrid system is basically a combination of an internal combustion engine and an electric machine (EM). In this paper, an optimal design of EM, that is, the brushless direct current motor for plug-in hybrid electric boat is introduced. The EM model is developed in Matlab/Simulink SimPowerSystem environment together with the closed-loop feedback PI controller. By using a power demand curve as a reference for the model, the optimal performance of the machine is obtained by using the particle swarm optimization. The results of optimal control parameters of the system are compared with those of the trial and error method. It has been found that the proposed optimal system design can improve the machine performance significantly.