This paper focuses on a full-duplex multiple-input multiple-output two-way untrusted relay system, and investigates the optimal beamforming design of such system to maximize its secrecy sum rate (SSR) based on the physical layer security technology. We first provide the modeling of SSR under a general beamforming setting as well as the theoretical formulation of the optimal beamforming design problem. Based on the ideal assumption that the full channel state information is available, we then develop a novel theoretical framework to solve the optimal design problem and thus establish an upper bound on SSR, where the techniques of alternate optimization, fractional programming, semi-definite programming and barrier function method are jointly employed. With the consideration of the constraints in practical implementations, we further propose two sub-optimal beamforming design solutions based on either Wiener prediction or asymptotic approximation. The issue of how to reduce the computational complexity in the optimal relay beamforming design is also discussed in this paper. Finally, we present extensive numerical results to illustrate our theoretical findings and to demonstrate the performance of the proposed sub-optimal beamforming design solutions.