Unmanned Aerial Vehicle (UAV) networks have huge potential for a variety of military and civilian uses, such as intelligent transportation system, smart city, and so on. The 6th Generation (6G) communication technology is expected to provide 3-Dimensional (3D) wireless coverage and greatly improve the performance of UAV networks. The UAV-to-UAV (U2U) message exchange (or message exchange for short) is an important basis of multi-UAV cooperation. However, due to the unique characteristics of UAV networks, the U2U messages (or messages for short) are vulnerable to both the external and internal attackers. In this work, we propose a Lightweight Trustworthy Message Exchange (LTME) scheme for UAV networks by efficiently aggregating the cryptography and trust management technologies. In the LTME scheme, a centralized Ground Control Station (GCS) periodically updates the reputation levels of registered UAVs (or UAVs for short) and securely distributes secret values to the UAVs. Based on the received secret values, each trustworthy broadcasting UAV can generate its encrypted messages so that only trustworthy receiving UAVs can decrypt them, and each trustworthy receiving UAV can accurately judge whether the received messages and the corresponding broadcasting UAVs are trustworthy in a lightweight manner. Furthermore, we present a simplified LTME (sLTME) scheme and conduct a comprehensive theoretical analysis and simulation evaluation for the LTME and sLTME schemes. The results demonstrate that the proposed schemes can provide rich functionality and strong robustness with low computation and communication overheads, and are significantly superior to the existing schemes in several aspects.