WTLS (wireless transport layer security) is an important standard protocol for secure wireless access to Internet services. WTLS employs public-key cryptosystems during the handshake between mobile client and WAP gateway (server). Several cryptosystems at different key strengths can be used in WTLS. The trade-off is security versus processing and transmission time. In this paper, an analytical performance model for public-key cryptosystem operations in WTLS protocol is developed. Different handshake protocols, different cryptosystems and key sizes are considered. Public-key cryptosystems are implemented using state-of-the-art performance improvement techniques, yielding actual performance figures for individual cryptosystems. These figures and the analytical model are used to calculate the cost of using public-key cryptosystems in WTLS. Results for different cryptosystems and handshake protocols are comparatively depicted and interpreted. It has been observed that ECC (elliptic curve cryptography) performs better than its rival RSA cryptosystem in WTLS. Performance of some stronger ECC curves, which are not considered in WTLS standard, is also analyzed. Results showed that some of those curves could be used in WTLS for high security applications with an acceptable degradation in performance.