Web usability is a critical factor for visitors' acceptance and satisfaction. The prevalence of various free and proprietary website testing tools enabled the fast evaluation of the usability of websites. However, their effectiveness in providing meaningful, consistent, and valid results remains questionable. In this study, we initially devised a usability framework that incorporates 19 usability dimensions and investigated the compliance of 10 popular web usability testing tools to this framework. Next, we applied the automated evaluation to nine websites spanning three major categories, namely e-commerce, vacation rentals, and education. On a positive note, the tools inspected a wide range of aspects, including performance, SEO, page size, accessibility, and security. However, our in-depth analysis revealed numerous critical issues. First, usability seems to be ignored or distorted by most of the tools. Second, the automated tools exhibited variable and contradictory scores concerning the analysis of the same websites. Third, the analysis reports were vague and complicated for non-technical users. Fourth, the tools identified and explained issues affecting the technical implementation of the websites, overlooking web usability flaws. Fifth, only four tools (i.e., SEOptimiser, Dareboost, Website Grader, and Sure Oak) gave recommendations to remedy performance bottlenecks and improve the quality of the websites. Lastly, the inner working of the tools does not seem to incorporate the theoretical foundations of usability, which calls for urgent collaboration between industry experts and HCI practitioners and researchers.