SOAP-based Web services is a middleware technology marketed as the solution to easy data exchange between heterogeneous IT architectures. The large number of scenarios, in which this technology is used, has introduced demands for new extensions raising its complexity. However, this has also introduced a large variety of new attacks. In this paper, we investigate an automatic evaluation of Web service specific Denial of Service (DoS) attacks. We present a new fully automated plugin for the WS-Attacker penetration testing tool implementing major DoS attacks. Our tool determines the attack success without having physical access to the target machine, using a novel blackbox approach. We give an overview of our design decisions and present the evaluation results using common Web service frameworks and systems.