Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results

T Hoefler, R Belli - Proceedings of the international conference for high …, 2015 - dl.acm.org
Measuring and reporting performance of parallel computers constitutes the basis for
scientific advancement of high-performance computing (HPC). Most scientific reports show …

Robust benchmarking in noisy environments

J Chen, J Revels - arXiv preprint arXiv:1608.04295, 2016 - arxiv.org
We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter
and other environmental fluctuations, and is insensitive to the highly nonideal statistics …

Accelerating architectural simulation via statistical techniques: A survey

Q Guo, T Chen, Y Chen… - IEEE Transactions on …, 2015 - ieeexplore.ieee.org
In computer architecture research and development, simulation is a powerful way of
acquiring and predicting processor behaviors. While architectural simulation has been …

Reproducible MPI benchmarking is still not as easy as you think

S Hunold, A Carpen-Amarie - IEEE Transactions on Parallel …, 2016 - ieeexplore.ieee.org
The Message Passing Interface (MPI) is the prevalent programming model used on today's
supercomputers. Therefore, MPI library developers are looking for the best possible …

Unit testing performance with stochastic performance logic

L Bulej, T Bureš, V Horký, J Kotrč, L Marek… - Automated Software …, 2017 - Springer
Unit testing is an attractive quality management tool in the software development process,
however, practical obstacles make it difficult to use unit tests for performance testing. We …

Performance evaluation of web service response time probability distribution models for business process cycle time simulation

R Ramakrishnan, A Kaur - Journal of Systems and Software, 2020 - Elsevier
Abstract Context The adoption of Business Process Management (BPM) is enabling
companies to improve the pace of building new capabilities, enhancing existing ones, and …

WSMeter: A performance evaluation methodology for Google's production warehouse-scale computers

J Lee, C Kim, K Lin, L Cheng, R Govindaraju… - Proceedings of the …, 2018 - dl.acm.org
Evaluating the comprehensive performance of a warehouse-scale computer (WSC) has
been a long-standing challenge. Traditional load-testing benchmarks become ineffective …

Rigorous Evaluation of Computer Processors with Statistical Model Checking

F Mazurek, A Tschand, Y Wang, M Pajic… - Proceedings of the 56th …, 2023 - dl.acm.org
Experiments with computer processors must account for the inherent variability in
executions. Prior work has shown that real systems exhibit variability, and random effects …

Ontology-based dynamic process collaboration in service-oriented architecture

WT Tsai, Q Huang, J Xu, Y Chen… - … Conference on Service …, 2007 - ieeexplore.ieee.org
Service collaboration is important in service-oriented architecture (SOA). While service
collaboration protocols for data exchange are well explored, the dynamic process …

On the Effectiveness of Bisection in Performance Regression Localization

FS Ocariza, Jr - Empirical Software Engineering, 2022 - Springer
Performance regressions can have a drastic impact on the usability of a software
application. The crucial task of localizing such regressions can be achieved using bisection …