Regression tests should consistently produce the same outcome when executed against the
same version of the system under test. Recent studies, however, show a different picture: in …

Mutation testing is widely used in research as a metric for evaluating the quality of test
suites. Mutation testing runs the test suite on generated mutants (variants of the code under …

Context: Flaky tests plague regression testing in Continuous Integration environments by
slowing down change releases and wasting testing time and effort. Despite the growing …

Many machine learning (ML) algorithms are inherently random–multiple executions using
the same inputs may produce slightly different results each time. Randomness impacts how …

Regression testing can greatly help in software development, but it can be seriously
undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on …

Developers typically rely on regression testing techniques to ensure that their changes do
not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which …

Library developers can provide classes and methods with underdetermined specifications
that allow flexibility in future implementations. Library users may write code that relies on a …

Flaky tests are test cases that can pass or fail without code changes. They often waste the
time of software developers and obstruct the use of continuous integration. Previous work …

Flaky tests are tests that pass or fail nondeterministically on the same version of code. These
tests can mislead developers concerning the quality of their code changes during regression …

Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests
run. While TCP has been studied for over 20 years, it was almost always evaluated using …