A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Flakeflagger: Predicting flakiness without rerunning tests

A Alshammari, C Morris, M Hilton… - 2021 IEEE/ACM 43rd …, 2021 - ieeexplore.ieee.org
When developers make changes to their code, they typically run regression tests to detect if
their recent changes (re) introduce any bugs. However, many tests are flaky, and their …

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

B Zolfaghari, RM Parizi, G Srivastava… - Software: Practice …, 2021 - Wiley Online Library
A flaky test is a test that may lead to different results in different runs on a single code under
test without any change in the test code. Test flakiness is a noxious phenomenon that slows …

An empirical study of flaky tests in python

M Gruber, S Lukasczyk, F Kroiß… - 2021 14th IEEE …, 2021 - ieeexplore.ieee.org
Tests that cause spurious failures without any code changes, ie, flaky tests, hamper
regression testing, increase maintenance costs, may shadow real bugs, and decrease trust …

Research progress of flaky tests

W Zheng, G Liu, M Zhang, X Chen… - 2021 IEEE International …, 2021 - ieeexplore.ieee.org
A flaky test is a test that both passes and fails periodically without any code changes, and its
uncontrolled uncertainty will destroy the value of the test suites and even cause developers …

The secret life of test smells-an empirical study on test smell evolution and maintenance

DJ Kim, TH Chen, J Yang - Empirical Software Engineering, 2021 - Springer
In recent years, researchers and practitioners have been studying the impact of test smells
on test maintenance. However, there is still limited empirical evidence on why developers …

When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla

J Lampel, S Just, S Apel, A Zeller - Proceedings of the 29th ACM Joint …, 2021 - dl.acm.org
Continuous delivery of cloud systems requires constant running of jobs (build processes,
tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent …

A replication study on the usability of code vocabulary in predicting flaky tests

G Haben, S Habchi, M Papadakis… - 2021 IEEE/ACM 18th …, 2021 - ieeexplore.ieee.org
Industrial reports indicate that flaky tests are one of the primary concerns of software testing
mainly due to the false signals they provide. To deal with this issue, researchers have …

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …

Know you neighbor: Fast static prediction of test flakiness

R Verdecchia, E Cruciani, B Miranda… - IEEE Access, 2021 - ieeexplore.ieee.org
Context: Flaky tests plague regression testing in Continuous Integration environments by
slowing down change releases and wasting testing time and effort. Despite the growing …