A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Flakeflagger: Predicting flakiness without rerunning tests

A Alshammari, C Morris, M Hilton… - 2021 IEEE/ACM 43rd …, 2021 - ieeexplore.ieee.org
When developers make changes to their code, they typically run regression tests to detect if
their recent changes (re) introduce any bugs. However, many tests are flaky, and their …

What is the vocabulary of flaky tests?

G Pinto, B Miranda, S Dissanayake… - Proceedings of the 17th …, 2020 - dl.acm.org
Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research
activity on this topic, no effort has been made on understanding the vocabulary of flaky tests …

Shake it! detecting flaky tests caused by concurrency with shaker

D Silva, L Teixeira, M d'Amorim - 2020 IEEE International …, 2020 - ieeexplore.ieee.org
A test is said to be flaky when it non-deterministically passes or fails. Test flakiness
negatively affects the effectiveness of regression testing and, consequently, impacts …

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …

A survey on how test flakiness affects developers and what support they need to address it

M Gruber, G Fraser - 2022 IEEE Conference on Software …, 2022 - ieeexplore.ieee.org
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …

A qualitative study on the sources, impacts, and mitigation strategies of flaky tests

S Habchi, G Haben, M Papadakis… - … IEEE Conference on …, 2022 - ieeexplore.ieee.org
Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …

A replication study on the usability of code vocabulary in predicting flaky tests

G Haben, S Habchi, M Papadakis… - 2021 IEEE/ACM 18th …, 2021 - ieeexplore.ieee.org
Industrial reports indicate that flaky tests are one of the primary concerns of software testing
mainly due to the false signals they provide. To deal with this issue, researchers have …

Evaluating features for machine learning detection of order-and non-order-dependent flaky tests

O Parry, GM Kapfhammer, M Hilton… - 2022 IEEE Conference …, 2022 - ieeexplore.ieee.org
Flaky tests are test cases that can pass or fail without code changes. They often waste the
time of software developers and obstruct the use of continuous integration. Previous work …

Test flakiness across programming languages

K Barbosa, R Ferreira, G Pinto… - IEEE Transactions …, 2022 - ieeexplore.ieee.org
Regression Testing (RT) is a quality-assurance practice commonly adopted in the software
industry to check if functionality remains intact after code changes. Test flakiness is a serious …