Flake it'till you make it: Using automated repair to induce and fix latent test flakiness

O Parry, GM Kapfhammer, M Hilton… - Proceedings of the IEEE …, 2020 - dl.acm.org
Since flaky tests pass or fail nondeterministically, without any code changes, they are an
unreliable indicator of program quality. Developers may quarantine or delete flaky tests …

What do developer-repaired flaky tests tell us about the effectiveness of automated flaky test detection?

O Parry, GM Kapfhammer, M Hilton… - Proceedings of the 3rd …, 2022 - dl.acm.org
Because they pass or fail without code changes, flaky tests cause serious problems such as
spuriously failing builds and the eroding of developers' trust in tests. Many previous …

Do Automatic Test Generation Tools Generate Flaky Tests?

M Gruber, MF Roslan, O Parry, F Scharnböck… - Proceedings of the 46th …, 2024 - dl.acm.org
Non-deterministic test behavior, or flakiness, is common and dreaded among developers.
Researchers have studied the issue and proposed approaches to mitigate it. However, the …

A qualitative study on the sources, impacts, and mitigation strategies of flaky tests

S Habchi, G Haben, M Papadakis… - … IEEE Conference on …, 2022 - ieeexplore.ieee.org
Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …

Modeling and ranking flaky tests at Apple

E Kowalczyk, K Nair, Z Gao, L Silberstein… - Proceedings of the …, 2020 - dl.acm.org
Test flakiness---inability to reliably repeat a test's Pass/Fail outcome---continues to be a
significant problem in Industry, adversely impacting continuous integration and test …

Identifying randomness related flaky tests through divergence and execution tracing

A Ahmad, EN Held, O Leifler… - 2022 IEEE International …, 2022 - ieeexplore.ieee.org
Developers often spend time to determine whether test case failures are real failures or
flaky. The flaky tests, known as non-deterministic tests, change their outcomes without any …

What made this test flake? pinpointing classes responsible for test flakiness

S Habchi, G Haben, J Sohn, A Franci… - 2022 IEEE …, 2022 - ieeexplore.ieee.org
Flaky tests are defined as tests that manifest non-deterministic behaviour by passing and
failing intermittently for the same version of the code. These tests cripple continuous …

Root causing flaky tests in a large-scale industrial setting

W Lam, P Godefroid, S Nath, A Santhiar… - Proceedings of the 28th …, 2019 - dl.acm.org
In today's agile world, developers often rely on continuous integration pipelines to help build
and validate their changes by executing tests in an efficient manner. One of the significant …

Surveying the developer experience of flaky tests

O Parry, GM Kapfhammer, M Hilton… - Proceedings of the 44th …, 2022 - dl.acm.org
Test cases that pass and fail without changes to the code under test are known as flaky. The
past decade has seen increasing research interest in flaky tests, though little attention has …

A survey on how test flakiness affects developers and what support they need to address it

M Gruber, G Fraser - 2022 IEEE Conference on Software …, 2022 - ieeexplore.ieee.org
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …