A qualitative study on the sources, impacts, and mitigation strategies of flaky tests

S Habchi, G Haben, M Papadakis… - … IEEE Conference on …, 2022 - ieeexplore.ieee.org
Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …

A survey on how test flakiness affects developers and what support they need to address it

M Gruber, G Fraser - 2022 IEEE Conference on Software …, 2022 - ieeexplore.ieee.org
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …

Surveying the developer experience of flaky tests

O Parry, GM Kapfhammer, M Hilton… - Proceedings of the 44th …, 2022 - dl.acm.org
Test cases that pass and fail without changes to the code under test are known as flaky. The
past decade has seen increasing research interest in flaky tests, though little attention has …

[HTML][HTML] Test flakiness' causes, detection, impact and responses: A multivocal review

A Tahir, S Rasheed, J Dietrich, N Hashemi… - Journal of Systems and …, 2023 - Elsevier
Flaky tests (tests with non-deterministic outcomes) pose a major challenge for software
testing. They are known to cause significant issues, such as reducing the effectiveness and …

A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Modeling and ranking flaky tests at Apple

E Kowalczyk, K Nair, Z Gao, L Silberstein… - Proceedings of the …, 2020 - dl.acm.org
Test flakiness---inability to reliably repeat a test's Pass/Fail outcome---continues to be a
significant problem in Industry, adversely impacting continuous integration and test …

Research progress of flaky tests

W Zheng, G Liu, M Zhang, X Chen… - 2021 IEEE International …, 2021 - ieeexplore.ieee.org
A flaky test is a test that both passes and fails periodically without any code changes, and its
uncontrolled uncertainty will destroy the value of the test suites and even cause developers …

iDFlakies: A framework for detecting and partially classifying flaky tests

W Lam, R Oei, A Shi, D Marinov… - 2019 12th ieee …, 2019 - ieeexplore.ieee.org
Regression testing is increasingly important with the wide use of continuous integration. A
desirable requirement for regression testing is that a test failure reliably indicates a problem …

Understanding flaky tests: The developer's perspective

M Eck, F Palomba, M Castelluccio… - Proceedings of the 2019 …, 2019 - dl.acm.org
Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) despite
exercising unchanged code. In this work, we examine the perceptions of software …

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

B Zolfaghari, RM Parizi, G Srivastava… - Software: Practice …, 2021 - Wiley Online Library
A flaky test is a test that may lead to different results in different runs on a single code under
test without any change in the test code. Test flakiness is a noxious phenomenon that slows …