A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Root causing flaky tests in a large-scale industrial setting

W Lam, P Godefroid, S Nath, A Santhiar… - Proceedings of the 28th …, 2019 - dl.acm.org
In today's agile world, developers often rely on continuous integration pipelines to help build
and validate their changes by executing tests in an efficient manner. One of the significant …

Flakeflagger: Predicting flakiness without rerunning tests

A Alshammari, C Morris, M Hilton… - 2021 IEEE/ACM 43rd …, 2021 - ieeexplore.ieee.org
When developers make changes to their code, they typically run regression tests to detect if
their recent changes (re) introduce any bugs. However, many tests are flaky, and their …

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

B Zolfaghari, RM Parizi, G Srivastava… - Software: Practice …, 2021 - Wiley Online Library
A flaky test is a test that may lead to different results in different runs on a single code under
test without any change in the test code. Test flakiness is a noxious phenomenon that slows …

A study on the lifecycle of flaky tests

W Lam, K Muşlu, H Sajnani… - Proceedings of the ACM …, 2020 - dl.acm.org
During regression testing, developers rely on the pass or fail outcomes of tests to check
whether changes broke existing functionality. Thus, flaky tests, which nondeterministically …

iFixFlakies: A framework for automatically fixing order-dependent flaky tests

A Shi, W Lam, R Oei, T Xie, D Marinov - … of the 2019 27th ACM Joint …, 2019 - dl.acm.org
Regression testing provides important pass or fail signals that developers use to make
decisions after code changes. However, flaky tests, which pass or fail even when the code …

An empirical study of flaky tests in python

M Gruber, S Lukasczyk, F Kroiß… - 2021 14th IEEE …, 2021 - ieeexplore.ieee.org
Tests that cause spurious failures without any code changes, ie, flaky tests, hamper
regression testing, increase maintenance costs, may shadow real bugs, and decrease trust …

Flakify: A black-box, language model-based predictor for flaky tests

S Fatima, TA Ghaleb, L Briand - IEEE Transactions on Software …, 2022 - ieeexplore.ieee.org
Software testing assures that code changes do not adversely affect existing functionality.
However, a test case can be flaky, ie, passing and failing across executions, even for the …

What is the vocabulary of flaky tests?

G Pinto, B Miranda, S Dissanayake… - Proceedings of the 17th …, 2020 - dl.acm.org
Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research
activity on this topic, no effort has been made on understanding the vocabulary of flaky tests …

Detecting flaky tests in probabilistic and machine learning applications

S Dutta, A Shi, R Choudhary, Z Zhang, A Jain… - Proceedings of the 29th …, 2020 - dl.acm.org
Probabilistic programming systems and machine learning frameworks like Pyro, PyMC3,
TensorFlow, and PyTorch provide scalable and efficient primitives for inference and training …