A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

[HTML][HTML] Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …

[HTML][HTML] An empirical study of automated unit test generation for Python

S Lukasczyk, F Kroiß, G Fraser - Empirical Software Engineering, 2023 - Springer
Various mature automated test generation tools exist for statically typed programming
languages such as Java. Automatically generating unit tests for dynamically typed …

Pyty: Repairing static type errors in python

YW Chow, L Di Grazia, M Pradel - Proceedings of the IEEE/ACM 46th …, 2024 - dl.acm.org
Gradual typing enables developers to annotate types of their own choosing, offering a
flexible middle ground between no type annotations and a fully statically typed language. As …

A survey on how test flakiness affects developers and what support they need to address it

M Gruber, G Fraser - 2022 IEEE Conference on Software …, 2022 - ieeexplore.ieee.org
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …

Preempting flaky tests via non-idempotent-outcome tests

A Wei, P Yi, Z Li, T Xie, D Marinov, W Lam - Proceedings of the 44th …, 2022 - dl.acm.org
Regression testing can greatly help in software development, but it can be seriously
undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on …

Evaluating features for machine learning detection of order-and non-order-dependent flaky tests

O Parry, GM Kapfhammer, M Hilton… - 2022 IEEE Conference …, 2022 - ieeexplore.ieee.org
Flaky tests are test cases that can pass or fail without code changes. They often waste the
time of software developers and obstruct the use of continuous integration. Previous work …

An empirical study of flaky tests in javascript

N Hashemi, A Tahir, S Rasheed - 2022 IEEE International …, 2022 - ieeexplore.ieee.org
Flaky tests (tests with non-deterministic outcomes) can be problematic for testing efficiency
and software reliability. Flaky tests in test suites can also significantly delay software …

Predicting flaky tests categories using few-shot learning

A Akli, G Haben, S Habchi, M Papadakis… - arXiv preprint arXiv …, 2022 - arxiv.org
Flaky tests are tests that yield different outcomes when run on the same version of a
program. This non-deterministic behaviour plagues continuous integration with false signals …

iPFlakies: A framework for detecting and fixing python order-dependent flaky tests

R Wang, Y Chen, W Lam - Proceedings of the ACM/IEEE 44th …, 2022 - dl.acm.org
Developers typically run tests after code changes. Flaky tests, which are tests that can
nondeterministically pass and fail when run on the same version of code, can mislead …