Preface
When implementing test automation, the most troublesome thing is probably encountering those “unstable tests”. These tests, without any change to the code, sometimes pass and sometimes fail. This inconsistency leads to distrust of the automation suite and wastes a great deal of precious time on constant debugging.
In this article we will share a simple but effective solution for handling unstable tests in Playwright: a custom retry utility function.
What Is an Unstable Test?
Unstable tests (or “flaky tests”) refer to tests that fail intermittently not because of code defects. Even if both the application and the test code remain unchanged, their execution results may differ.
Why Do Tests Become Unstable?
There are many reasons tests become unstable; the most common include the following:
- Network latency: APIs or page resources load slowly, causing test timeouts.
- Slow server response: The backend server is overloaded or business processing takes too long, leading to an excessively long response time.
- Third-party dependencies: External services relied on (such as payment gateways or social media APIs) are not stable enough, working intermittently.
- Race conditions: The test tries to interact with an element before the application has fully rendered or updated its state.
- Browser differences: Different browsers handle page loading and JavaScript execution slightly differently.
A Simple Solution
Playwright has a built-in automatic retry mechanism, but it targets the entire test.
Sometimes, if we only want to retry a part of the test instead of executing it from the start, we can define a custom retry function to handle it.
The function takes an operation (a function) as a parameter and tries to execute it multiple times on failure.
Defining the Retry Decorator
Create the retry_decorator.py file
| |
The core idea of this retry decorator:
- Loop
max_Retriestimes. - Execute the passed
Callablefunction in thetryblock. - If it succeeds, return the result immediately.
- If it fails, catch the error, wait one second, then make the next attempt.
- If all attempts fail, raise the last error.
Using it in a test script file
| |
Pros and Cons of the Solution
Pros:
- Simplicity: Easy to understand and implement.
- Control: You can precisely control which operations need retrying, instead of retrying the entire test.
- Reusability: The decorator can be reused across multiple tests.
Cons:
- Masking problems: Overuse may mask the real root cause of instability (such as performance issues or race conditions).
- Increased test time: If tests fail frequently, retries significantly increase total execution time.
- Harder debugging: When a test fails, it may be unclear which attempt failed, and the logs may become messy.
Conclusion
Through this custom retry utility, we provide a fine-grained control layer for handling unstable tests with intermittent failures in Playwright, allowing recovery when a specific operation fails without abandoning the entire test.
Of course, in actual use, you should also avoid overuse based on the real situation of the business under test. It is best treated as a temporary solution or a patch for known, hard-to-fix instability.
For unstable tests, the most fundamental approach is still to find and resolve the root cause of the instability.