Featured image of post Should Testers Provide Test Cases for Developers' Self-Testing?

Should Testers Provide Test Cases for Developers' Self-Testing?

Background

I have received a good number of inquiries asking whether, as a tester, one should provide test cases or test data when developers carry out self-testing.

This is a real dilemma that people often encounter in actual work, especially as many teams now emphasize shift-left testing, making it quite common to require testers to provide cases and prepare test data. At the same time, some teams still use defects as a KPI metric, and this gives rise to a host of side effects:

  1. Problems found during the development phase using the cases provided by testers will inevitably reduce the output of problems found during the testing phase.
  2. If testers provide the cases and problems appear later, is that a development omission or a testing omission?
  3. Once testers have handed the cases to developers, is it still necessary to run the same cases again during the testing phase?
  4. What is the value of testers? Is it simply to write as thorough a set of cases as possible?

The Responsibility of Testing and the Role of Test Cases

In a previous article【】we discussed the true purpose of testing in detail.

So we should be clear: the responsibility of testing is not actually to find bugs, but to reflect the product’s quality status.

In testing work, the main role of test cases is, in fact, to help us sort out and summarize the various test paths before testing begins, and to arrange in advance, according to the product requirements, the key points that need to be tested, so as to avoid omissions during actual execution.

All of the above is the basis for discussing the handover of test cases when we talk about shift-left testing.

Should Test Cases Be Handed Over to Developers?

From the team’s perspective, the ultimate concern must be product quality. According to the well-known defect-cost relationship curve, the earlier a defect is found, the lower the cost, and the stronger the confidence in delivery quality.

So from this angle, shift-left testing—including providing test cases to developers—is consistent with the quality goal.

But in actual practice, a team’s KPI orientation affects how things actually turn out.

That is, if a team takes the number of bugs as its KPI, as many do, then the effect of handing over test cases will inevitably be greatly diminished.

This KPI artificially splits the output of the development team and the testing team: developers need to produce as few bugs as possible, while testers need to find as many bugs as possible.

Test cases and test data actually reflect the testing mindset behind finding bugs. Handing the cases to developers will help them reduce bugs, but it will affect the testers’ own output of found bugs.

So under this orientation, even if testers do provide cases, it is hard to guarantee the coverage of those cases—it only becomes a matter of form.

From the standpoint of product quality, then, as a testing role, it is the duty of the job to provide test cases during the development phase and help the development team complete thorough self-testing—but for this to be truly implemented and to produce results, the prerequisite is that this metric of evaluating by bug count must be abolished.

How Should Omissions Not Covered by Cases Be Defined?

First, we must recognize one fact: it is impossible for testing to achieve 100% full coverage. In the early stages of a project, test cases written on the basis of requirements and a development build that is not yet fully complete can cover an even more limited set of test scenarios.

In actual work, even in system testing, the proportion of bugs found by executing test cases is extremely limited compared with the bugs discovered through exploration in real testing. This is precisely the starting point for promoting exploratory testing.

In other words, in the development phase, after testing has been done according to the cases, it is a 100% certainty that the subsequent system testing phase will go on to find problems that were not covered.

So how can we ensure adequate case coverage at this stage?

First, it must be emphasized once again that quality is a team goal. There is no need to agonize over defining whether it is a development omission or a testing omission—the shared goal is to reduce omissions.

Second, test cases should be viewed as the output of a thinking tool and an embodiment of testing ideas. The cases provided by testers are not necessarily the whole story; developers should likewise think about which scenarios to cover.

The requirement coverage and code coverage of the test cases are relatively suitable quantitative metrics for measuring how well cases cover this stage.

The principle for completing self-testing in the development phase is to ensure that the product’s requirements are fully implemented, that no serious blocking problems arise, and that the product’s main logic and architecture are verified.

Once the goals above have been reached, the development self-testing phase can be considered successfully completed. There is no need to dwell on whether a specific bug was covered during the self-testing phase.

Should the Same Set of Cases Still Be Used in the Testing Phase?

For a product that has undergone thorough self-testing, the test cases that have already been executed in principle do not need to be tested again after the product is submitted for testing. That is, by default, scenarios that have already been verified will not produce problems.

In actual work, however, is the developers’ self-testing thorough? Is case execution rigorous? Is there any gray area in the coverage of the cases themselves? All of these are unknowns.

The tester’s job is to prepare and assess the product’s quality status before it is released. It is hard to say that the parts already self-tested by development need no further assessment.

Therefore, even if the cases themselves have not changed, the same set of cases still needs to be re-verified. Of course, this is a very suitable place to use automated testing to cover the existing cases, so that testers can devote more of their energy to exploratory testing.

Is the Main Value of Testers Producing Test Cases?

From the description above, we can also see that, in the testing phase, a more important responsibility of testers is to complete exploratory testing. Test cases themselves are more of an embodiment of testing thinking and a form of expressing testing ideas; they are only an auxiliary output, not the core of testing.

For testing, as I have mentioned many times in my previous articles, James Bach has a famous paper discussing the difference between testing and checking. Executing according to cases is merely checking and cannot count as testing. Real testing, beyond checking, also includes many abilities such as experimentation, research, exploration, questioning, observation, and reasoning—these are the true abilities that help us uncover deeper product defects and expose product quality problems.

So the value of testers is of course not limited to producing cases. It lies much more in revealing the unknown and in assessing the quality status of a product that is still a black box to the team.


Built with Hugo
Theme Stack designed by Jimmy