Analysis of Workflow Cases in Testing

Real-World Case Analysis of Bug-Handling Disputes in Testing (Part 1)

In the software development process, testing and development are two roles that are both loving and combative with each other: one tries its best to avoid bugs, while the other strives to find bugs. Therefore, disputes and conflicts around the handling of bugs at work are extremely common.

In this series, we will analyze the handling principles when bug disputes arise by looking at some real dispute cases

Case 1 - The Bug Assignment Dispute

Test engineer Xiao Zhao found during testing that refreshing an information bar on the software interface always prompted an error, so he submitted a Bug to front-end developer Xiao Li. But shortly after, Xiao Zhao received a notification that the Bug had been rejected. The reason for the rejection was that Xiao Li believed the error message for this information refresh was returned to him by a back-end module, and that his normal display had no problem. In this situation, should Xiao Zhao accept the Bug rejection?

Analysis

This case is a classic scenario of a bug dispute between testing and development in software testing.

We should first clarify the responsibilities of the two roles with respect to bugs:

  • The responsibility of the testing role is to discover bugs in the product and complete verification after development fixes them.

  • The responsibility of the development role is to analyze the bugs found by testing and complete the fixes.

In this case, then, Xiao Zhao found a problem, judged it to be a bug, and submitted a bug ticket. Because the problem appeared on the front-end interface, he assigned it to the front-end developer Xiao Li. From a process standpoint, there is nothing wrong with this.

The dispute lies in the fact that, after Xiao Li analyzed the Bug, he believed the front end is only responsible for displaying the error message returned by the back end, so the error lies in the back-end module and there is no problem for him to solve, hence he chose to reject it.

From Xiao Li’s perspective, this seems understandable. But from a process standpoint, it is still a problem with how Xiao Li handled it.

Because although the bug analysis was done and it was acknowledged as a Bug, the fix was not completed, so rejection should not have been chosen.

Rejecting a Bug should correspond to only two situations: a false-positive bug report, or a duplicate report of an existing Bug. After Xiao Li’s analysis, for this Bug we cannot consider the analysis to be complete; the status of the bug ticket is actually still in the analysis stage, except that after the front-end analysis, the problem still requires back-end developers to continue the analysis.

So from a process standpoint, the correct action would be for Xiao Li to reassign the bug to back-end development, rather than choosing to reject the bug back to testing.

Actual Handling

Back to Xiao Zhao: in this situation, the bug ticket has been rejected back, so how should this bug ticket be handled?

Because the problem has not been solved, this bug must certainly be resubmitted.

From a normal process standpoint, it is also acceptable to resubmit the problem directly to Xiao Li. But because at this point the front end has already partially completed the analysis and clearly identified the downstream module, when reopening this bug it is best to resubmit the bug directly to the back-end developers, the goal being to avoid pointless repeated communication and improve efficiency.

At the same time, regarding Xiao Li’s process-handling error, an explanatory note should be added when resubmitting the Bug, and the project management team should also be informed, rather than accepting this behavior of direct rejection.

Additional Reflections

Here we have clarified the standard practice from a process standpoint. But should testers first make the assignment correctly in the first place?

In real work, given the complexity of software systems, it is normal for testers to find it difficult to accurately determine the specific module a problem belongs to and the developer actually responsible for analyzing it, and in many cases inaccurate assignment is unavoidable. However, from the perspective of delivering value, as testers we should still submit the issue ticket to the corresponding developer as accurately as possible, in order to improve the efficiency and speed of problem handling and resolution.

Returning to the example above, the back end returns an error message to the front end, and although it is displayed on the front end, if Xiao Zhao had traced the interface return information from the front-end page, it should not have been difficult to spot the anomaly in the interface response, and thereby judge that the problem likely lies in the back end, so as to assign it directly to the back end.

Conclusion

To summarize, after a bug ticket is submitted, if there is a mismatch between the module and the first assigned handler, the handler has an obligation to reassign the Bug to the actual handler, rather than incorrectly rejecting the Bug.

At the same time, on the testing side, testers should also strengthen their understanding of the system and their familiarity with team members, so as to get the first assignment right as much as possible.

In addition, for relatively large development teams and complex products, if it is difficult to determine problem ownership, the development side usually sets up a default analysis engineer role, specifically to perform initial analysis of Bugs and uniformly carry out secondary assignment.

In short, workflow goals prioritize efficiency and avoid any possible waste of extra communication and handling time.


Case Analysis of Bug-Handling Disputes in Testing (Part 2)

In the software development process, disputes between the development and testing roles over bugs are almost an everyday occurrence. In the previous article, we analyzed the handling principles for a bug assignment dispute case.

In this installment, we continue to analyze dispute scenarios and handling principles for bug handling in software development through case studies.

Case 2 - Handling a Bug That Cannot Be Reproduced

When test engineer Xiao Zhao was testing an App, after completing a certain operation, the App crashed. Since screen recording was enabled during testing, Xiao Zhao compiled the screen recording of the crash and the operation steps and submitted a Bug ticket to App developer Xiao Sun. The next day, Xiao Sun rejected the Bug ticket back, on the grounds that he could not reproduce the bug and could not identify a problem from the recording, the operation steps, and code analysis. And Xiao Zheng tested again himself, trying to reproduce the problem, also without success. In this situation, should the Bug rejection be accepted?

Analysis

This case is also a fairly typical Bug dispute scenario.

Many times, due to various factors in testing—such as environmental interference, incorrect data, and even misoperations by the testers themselves—testers do submit some issue tickets that are actually not Bugs. After analysis, these tickets are confirmed not to be problems and are rejected as Not a Bug.

Among the reasons for rejecting an issue ticket, a rather special one is the problem cannot be reproduced, which means the problem symptom cannot be produced again in either the test environment or the development environment, and the issue is therefore judged to be approximately non-existent and rejected.

However, rejecting a bug ticket because “the problem cannot be reproduced” should be a reason used with caution.

Because theoretically speaking, for any problem symptom—excluding false positives—if it truly exists but cannot be reproduced, it usually just means its trigger condition has not been found.

In this case, a crash is a problem symptom with relatively serious impact, and this symptom is clearly evidenced by a screen recording as truly existing, yet neither development nor testing can reproduce the problem. It may simply be that the actual condition triggering the problem has not been discovered (that is, after completing a certain operation, a crash occurs as stated in the case description is not the root cause of the problem.)

There is a very important measure in quality management, RCA (Root Cause Analysis). From a quality-oriented perspective, for relatively clear problem symptoms that cannot be reproduced immediately, one should follow the RCA process to complete a root cause analysis, and then determine the subsequent handling of the Bug.

Actual Handling

For this case, because the problem symptom is clear and relatively serious, if the problem truly exists, its appearance on the user side will inevitably cause fairly adverse quality impact.

So for the handling of this problem, after Xiao Sun’s initial analysis, he actually did not find the root cause of the problem; at this point the correct approach should be the following process:

  1. First, the developer should deeply consider the possible causes of the problem and collect further information.

According to my earlier introduction to the different Bug status transitions (see the previous article), at this point, after his analysis, Xiao Sun can still send the Bug back, but it should not be Reject; rather it should be Need More Info, collecting more information related to the problem for further in-depth analysis, such as:

  • The model of the test device? The operating system version?
  • The device’s storage and memory status?
  • System resource usage when the problem occurs?
  • The network used? How stable is it?
  • The executed operation path; are there any abnormal paths?
  1. Then complete RCA based on the supplemented information, including reproduction paths, walkthroughs of potentially problematic code, and inviting other development experts to join a consultation. If the problem still cannot be reproduced, then because relatively sufficient argumentation and analysis have been done, it can actually be considered that the likelihood of the problem appearing on the user side is not very high, and the priority of the problem is actually quite low. At this point, the issue can be submitted to the project CCB (Change Control Board) to decide the subsequent handling strategy for the problem.

  2. The project CCB makes a decision on the problem

This usually in turn includes several different subsequent handlings:

  • Judge that the problem has limited impact, consider it not a problem, and apply Bug rejection
  • Although the problem is unlikely to appear, from a quality-first perspective we still want to find the cause of the problem. In this case the problem can be deferred. That is, we still consider that a problem exists here, but it will not be resolved in the current version; we continue to observe, and developers continue to analyze, trying to resolve it in a subsequent version. If the cause truly cannot be located over a long period, then consider closing the issue.
  • Analyze workaround strategies for the problem, such as adding higher-level exception protection, or adjusting the version’s functional path to avoid the triggering action. Add relevant protective measures in the version to avoid the problem path

In short, for unreproducible problems with relatively serious impact, they should be taken seriously and not simply dismissed with a rejection.

Additional Reflections

For problems of this kind whose causes are hard to locate, is the final outcome necessarily that they just fizzle out with no resolution?

Let me re-emphasize the point above: A problem that cannot be reproduced does not mean the problem does not exist; it means the real root cause or trigger condition has not been found.

Therefore, when the cause of a problem cannot be found, besides collecting more and fuller information as mentioned above, there is another important aspect that the product and development teams need to value and continuously improve, namely the product’s testability requirements or maintainability requirements.

That is, the product should include necessary and sufficient means for problem tracking, log collection, and fault diagnosis; from the product design and development stages, the synchronous development of relevant diagnostic tools should be considered, so as to avoid the awkward situation in subsequent system testing and even user usage scenarios where problems are difficult to analyze and locate.

Conclusion

Finally, some people may still say that although the correct handling process described above is very reasonable, in real work, in most cases schedules are tight and people are already quite overwhelmed, so it is hard to really do in-depth analysis for this kind of problem, and under such circumstances the process is actually difficult to put into practice.

That is true, but the role of processes and standards is to tell everyone the correct norms of behavior, not necessarily to be executed rigidly; understanding the principles behind the processes and standards is more important.

The focus of this case is actually the final judgment of the impact of this problem. The product’s or project’s expectations for quality will also influence how the problem is handled.

The testing role, from the standpoint of assessing quality status, when the problem symptom is clear, should not simply accept that the problem does not exist


Case Analysis of Bug-Handling Disputes in Testing (Part 3)

In the previous two articles of this series, we respectively analyzed two typical Bug dispute cases in the software development process:

  1. Dispute over the assignee
  2. Dispute over handling an unreproducible Bug

In this article, we continue to bring the handling analysis of another dispute scenario

Case 3 - Bug Determination

When Xiao Zhao was testing an App, he found that during a batch operation, if the volume of test data was large, the system would wait for a relatively long time and no other operations could be performed (more than 5 seconds). Xiao Zhao considered that this wait affected the user experience and submitted a usability bug. But this bug was rejected by back-end developer Xiao Wang, on the grounds that according to the requirements, “the system should support batch data operations, and after receiving a batch data request, it should complete data processing and update the data status one by one”, so the response time would inevitably increase as the volume of test data grows, and therefore a longer response time is a normal phenomenon. How should Xiao Zhao continue to handle this Bug here?

Analysis

First, from the user’s perspective, this problem symptom does indeed affect the user experience and causes confusion for users, so the test engineer would of course treat it as a discovered Bug.

But from the back-end developer’s perspective, the current implementation follows the requirements, and the appearance of this phenomenon is an inevitable result, so it cannot be considered a Bug

Here, thinking from the perspective of software quality, this problem symptom is of course a quality issue, so there is no problem in defining this issue as a Bug.

But the back-end development did indeed implement it according to the requirements, so we cannot say this Bug is a development problem either. Then the issue is actually very clear: this problem is not a development Bug, but a requirements Bug.

That is, the requirement itself that was proposed, being vague and insufficiently considering scenarios, led to the above quality problem after being implemented as specified.

Actual Handling

So for the handling process of this problem, it is right for Xiao Zhao to raise the Bug from a quality perspective; here Xiao Wang, following the handling principle of our Case 1, should not reject a quality problem that does not belong to him, but should transfer it to the actual handler.

So here the correct action for Xiao Wang should be to reassign the Bug to the business analyst, have the business analyst re-clarify the requirements solution, and then reconsider the code implementation of this part.

Additional Reflections

When a quality problem caused by a requirements issue is discovered at the system testing stage, fixing it often involves adjustments to the implementation solution, and in many cases the changes are substantial and may well lead to a delay in delivery.

Take this case as an example; possible adjustments to the implementation solution:

  • Limit the volume of data processed in a batch?
  • The back end processes asynchronously, returning the processing status to the front end as soon as processing starts, and then pushing a completion notification to the front end once finished.
  • Front end and back end redefine the processing interface, with the page presenting processing-progress feedback?

These implementations may all have a significant impact on the original implementation solution.

Therefore, although the problem is categorized as a requirements Bug, discovering a requirements Bug at the system testing stage is actually something that should be avoided as much as possible. For this kind of problem, the best stage to raise it is the early requirements analysis stage or solution design stage, and this is precisely the significance of shift-left testing and of testers fully participating in early requirements discussions and solution reviews.

Conclusion

Not all Bugs discovered during the software testing stage are the developers’ Bugs; problems caused by inadequate consideration of requirements, unclear requirements, and similar reasons are also not uncommon.

But for Bugs caused by requirements, discovering and raising them only at the testing stage often means higher fix costs and greater harm to the product, so we should fully understand the requirements and clarify potential problems as early as possible before testing begins, rather than trying to mend the fold after the sheep are lost at the testing stage.


Built with Hugo
Theme Stack designed by Jimmy