Ten Misconceptions That Make Testers Harm Quality

Is a test engineer's work always helpful to quality improvement? In practice, some misconceptions can instead cause testers' work to have a negative impact on quality.

Many companies call the engineers who perform testing QA (quality assurance), but QA and Tester are actually two different roles. Even so, there is no denying that a test engineer’s primary responsibility is also oriented toward quality — helping the team improve quality by assessing the state of quality.

But even so, is a test engineer’s work necessarily helpful to quality improvement? In real work, some misconceptions can instead make testers’ work have a negative impact on quality.

Here Qiucao has sorted out and summarized ten misconceptions by which testers may harm quality, along with the corresponding correct practices.

1. Testing’s job is to find and report Bugs, but not to confirm them

Misconception: Developers and testers often argue over whether a certain issue should be reported as a Bug. “The requirement doesn’t ask for it,” “this issue doesn’t affect users,” and similar reasons come up all the time. Many testers do stop reporting such issues because of these reasons, and some even check with developers whether they are allowed to file a Bug before reporting it.

This is a misconception that does great harm to quality. The value of testing lies in assessing the state of quality, and the key to assessment is independence. Quality is not about right or wrong, only about good or bad. A behavior not defined in the requirements does not mean it is unwanted; not affecting users does not mean it is a good implementation. Testing stands from the perspective of assessing quality, so pointing out defects or raising suggestions for improvement is part of the job. Developers are responsible for analyzing, confirming, and fixing defects. If developers and testers disagree on the confirmation, the PO, business analysts, or CCB members can arbitrate.

But for testers, reporting every product behavior that they feel is problematic or of poor quality is a professional obligation. Deliberately or inadvertently hiding or covering up defects is the real harm to product quality.

2. Finding a Bug is only the beginning; we should also provide as much context as possible for fixing it

Misconception: Some testers think that once they have found a problem and reported it, their work is done, that the subsequent fixing process has nothing to do with them, and that all they need to do is verify it after the fix

Testing is one link in the entire software product development lifecycle, and it ultimately serves the overall quality of the product. The Bug itself, in fact, has no value for the product, because only a Bug that has been resolved helps product quality. From this perspective, although reporting Bugs is a tester’s main job, for the product the ultimate goal is to resolve the Bug. So as one part of product development, even though fixing Bugs is not directly the tester’s responsibility, we should still provide the necessary support for resolving them.

This part was discussed in detail in a previous article 【】, which you may refer to if interested.

3. Testing is not just about verifying requirements

Misconception: Testing only needs to verify that the parts explicitly defined in the requirements are correct; undefined parts need not be tested

This is somewhat related to the first misconception and is the premise of many developer–tester arguments about bugs. In the earlier article 【】 on misconceptions about testing goals, we also mentioned this point: beyond verifying requirements, testing should cover much more exploratory work. Testing is not merely checking.

Beyond that, here I want to emphasize even more the integrity and systematic nature of the product. Especially in an agile development model, this misconception is easily amplified.

In agile, the output of an agile iteration is the Increment (product increment), which holds true for development work. But testing work is oriented toward the entire product system and cannot cover only the increment. Requirements are incremental too, so of course testing cannot cover only the requirements of the increment. This is why regression testing is a category that must be included in testing, and it is also the fundamental reason test automation is receiving more and more attention. (For analysis and introductions related to agile, see the earlier agile series.)

4. Testing is endless; use good steel on the blade’s edge

Misconception: To find more Bugs, explore and diverge as much as possible, and exploit the clustering effect of Bugs to discover as many Bugs as possible

Still in the article 【】, we also clarified that the goal of testing is not to find more Bugs either. Moreover, testing is in fact endless; the ultimate purpose of testing is to provide a reliable, complete assessment of the quality state of the whole product or system. And whether this quality state is objective and accurate is not necessarily positively correlated with the number of Bugs.

While finding as many Bugs as possible, testers also need to recognize that we are systematically assessing quality, yet testing can never be exhaustive. Therefore we must allocate our limited time scientifically, giving priority to high-priority parts — new features, code changes, and high-risk modules are all parts that should be tested first and given key assurance.

Precision Testing and Risk-Based Testing (RBT) are practices under this way of thinking that put our testing resources and time first into the high-value parts.

5. Testing work does not begin only at the test-submission stage; it starts as early as product requirement discussions

Misconception: Testing should formally begin only after the product is submitted for testing, and Bugs should also be reported only after submission

Today, the above misconception is of course incorrect, because both shift-left testing and various testing theories emphasize the importance of early testing involvement.

But theory is theory; in actual practice, testing usually still really begins only after submission for testing.

On one hand, in the early stage of a product, testing has a low presence and low participation and is easily overlooked;

on the other hand, many testing teams are assessed heavily on the number of Bugs, which further erodes the motivation for early testing involvement and makes testers less willing to help the team expose product problems early.

But from the perspective of product delivery, the later the stage, the higher the cost of fixing a problem. So from a product management standpoint, we should abandon the orientation of assessing by Bug count and guide active early participation by testing. Testing can bring its sensitivity to problems into play, identifying and considering many exceptional scenarios starting from requirement discussions, and helping the team avoid them in advance.

6. In design and architecture discussions, testing should not be invisible

Misconception: Architecture and system design, as well as detailed design, are the development team’s tasks; testers need not understand or participate in the discussions and can simply wait until product development is complete and then test against the requirements

Similar to the previous point, shift-left testing does not merely mean that testers understand requirements better; it also requires testers to participate in design and even coding stages.

We must return to the goal of testing: to assess the quality state of the product as accurately as possible. And quality is necessarily closely tied to the concrete implementation of the product; the more clearly we understand the implementation details, the more targeted our testing can be, including helping the team avoid quality problems in advance during the design and coding stages.

Conversely, if design and code are a complete black box, it is very likely that some key test scenarios will be missed and problems will leak through.

  • Without understanding how load balancing is implemented in a cloud application, verifying only on a single node makes it impossible to catch problems caused by load-balancing failures
  • Without understanding the system’s caching implementation, it is hard to purposefully construct cache-hit scenarios
  • Without understanding the responsive layout implementation of the UI, it is hard to efficiently carry out compatibility testing
  • Without understanding the interface’s validation mechanism, it is hard to construct effective API test data

All of the above show that testing should not stay outside the design stage and treat product design as outside the scope of testing, thereby ignoring participation in this stage.

7. An independent test environment is a prerequisite for adequate testing

Misconception: Test execution only needs to focus on the product itself; testing can be done on an integration environment or even a development environment without affecting test results

It is true that the object of testing is the product itself. But the test environment is actually a key constraint on whether the product can be tested comprehensively and efficiently.

In software testing and exploration, using the product usually involves the superposition of many different operations and interactions. And when discovering product problems, an important prerequisite for judging the problem is that the relevant preconditions and associated factors of the product itself are clear and definite. To ensure this, an independent, interference-free test environment is especially necessary. If a development environment is used for testing, developers are simultaneously debugging and making changes on that environment during testing, which not only produces a large number of invalid issues but is also a great waste of resources for testing, problem localization, and the like.

Beyond that, testing inevitably involves creating various exceptional scenarios, and an independent test environment makes it easier for testers to adjust different test scenarios, such as:

  • Modifying network configurations to create scenarios such as proxy access, internal and external IPs, and weak networks
  • Modifying the system time, triggering scheduled tasks, simulating data for different time periods, and similar scenarios
  • Simulating scenarios such as insufficient resources, full disk space, and massive or huge files
  • Some limiting factors in the product, enabling Debug mode, different authentication settings, and various configuration changes

All these tests are built on the foundation of an independent, controllable test environment.

8. Test automation code is also a program; treat test code the way you treat the product

Misconception: Writing test automation is simply to replace part of manual test execution; once the automation code is written and can run and cover the relevant cases, that is enough

The purpose of test automation is indeed to replace a great deal of manual testing work and achieve efficiency gains. But implementing test automation is not a one-off deal; it is also a long-term process. Moreover, test automation itself is implemented through program code, and where there is a program there will be bugs, so quality assurance and maintenance of the automation scripts and tools themselves is also an important part of a tester’s job.

It is not the case that writing a runnable automation script means we have “done automation.” Automation scripts, like software products, should go through a lifecycle of requirements analysis, design, coding, testing, and subsequent maintenance.

So test automation itself is also a fairly large investment. For the relevant scripts to keep delivering value over the long term and truly achieve the goal of improving execution efficiency, good design and continuous maintenance of the test automation code are indispensable.

9. CI/CD pipelines are also testing’s responsibility

Misconception: CI/CD, as infrastructure for many development teams today, is usually built and maintained by dedicated DevOps engineers; as testers, we only need to write the automation scripts for the testing portion

Admittedly, within the entire CI/CD pipeline, the part directly related to testing is mainly test automation. But again: testing, as a member of the whole development process, cannot exist independently of that process. Especially in agile development, the integrity of the team is emphasized even more, and the team as a whole is responsible for the development process.

Therefore, in the process of establishing, operating, and keeping the CI/CD pipeline flowing, there is no clear notion that testing is only responsible for the test automation stage.

As professionals with a quality perspective, we can also play a greater role in building the CI/CD pipeline:

  • Defining quality gates for the flow between stages in the pipeline, reflecting shift-left testing in the CI/CD pipeline
  • Controlling and identifying code changes, which is highly significant for precision testing in the testing phase
  • Monitoring of development, integration, test, and even online production environments, especially the collection and implementation of many quality-related metrics
  • Visualization, including automated test reports and the status presentation of various data tracking

10. Cherish every opportunity to understand end users’ usage scenarios

Misconception: As a member of the development team, testing need not proactively engage with end users. Interfacing with users is more the job of the sales, after-sales, and customer service teams

Testing work is oriented toward quality, and the ultimate judge of quality is in fact the end user. So many teams emphasize that testing is the voice of the end user, a role that uses and experiences the product from the user’s perspective and discovers problems in advance.

Therefore, in this sense, testing should actually engage with users more, understanding the different scenarios, demands, and even preferences with which users use the product.

So, when there is an opportunity to engage with users, testing should be proactive rather than resistant. As testers, there are quite a few such opportunities; make full use of chances to engage with users to enrich our understanding of the industry the product belongs to and apply it to our testing work.

  • User issue support. When some customer issues fed back to the development team require technical support, testers often need to face customers directly, and this is also a place where we intuitively understand users’ usage scenarios of the product and the problems they face.
  • External testing. Before choosing a product, many users will have acceptance testing for it. These tests are often carried out on the customer’s side, but product testers often provide support or perform testing as the product side, which is field testing. This is also a great window for deeply understanding user scenarios
  • Customer visits. Some companies arrange visits and interviews with end customers in the early or middle stages of product development, and sometimes invite the development team to participate. Although in most cases roles such as system analysts and architects participate more, testing also takes part in such visits, which is an excellent opportunity to deeply understand product application scenarios at an early stage.
  • Industry reports. Besides direct communication with end customers, paying attention to industry reports, news, and trend analyses for the industry the product belongs to is likewise an excellent channel for deeply understanding the industry and users the product serves.

The above is a summary of the ten misconceptions by which testers harm quality; everyone is welcome to discuss and add to them. In addition, for Qiucao’s course on systematically improving testing skills, you can reply with “Outline” or check the course catalog in the official account menu.

Built with Hugo
Theme Stack designed by Jimmy