
When AI Writes Applications, a Blessing or a Challenge for Quality?
As AI models become increasingly capable, AI programming tools represented by Copilot, Cursor, Claude Code, and Gemini Cli are gradually entering everyday software development, and “vibe coding” is increasingly becoming a trend in application development.
So when AI becomes more and more involved in the development of real applications, is the quality of applications built in this way good or bad?
This article shares my views based on my recent experience using several vibe coding tools.
AI Programming Looks Beautiful
From the moment people first encounter AI programming, many are amazed by its extremely fast and efficient power. Combined with good prompt engineering, it can often complete prototype design in a very short time, and the resulting product is frequently more feature-complete than expected.
As long as we have some understanding of software application development, combined with prompt engineering, AI can help us with everything from design to code, from the front end and back end to the database. What engineers need to do seems to be continuously communicating with the AI and constantly optimizing and adjusting the output.
It can be said that the emergence of AI programming tools has enabled many non-programmers to develop a wide range of creative applications, greatly lowering the barrier to software development.
In major software development companies, it is gradually becoming the norm for developers to write more and more code with the help of AI tools.
The future of AI programming is so close, and so beautiful~
However, if the goal is not merely to validate a prototype, and we judge an AI-developed application by the standards of a production-grade application, then a complex and painful debugging process will inevitably be part of the picture.
“Ten minutes of development, two whole days of debugging” is actually the norm when it comes to AI programming.
When the AI’s Imagination Leaves You Speechless
Although “prompt engineering” summarizes and emphasizes some of the best ways to collaborate with AI, in real work, engineers often still tend to communicate with the AI using brief instructions.
When such instructions are insufficient and imprecise, the AI will naturally imagine a great deal of code that it considers necessary but that is actually plausible yet wrong. At this point, engineers inevitably need to spend a large amount of time checking and adjusting the AI’s behavior.
Anyone who has coded with AI has likely had the experience of spending a huge amount of time on some basic logic or front-end interface adjustment.
For example, I once encountered a case where I simply wanted to add a field to a page form and adjust the back-end database accordingly. Yet the AI always misread the intent — either it didn’t sync the change, or it modified other unrelated forms. The back-and-forth dragged on and consumed a lot of time.
What should have been a two-line change turned into a situation where, because I wasn’t familiar with the structure of the AI-generated code, I relied on the AI to revise it over and over. Although the desired result was finally achieved, a large amount of needless spaghetti code was inserted along the way.
Weaknesses in Integration and Decoupling
When AI generates code, it generally first determines the project’s code structure based on the initial requirements, and then fills in the code according to specific decomposed tasks.
But as the project progresses and requirements are continuously refined, the main logic usually ends up concentrated in a few main files, which easily makes some of those files overly bloated. And because AI relies on large models, which usually have context window limitations, bloated files greatly constrain the efficiency and quality of code generation.
Therefore, as requirements are continuously detailed and adjusted, the code becomes increasingly bloated and increasingly chaotic. It rarely undergoes proper decoupling and refactoring as it would in real engineering practice, and the AI’s own understanding of the code often becomes more and more difficult, making it hard to recapture that initial sense of amazement.
At this point, engineer intervention is essential; otherwise, AI coding becomes more like trying to carve flowers on a dunghill.
What You See Is Not Necessarily What You Get
In order to present results quickly, AI-generated code sometimes “cheats” by hard-coding the presentation of certain data effects.
In the early stage, some mock data may be indispensable to demonstrate results. But you cannot fully trust the AI to distinguish hard-coding from properly handled logic.
In an AI-assisted tool I once used, I pointed out to it that after I entered a certain specific piece of data, the actual result didn’t match expectations. After pretending to analyze things for a long time, the AI told me it had found the cause. But when I actually checked the code, I discovered that its handling of this part was simply an if check targeting that specific piece of data, forcing a special case and calling it a day.
Challenges to Quality
So, from the explanations (and complaints) above, although AI has achieved astonishing results in assisting coding, and there is no lack of cases where it cooperates well with engineers, many applications generated with AI assistance still cannot dispel concerns about questionable quality.
Perhaps AI itself can help you generate unit tests, but just as intelligent driving still cannot replace human decision-making, in the field of quality, the role of human test engineers remains irreplaceable. Indeed, as AI applications become more widespread, the importance of manual testing becomes ever more prominent.
- The quality of the code itself
- Overall planning of complex applications and broad-context understanding
- Deficiencies in scalability, performance, security, and other aspects
- Mock data and hard-coding
How Test Engineers Should Respond
Given the quality challenges of AI-generated applications, test engineers should also carry out targeted testing and verification of their output.
The following aspects can be considered as priorities:
Systematic Verification Against Traditional Standards
As a product, testing should first and foremost verify the finished AI application from the perspectives of requirements, product goals, and user usage, ensuring that it meets product requirements and quality standards.
Product Integration, Scalability, Performance, and Security
In view of the weak points of AI code, focus on scenarios such as product integration, scalability, performance, and security, and carry out targeted testing.
From Black Box to White Box
The existence of AI illusions requires test engineers not only to verify the application from a black-box perspective, but also to pay attention to the process by which the AI generated the application. In particular, during the development process, the relevant prompts, context, and records of the AI’s responses are all sources for understanding potential quality problems. Many issues, such as hard-coding, may be hidden in some misleading interactions.
Embrace AI and Fight Magic with Magic
Although AI still has many problems, and using it incorrectly can produce products of poor quality, using it properly can indeed provide great help.
With AI, we can analyze product code and sort out the project architecture, helping us better understand the implementation of the project. This includes using AI to assist with many testing tasks, including automated testing.
Conclusion
All in all, a good-quality product cannot do without a mature team and standardized processes. For testers, the AI era is also an opportunity to redefine our role. Embracing the challenges of AI-driven development, testers will also be indispensable participants and collaborators. Embrace AI, and embrace the future!
