The Art of AI Evaluation

The original comes from the blog of Jeff Nyman, discussing the shift in thinking and the evaluation principles we should apply to AI testing. Translated here to share.

Beyond the Model Itself: The Art of AI Evaluation

When we talk about “AI testing”, most people first think of the large language model itself. They focus on the “engine” and overlook the “vehicle”. Yet in the context of customized enterprise applications, what we test is not just a model but a complete system.

This is crucial! More and more enterprises rely on embedding AI into their applications. But usually they are not building a simple architecture that just lets users “chat” with a bot. Instead they are building complex architectures in which AI is only one component of a large machine. Specifically, enterprises are leveraging AI through patterns that are already very clear:

  • Retrieval-augmented generation (RAG): Think of it as AI’s “library”. The model no longer merely guesses from training data; before it “speaks” it is forced to look up specific company documents (the “source of truth”).
  • Agentic workflows: Here AI plays the role of “project manager”. It is given a goal and allowed to use tools (such as querying a SQL database, searching the web or calling an API) to complete a multi-step task.
  • Structured output pipelines: In this scenario AI acts as a “data translator”. It takes messy, unstructured human text and turns it into clean, executable code or JSON that other parts of the business software can understand.

In traditional software testing we tend to look for “decisiveness”: “if I click X, Y must happen”. In AI testing we deal with “randomness”, meaning probability-based results. Since AI may describe the same fact in three different ways, our tests cannot rely solely on finding an exact text string. We must test its intent, accuracy and logic.

This is where the concept of the “evaluator” comes in. If the LLM is a witness giving testimony, then the evaluator is the forensic expert who checks that testimony against the physical evidence. We are not testing whether the AI is “smart”; we are testing whether the system is faithful to the data and relevant to the user’s needs.

The Distinction: Potential and Reality

It is very helpful to separate the general capability of a model from its concrete application.

  • Model (potential): A raw LLM is a vast library of statistical probabilities. Testing it in isolation only tells you what it can do in a vacuum.
  • Application (reality): Your enterprise application is the “implementation layer”. It contains your prompts, your RAG pipeline and your business logic.

If I had to use an analogy, I would say testing a raw LLM is like testing the quality of a chef’s knife; testing an AI application is like tasting the final beef bourguignon. A sharp knife is necessary (or at least very helpful!), but it does not guarantee the stew will not be too salty.

A Broader Evaluation Scope

Evaluating these AI-infused applications requires us to focus on the “three pillars” of output quality in that context:

  1. Answer relevancy: Does the reply actually address the user’s intent, or is it just a “hallucination” that sounds right but is in fact irrelevant?
  2. Faithfulness: Is the answer strictly derived from the provided context (the “source of truth”), or did the model draw on external, possibly outdated training data?
  3. Context precision: How well does the system perform when it first retrieves the correct information?

“LLM-as-a-Judge”

To handle these problems at scale we can use dedicated tools such as DeepEval or RAGAS. These tools adopt a meta-cognitive approach usually called “LLM-as-a-Judge”. Frankly, I am not sure that is a good name. I prefer to think of it as “forensic examination” of a conversation.

What I mean is that we use a secondary, usually more powerful model to act as an objective evaluator. It looks at the “evidence” (the retrieved context) and the “verdict” (the generated answer) to determine whether the logic holds up. This lets us turn qualitative human intuition into quantitative, reproducible metrics such as context recall and answer correctness.

Setting the Stage

I have a lot more to say on this topic, but here I want to “show” rather than merely “tell”. What I can say is that we are moving from a world of “testing by feel” (where we just ask the AI a few questions and see whether we like the answers) towards a world with rigorous evaluators. You may not be testing the LLM directly, but you are testing the governance of a given model’s power within a specific domain.

I think that, for test experts working with companies that have chosen to make AI part of their operations (whether customer-facing or not), this largely constitutes their ethical mission.

As for my own leaning, I am a Thomist. In Thomist terms, the LLM is the “material cause” (raw potential), while your application is the “formal cause” that gives it a specific shape and purpose. Testing only the LLM is like studying the properties of marble when you should be assessing the structural integrity of the cathedral built from it.

So, to set up what is coming, this series will introduce the concepts and tools that help us “examine the cathedral”. It requires the reader to dig in a little. I will use Python mostly (or entirely), and show testers how to actually use these tools in combination with these broader concepts.

I should point out that I consider this a completely empty area of modern testing. I see many people voicing concern about AI (perhaps legitimately), but very few actually learning how to use these tools to improve their market competitiveness in this future that has already arrived.

I have talked before about the ethical mission of the “error expert”. As everyone knows, AI makes mistakes. In fact, Claude tells you so right at the bottom of any prompt you submit.

ChatGPT does the same.

That is great! Self-awareness is a good thing. But it leaves an open question: how do we “double-check replies” and “verify important information” at scale? For testers who want to stay ahead, this is a superb patch of ground.

So our ethical mission has not been removed or replaced; it has actually expanded. As “error experts”, our area of focus grows with it.

https://testerstories.com/2026/01/ai-and-testing-ollama-and-models/

Built with Hugo
Theme Stack designed by Jimmy