Microsoft has introduced a new open-source framework aimed at one of the most stubborn problems in modern AI development: how do you reliably test an AI system’s behavior, not just its outputs, and do it in a way that stays useful as models and prompts evolve?
The company’s latest release, Adaptive Spec-driven Scoring for Evaluation and Regression Testing, is designed to let developers spin up AI behavior evaluations using text descriptions. In other words, instead of building every test case and scoring rubric by hand—often across multiple scripts, spreadsheets, and brittle prompt templates—teams can describe what “good” looks like in plain language and use that description to drive evaluation and regression testing.
At first glance, this may sound like another tool in the growing ecosystem of AI testing. But the emphasis here is on adaptive, spec-driven scoring—an approach that tries to make evaluations more maintainable and less dependent on hard-coded logic. That matters because AI systems don’t behave like traditional software components. A small change in a model version, a retrieval index, a system prompt, or even the formatting of context can shift behavior in subtle ways. Those shifts are exactly what regression testing is meant to catch, yet most AI teams still struggle to implement regression testing that is both scalable and meaningful.
What Microsoft is betting on is that the “spec” can be expressed in natural language, and that the evaluation process can adapt to the spec rather than requiring developers to constantly rewrite scoring code. This is a practical direction: many teams already write behavioral requirements in text—acceptance criteria, policy rules, product requirements, safety guidelines, and “how the assistant should respond” documents. The missing link has been turning those documents into repeatable, measurable tests.
A framework built around behavior, not just answers
In traditional testing, you can often define correctness with deterministic rules: a function returns the right value, a unit test passes, a response matches a schema. With AI behavior, correctness is rarely that clean. Even when you can define a target format, the content can vary widely while still being acceptable. Conversely, two responses can look similar but differ in compliance, helpfulness, or safety.
Adaptive Spec-driven Scoring is positioned to address that gap by treating evaluation as a structured process driven by textual specifications. Developers can describe the behavior they want to test—what the model should do, what it should avoid, what constraints matter, and what outcomes indicate success or failure. The framework then uses those specs to score behavior during evaluation runs.
This is important because it reframes evaluation from “did the model produce the exact expected output?” to “did the model exhibit the desired behavior under these conditions?” That shift aligns with how AI products are actually judged in practice: by whether the assistant is useful, safe, accurate enough, and consistent with policy and user intent.
The “adaptive” part is where the promise becomes more than just a convenience feature. If scoring were purely static—always applying the same rigid rubric—then any change in the spec would require retooling. Adaptive scoring suggests the framework can adjust how it interprets and applies the specification, making it easier to keep evaluations aligned with evolving requirements.
Why regression testing is harder for AI than for software
Regression testing is straightforward when the system is deterministic. For AI, regression testing becomes a moving target because the system is probabilistic and context-dependent. Even if you keep the same model, the environment can change: different documents retrieved, different user phrasing, different tool outputs, different system prompts, different temperature settings, and so on.
Teams often respond by creating a set of golden test prompts and comparing outputs. But golden tests have two major weaknesses. First, they can be overly sensitive to harmless variation, causing noisy failures. Second, they can miss meaningful regressions when the output changes in ways that still pass superficial checks.
Behavioral evaluation aims to solve both issues. Instead of comparing raw text, it scores whether the response meets the spec. That reduces brittleness and makes it easier to detect regressions that matter—like a model becoming more verbose, less compliant, more likely to hallucinate, or less able to follow instructions.
Microsoft’s framework explicitly targets evaluation and regression testing in the same workflow. That’s a subtle but significant design choice. Many tools focus on evaluation as a one-time activity during development. Regression testing implies ongoing discipline: you run the same evaluation suite repeatedly as you update models, prompts, retrieval pipelines, or system components. If the evaluation setup is too expensive to maintain, teams stop running it regularly. By making the spec-driven approach central, Microsoft is trying to lower the cost of keeping regression tests current.
Text descriptions as a bridge between product requirements and engineering reality
One of the most compelling aspects of this release is the idea that developers can “spin up” AI behavior tests using text descriptions. That phrase matters because it suggests a workflow where the barrier to entry is lower. Instead of requiring deep expertise in evaluation engineering—writing custom scoring functions, designing complex judge prompts, calibrating thresholds, and maintaining evaluation harnesses—teams can start with the behavioral requirements they already have.
In many organizations, those requirements live in documents. They might be written by product managers, safety teams, compliance officers, or UX researchers. Translating them into executable tests is often a bottleneck. A spec-driven framework can act as a translation layer: it takes the intent behind the requirements and turns it into something that can be executed repeatedly.
This also creates a feedback loop. If evaluation results show consistent failures, teams can refine the spec. Over time, the spec becomes a living artifact that captures what the system is supposed to do. That’s a different mindset than treating evaluation as a static checklist.
Of course, there’s a risk in any spec-driven approach: natural language can be ambiguous. If the spec is vague, the scoring will be vague. But that’s not unique to this framework; it’s a general challenge in AI evaluation. The difference is that the framework encourages teams to formalize their behavioral expectations in a structured way, even if the structure is expressed through text.
Open source and the push toward standardization
Microsoft is releasing this as an open-source framework. That matters for two reasons.
First, open source increases the chance that teams can adopt the approach without being locked into a proprietary evaluation pipeline. AI evaluation is notoriously fragmented across vendors and internal tooling. Standardization—at least at the level of evaluation harness patterns—can reduce duplicated effort.
Second, open source invites community scrutiny. Evaluation frameworks are only as good as their assumptions. When a tool is open, others can inspect how scoring works, how specs are interpreted, what failure modes exist, and how robust the system is across different types of tasks.
In the broader AI ecosystem, we’ve seen a pattern: evaluation tooling evolves quickly, and teams often end up rebuilding similar harnesses multiple times. An open-source framework can become a shared foundation, allowing teams to focus on their domain-specific specs and test cases rather than reinventing the evaluation engine.
How spec-driven scoring could change day-to-day AI development
To understand why this release is notable, it helps to imagine how it fits into a typical AI engineering workflow.
A team builds an assistant that answers questions, summarizes documents, or performs actions via tools. Early on, they test manually. Then they add automated checks: unit tests for tool calls, schema validation for structured outputs, and a handful of prompt-based tests. As the assistant matures, they need deeper evaluation: does it follow instructions? does it refuse unsafe requests? does it cite sources correctly? does it maintain tone and formatting? does it handle edge cases?
At that stage, evaluation becomes a craft. Teams create “judge” prompts or scoring scripts. They tune rubrics. They calibrate thresholds. They maintain datasets of test prompts and expected behaviors. And when they update the system prompt or swap the model, they revisit everything.
Adaptive Spec-driven Scoring aims to streamline that last mile. If the scoring logic is driven by text specs, then updating behavior requirements may require fewer code changes. Instead of rewriting scoring scripts, teams can update the spec and rerun the evaluation suite. That can make regression testing more feasible as changes accelerate.
This is especially relevant now because AI systems are updated frequently. Many teams deploy improvements weekly—or even daily—because user feedback and model iteration cycles are fast. Without a robust regression testing workflow, teams either ship cautiously (slowing progress) or ship aggressively (risking silent regressions). A spec-driven evaluation framework can help teams move faster while maintaining confidence.
A unique take: treating evaluation as a living contract
There’s a philosophical angle to this release that goes beyond tooling. By centering evaluation on text descriptions, Microsoft is effectively encouraging teams to treat evaluation specs as a contract between stakeholders and the model.
Product teams can articulate what the assistant should do. Engineering teams can encode those expectations into specs. Safety and compliance teams can refine the boundaries. Then the evaluation suite becomes a continuous check that the model still honors that contract after changes.
This is a shift from “we tested it once” to “we continuously verify behavior.” It also makes evaluation more transparent. When evaluation fails, the spec provides context for why it failed. That can shorten debugging cycles because teams can see whether the issue is a misunderstanding of requirements, a weakness in the model, or a mismatch between the spec and the actual task.
Of course, transparency doesn’t automatically guarantee correctness. AI evaluation can be tricky because the evaluator itself may be an AI model. If the scoring mechanism relies on another model to judge behavior, then evaluator bias and inconsistency become concerns. The framework’s adaptive scoring approach may help, but it doesn’t eliminate the fundamental challenge: you need evaluation that correlates with human judgment.
Still, the direction is promising. Many teams already use AI-based judges because human evaluation at scale is expensive. The key is to make those judges consistent and aligned with the spec. A spec-driven framework can help align scoring with the intended criteria, rather than relying on ad hoc judge prompts.
Where this could be most valuable
Spec-driven behavior evaluation is likely to be most impactful in