v0.9.0 is live on npm Read the launch story
$ llm-contract v0.9.0
docs/overview
◆ BEHAVIORAL CONTRACT TESTING FOR AI

Stop shipping
broken AI.

Define what your AI must do. Test it against real cases. Catch prompt, model, tool, and RAG regressions before release.

Start testing → View source ↗
6validation stages
53automated tests
4Node versions in CI
0hidden network calls
support-contract.test.ts 
// One contract. A real output. Evidence you can act on.
const result = await evaluate(supportContract, {
  input: { query: 'Can I return this?' },
  context: 'Returns are accepted within 30 days.',
  output: modelResponse,
});

expect(result.passed).toBe(true);
// failures include a code, path, explanation, and evidence
PASS schemagroundingbehavior score 1.00
THE GAP

Valid JSON is not valid behavior.

Ordinary tests can verify deterministic code. AI systems also need explicit checks for meaning, grounding, refusal behavior, and change over time.

schema-only
{ "refundDays": 90 }

Structurally valid. Factually wrong when the supplied policy says 30 days.

× ships the bug
with llm-contract
FACT_CONTRADICTION
expected: 30 days
observed: 90 days

The contract fails with a stable code and the evidence behind it.

✓ blocks the regression
QUICKSTART

One command to a runnable AI test.

Initialize a working suite and agent instructions, then customize the generated requirements for output produced by any model or agent stack.

01

Install and initialize

$ npm install llm-contract && npx llm-contract init
02

Define and evaluate

contract.ts
import { z } from 'zod';
import {
  defineContract, evaluate, zodAdapter,
  mustPreserveFacts, mustNotInvent,
} from 'llm-contract';

const supportContract = defineContract({
  name: 'support-answer',
  schema: zodAdapter(z.object({
    answer: z.string(),
    needsHuman: z.boolean(),
  })),
  invariants: [
    mustPreserveFacts({ threshold: 1 }),
    mustNotInvent({ mode: 'strict' }),
  ],
});

const result = await evaluate(supportContract, {
  input: 'What is the return window?',
  context: 'Returns are accepted within 30 days.',
  output: modelResponse,
});
i
Your model stays yours. `evaluate()` receives output you already generated; `runSuite()` accepts your generation callback. The package does not hide provider calls inside evaluation.
CONTRACT ANATOMY

Requirements that read like intent.

A contract combines normalization, structure, hard invariants, and weighted assertions under one reusable name.

A

Normalize conservatively

Trim whitespace or unwrap code fences without silently rewriting meaning.

normalization
B

Enforce structure

Validate structured output through Zod, Valibot, or the built-in JSON Schema subset.

schema
C

Block hard failures

Invariants fail the contract when a non-negotiable behavior is broken.

invariants[]
D

Score softer signals

Weighted assertions contribute to a score without obscuring deterministic failures.

assertions[]
VALIDATION PIPELINE

Six stages. One inspectable result.

Each stage answers a different question, so a passing parser cannot mask a broken business rule.

  1. 01

    Normalize

    Conservative cleanup only.

    raw → normalized
  2. 02

    Syntactic

    Can the expected representation be parsed?

    PARSE_ERROR
  3. 03

    Structural

    Does the output satisfy the declared shape?

    SCHEMA_VIOLATION
  4. 04

    Semantic

    Are values, enums, ranges, and rules allowed?

    NUMERIC_OUT_OF_BOUNDS
  5. 05

    Grounding

    Are identifiable claims supported by supplied context?

    UNSUPPORTED_CLAIM
  6. 06

    Behavioral

    Did the system clarify, refuse, and cover required topics?

    UNCERTAINTY_VIOLATION
FAILURE TAXONOMY

Debug signals, not vague verdicts.

Every failure can carry a stable code, exact path, human explanation, and supporting evidence for reports and CI.

  • SCHEMA_VIOLATIONwrong shape or field type
  • UNSUPPORTED_CLAIMclaim absent from supplied context
  • FACT_CONTRADICTIONoutput conflicts with context
  • REQUIRED_TOPIC_MISSINGmandatory concept omitted
  • UNEXPECTED_REFUSALbenign request incorrectly refused
  • UNCERTAINTY_VIOLATIONambiguous input answered without clarification
Browse the complete failure taxonomy →
REGRESSION SUITES

Know what changed before users do.

Run a dataset through your generation function, compare it with historical outcomes, and repeat cases to expose nondeterminism.

01cases.jsonreal inputs + context
→
02your modelprompt / agent / RAG
→
03contractsrepeat N times
→
04baseline diffregressions + fixes
BASELINE COMPARISON
  • + new failures
  • + detected fixes
  • + pass-rate change
  • + score delta
STABILITY
  • + every attempt recorded
  • + stability score 0–1
  • + flaky cases surfaced
  • + no retry-until-pass
RELEASE GATES

Make AI quality a merge condition.

Policies translate suite evidence into a deterministic exit code for any CI provider.

ci-check.ts
import {
  runSuite, evaluatePolicy, standardCIPolicy,
} from 'llm-contract';

const suite = await runSuite(
  'support-v2', cases, generate,
  { runsPerCase: 3, concurrency: 4 },
);

const policy = evaluatePolicy(suite, standardCIPolicy);
process.exit(policy.exitCode);
minimum pass rateregression tolerancemaximum flaky ratezero-tolerance codes
CLI & REPORTS

Local feedback. CI artifacts. Same evidence.

$ npx llm-contract run cases.json --contract ./contract.js --runs 3
{ }

JSON

Machine-readable automation and stored baselines.

MD

Markdown

Review-friendly summaries for pull requests.

▤

HTML

Portable reports with readable failure detail.

>_

Terminal

Fast local output with policy status.

!
CLI detail: running without --contract performs a non-empty-output smoke check. Supply a contract module for behavioral validation.
COMPOSABLE BY DESIGN

Bring your schema. Bring your model.

The core stays provider-agnostic. Optional peers unlock familiar validation, while generation remains an ordinary callback.

adapterimportrole
Zodllm-contract/adapters/zodfull Zod validation
Valibotllm-contract/adapters/valibotValibot safe parsing
JSON Schemallm-contract/adapters/json-schemabuilt-in deterministic subset
CustomdefineContract(...)business-specific checks

works around OpenAI · Anthropic · Gemini · Ollama · LangChain · custom agents through your callback

HONEST LIMITS

Transparent checks beat magical claims.

≠

Not a universal truth engine

Grounding checks compare output against context you supply. They do not verify every fact in the world.

≈

Does not remove nondeterminism

Repeated runs measure and expose flakiness; they cannot make probabilistic models deterministic.

⊂

JSON Schema is a subset

The built-in adapter intentionally covers a deterministic subset. Use Zod, Valibot, or custom logic for more.

◎

Deterministic-first, not deterministic-only

Probabilistic judge signals can be added separately without overwriting hard failures.

FAQ

Before you install.

What problem does llm-contract solve?+

It turns expected AI behavior into reusable checks, runs those checks across datasets, compares results with a baseline, measures stability, and produces evidence CI can enforce.

Does it depend on a specific model provider?+

No. Evaluate any output string directly or pass your own generation function to a suite. This keeps credentials, provider SDKs, tracing, and retry behavior under your control.

Can it test RAG answers and AI agents?+

Yes. Contracts can compare claims against supplied retrieval context and check behaviors such as required topics, clarification, refusal, citations, forbidden phrases, or custom business rules.

Is it ready for production workflows?+

Version 0.9.0 includes a typed API, CLI, suite runner, policies, reporters, examples, and CI across Node 18, 20, 22, and 24. As a pre-1.0 release, review version changes before upgrading.

SHIP WITH EVIDENCE

Your next AI change should come with a contract.

Open source. MIT licensed. Install from npm and run your first real case.

Star on GitHub ↗
Copied to clipboard