Agent Tool Eval Harness

$39

Dependency-light evaluation harness for agent tools and MCP servers: golden cases, matchers, a runner with latency metrics, and console + JSON reporters for CI regression gating.

📁 8 files🏷 v1.0.0 (updated 2026-08-02)

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 8 files

✓ LICENSE—
✓ README.md—
✓ examples/run.ts—
✓ package.json—
✓ src/harness.ts—
✓ src/matchers.ts—
✓ src/report.ts—
✓ manifest.json—

📁 File Structure 8 files

agent-eval-harness/
├── LICENSE
├── README.md
├── examples/
│ ├── run.ts
├── package.json
├── src/
│ ├── harness.ts
│ ├── matchers.ts
│ ├── report.ts
├── manifest.json

📖 Documentation Preview README excerpt

Agent Tool Eval Harness

A tiny, dependency-light evaluation harness for agent tools and MCP servers.

Define golden cases (input → expected), run them against your tool, and get a

pass/fail report with latency and optional cost — so you catch regressions before

your users (and their agents) do.

Why eval your tools

Prompt and model changes silently break tool behavior. Without evals you find out

in production. This harness gives you a fast, deterministic feedback loop: a set of

cases you can run in CI on every change.

Features

**Golden cases** — declare `{ name, input, expect }` where `expect` is a value,

a predicate, or a partial-match object.

**Runner** — executes cases against any async `(input) => output` function

(wrap your MCP tools/call, HTTP endpoint, or plain function).

**Metrics** — pass rate, per-case latency (p50/p95), and an optional cost hook.

**Matchers** — exact, `partial` (subset match), and `predicate` (custom fn).

**Reporters** — a console table and a machine-readable JSON summary for CI.

Requirements

Node.js 18+ (uses only the standard library + your tool under test)

Quick start


npm install
npx tsx examples/run.ts     # runs the sample suite against a demo tool



import { runSuite } from './src/harness.js';
import { partial } from './src/matchers.js';

const report = await runSuite('search', mySearchTool, [
  { name: 'finds by keyword', input: { query: 'airflow' }, expect: partial({ total: (n: number) => n > 0 }) },
  { name: 'empty query is safe', input: { query: '' }, expect: (out) => Array.isArray(out.results) },
]);

process.exit(report.failed === 0 ? 0 : 1);


File tree


src/
  harness.ts    # runSuite(): executes cases, collects latency, builds the report
  matchers.ts   # exact / partial / predicate matchers
  report.ts     # console table + JSON summary
examples/
  run.ts        # a runnable sample suite against a demo tool


Using it in CI

Exit non-zero when any case fails; the JSON summary can gate a deploy:

... preview truncated, see full README in product download.

`📄 Code Sample .ts preview`

examples/run.ts/**
 * run.ts — a runnable sample suite against a demo "search" tool.
 *   npx tsx examples/run.ts
 */
import { runSuite } from '../src/harness.js';
import { partial } from '../src/matchers.js';
import { printReport, jsonSummary } from '../src/report.js';

// A stand-in for your real tool: replace with an MCP tools/call or HTTP request.
async function searchTool(input: { query: string }) {
  const catalog = [
    { id: 'a', title: 'Airflow DAG Templates', tags: ['airflow', 'data'] },
    { id: 'b', title: 'RAG Pipeline Framework', tags: ['rag', 'ai'] },
  ];
  const q = input.query.toLowerCase();
  const results = q ? catalog.filter((c) => (c.title + ' ' + c.tags.join(' ')).toLowerCase().includes(q)) : catalog;
  return { query: input.query, total: results.length, results };
}

const report = await runSuite('search', searchTool, [
  {
    name: 'finds by keyword',
    input: { query: 'airflow' },
    expect: partial({ total: (n: number) => n === 1 }),
  },
  {
    name: 'empty query returns all, safely',
    input: { query: '' },
    expect: (out: any) => Array.isArray(out.results) && out.total === 2,
  },
  {
    name: 'no match returns zero',
    input: { query: 'nonexistent-xyz' },
    expect: partial({ total: 0 }),
  },
]);

printReport(report);
console.log('\nJSON summary:\n' + jsonSummary(report));
process.exit(report.failed === 0 ? 0 : 1);

`📅 Changelog`

v1.0.0 — Initial release. Last updated 2026-08-02.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $39
        
        Back to Products