Skip to main content

Pipeline Benchmarking

Benchmarking lets a manager compare several pipelines (or AI model variants) on the same test document, based on expected annotations defined ahead of time — no final annotated document is produced, only the quality of the annotations matters.

Setting up a benchmark​

  1. Upload a test document (and any support documents) and define the ground truth: the list of expected annotations (a comment, and optionally an associated note).
  2. Add one or more items to compare: each item is a given pipeline (portfolio), with the option to override the AI model used by a specific node (Agent or Supervisor) without modifying the pipeline itself — this is how you compare, for example, the same pipeline with two different model providers.
  3. Each item runs as a real analysis (same engine, same pipeline executor), categorized separately from normal jobs so it doesn't skew production statistics.

Evaluation​

A "judge" AI model (the organization's default model) compares each expected annotation to the annotations actually produced, by meaning, not by exact text match. Each item's report shows:

  • Tokens consumed (input/output/total) and execution time.
  • Number of expected annotations found, duplicated, or extra (found but matching nothing expected).
  • A precision percentage (expected annotations correctly found / total expected).

These results let you objectively pick, before going to production, the pipeline/model combination with the best trade-off between precision, cost and speed.