Skip to main content

Pipeline Benchmarking

Benchmarking lets a manager compare several pipelines (or AI model variants) on the same test document, based on expected annotations defined ahead of time — no final annotated document is produced, only the quality of the annotations matters.

Only annotation pipelines can be compared here: a chat pipeline or a JSON extraction portfolio produces no annotations and is not offered.

Setting up a benchmark​

  1. Upload a test document (and any support documents) and define the ground truth: the list of expected annotations (a comment, and optionally an associated note).
  2. Add one or more items to compare: each item is a given pipeline (portfolio), with the option to override the AI model used by a specific node (Agent or Supervisor) without modifying the pipeline itself — this is how you compare, for example, the same pipeline with two different model providers.
  3. Each item runs as a real analysis (same engine, same pipeline executor), categorized separately from normal jobs so it doesn't skew production statistics.

Evaluation​

A "judge" AI model (the organization's default model) compares each expected annotation to the annotations actually produced, by meaning, not by exact text match. At creation, the Evaluator field lets you replace it with a decision model: each found annotation is linked to an expected one with a probability (a match counts from 50%), which is faster, cheaper and more stable from run to run. The same choice exists for RAG evaluation, where the decision model judges each answer's faithfulness and correctness. Each item's report shows:

  • Tokens consumed (input/output/total) and execution time.
  • Number of expected annotations found, duplicated, or extra (found but matching nothing expected).
  • A precision percentage (expected annotations correctly found / total expected).

These results let you objectively pick, before going to production, the pipeline/model combination with the best trade-off between precision, cost and speed.