Skip to content

Evaluation

See the evaluation guide.

EvalSuite

Bases: BaseModel

A named collection of EvalCases.

from_yaml(path: str | Path) -> EvalSuite classmethod

Load an EvalSuite from a YAML file.

from_json(path: str | Path) -> EvalSuite classmethod

Load an EvalSuite from a JSON file.

load(path: str | Path) -> EvalSuite classmethod

Auto-detect format from file extension (.yml/.yaml or .json).

from_cases(cases: list[EvalCase], name: str = 'inline') -> EvalSuite classmethod

Build a suite directly from a list of EvalCase objects.

filter_by_tag(tag: str) -> EvalSuite

Return a new EvalSuite containing only cases with tag.

filter_by_doc_type(doc_type: str) -> EvalSuite

Return a new EvalSuite containing only cases matching doc_type.

EvalCase

Bases: BaseModel

A single labeled eval case.

get_weight(field_name: str) -> float

Return the importance weight for field_name (default 1.0).

has_input() -> bool

Return True if at least one input source is set.

EvalHarness

Run eval suites and compute extraction quality metrics.

Parameters:

Name Type Description Default
extractor_fn ExtractorFn

Callable that accepts an EvalCase and returns a dict of extracted values. May raise; errors are caught and counted as error_cases unless fail_fast=True.

 required
fail_fast bool

Stop at the first extractor error instead of continuing.

 False

Example::

harness = EvalHarness(my_extractor, fail_fast=False)
metrics = harness.run(suite)
assert metrics.field_accuracy >= 0.90

run(suite: EvalSuite) -> ExtractionMetrics

Run all cases in suite and return aggregated metrics.

Parameters:

Name Type Description Default
suite EvalSuite

The EvalSuite to evaluate.

 required

Returns:

Type Description
ExtractionMetrics

ExtractionMetrics with per-field and aggregate stats plus the
ExtractionMetrics

full list of CaseResult objects for inspection.

run_case(case: EvalCase) -> CaseResult

Run a single eval case (useful for debugging a specific failure).

run_privacy(suite: EvalSuite, *, pii_detector_fn: Callable[[EvalCase], list[str]] | None = None, redactor_fn: Callable[[EvalCase], int] | None = None, router_fn: Callable[[EvalCase], bool] | None = None) -> PrivacyMetrics

Run privacy / redaction evaluation across suite.

Parameters:

Name Type Description Default
suite EvalSuite

The EvalSuite to evaluate.

 required
pii_detector_fn Callable[[EvalCase], list[str]] | None

Callable (EvalCase) → list[str] — returns the entity types detected (e.g. ["PERSON", "EMAIL_ADDRESS"]). Required for cases with expected_pii_entities.

 None
redactor_fn Callable[[EvalCase], int] | None

Callable (EvalCase) → int — returns the number of redaction spans applied. Required for cases with expected_redaction_count.

 None
router_fn Callable[[EvalCase], bool] | None

Callable (EvalCase) → bool — returns True when the privacy router blocks the LLM call. Raising PrivacyBlockedError also counts as blocked. Blocking is scored only when a router_fn is given.

 None

Any other exception raised by one of these callables is recorded in the case's error and scored as a failure for that metric.

Returns:

Type Description
PrivacyMetrics

PrivacyMetrics with precision/recall/F1 for PII detection and
PrivacyMetrics

blocking accuracy.

ExtractionMetrics

Bases: BaseModel

Aggregate quality metrics across an entire EvalSuite run.

case_accuracy: float property

Fraction of cases where all fields matched (no partial credit).

field_accuracy: float property

Weighted fraction of individual fields that matched.

report() -> str

Return a human-readable summary table.

assert_min_field_accuracy(threshold: float) -> None

Assert that weighted field accuracy is at or above threshold.

Designed for use in pytest::

metrics.assert_min_field_accuracy(0.90)

Raises:

Type Description
AssertionError

With a descriptive message including the report.

assert_min_case_accuracy(threshold: float) -> None

Assert that case accuracy is at or above threshold.

PrivacyMetrics

Bases: BaseModel

Aggregate privacy / redaction metrics across an EvalSuite run.

from_case_results(results: list[PrivacyCaseResult]) -> PrivacyMetrics classmethod

Aggregate a list of PrivacyCaseResult into summary metrics.

report() -> str

Return a human-readable privacy metrics summary.

assert_min_pii_recall(threshold: float) -> None

Assert that PII recall is at or above threshold.

assert_perfect_blocking() -> None

Assert that all blocking decisions were correct.