Chunking and provenance¶

`ClauseAwareChunker` ¶

Bases: ChunkingStrategy

Clause-aware chunking strategy that tries to keep legal clauses intact.

The text is cut into sections at heading lines (see SECTION_PATTERNS); with no headings it is cut at blank lines. Consecutive sections are packed into chunks of at most max_chunk_size tokens (count_tokens). A section too large on its own is split at sentence boundaries, and a sentence too large on its own at word boundaries.

Guarantees:

every chunk is an exact substring of the input, so text.find(chunk) recovers its character offset;
no chunk exceeds max_chunk_size tokens;
chunks never start or end mid-word (except a single word longer than max_chunk_size, which is cut);
a heading stays in the same chunk as the start of its body;
with overlap > 0, each chunk after the first begins with up to overlap tokens from the end of the previous chunk, starting at a sentence (or failing that, word) boundary, when that still fits.

`init(max_chunk_size: int = 4000, overlap: int = 200, preserve_sentences: bool = True)` ¶

Initialize clause-aware chunker.

Parameters:

Name	Type	Description	Default
`max_chunk_size`	`int`	Maximum size of each chunk in tokens	`4000`
`overlap`	`int`	Number of tokens to overlap between chunks	`200`
`preserve_sentences`	`bool`	Split oversized sections at sentence boundaries (otherwise at word boundaries only)	`True`

`chunk(text: str) -> list[str]` ¶

Split text into chunks while preserving clause boundaries.

Parameters:

Name	Type	Description	Default
`text`	`str`	Full document text	required

Returns:

Type	Description
`list[str]`	List of text chunks, each a substring of text

`SemanticChunker` ¶

Bases: ChunkingStrategy

Semantic chunking strategy that splits on natural language boundaries.

Splits text into chunks based on paragraphs and sentences while maintaining semantic coherence.

`init(max_chunk_size: int = 4000, overlap: int = 200, split_on: str = 'paragraph')` ¶

Initialize semantic chunker.

Parameters:

Name	Type	Description	Default
`max_chunk_size`	`int`	Maximum size of each chunk in tokens	`4000`
`overlap`	`int`	Number of tokens to overlap between chunks	`200`
`split_on`	`str`	Primary split unit ('paragraph' or 'sentence')	`'paragraph'`

`chunk(text: str) -> list[str]` ¶

Split text into semantic chunks.

Parameters:

Name	Type	Description	Default
`text`	`str`	Full document text	required

Returns:

Type	Description
`list[str]`	List of text chunks

`ChunkingStrategy` ¶

Bases: ABC

Abstract base class for document chunking strategies.

`chunk(text: str) -> list[str]` `abstractmethod` ¶

Split text into chunks.

Parameters:

Name	Type	Description	Default
`text`	`str`	Full document text to chunk	required

Returns:

Type	Description
`list[str]`	List of text chunks

Raises:

Type	Description
`ChunkingError`	If chunking fails

`count_tokens(text: str) -> int` ¶

Estimate token count for text.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to count tokens for	required

Returns:

Type	Description
`int`	Approximate token count

`ProvenanceTracker` ¶

Registers text chunks and resolves extracted values back to source spans.

Resolution strategy¶

Exact substring — O(n) scan per chunk; preferred.
Jaccard token overlap — bag-of-words similarity for paraphrased or truncated values; only returns a match when the score exceeds similarity_threshold (default 0.85) to avoid false attributions.

The tracker is stateful: every call to register_chunks() appends to the internal chunk list and advances the global character offset. Call clear() to reset between documents.

Parameters:

Name	Type	Description	Default
`source_url`	`str \| None`	Default source URL attached to registered chunks.	`None`
`similarity_threshold`	`float`	Jaccard threshold for the fallback matcher.	`_SIM_THRESHOLD_DEFAULT`

`chunks: list[ChunkRecord]` `property` ¶

Read-only view of registered chunks.

`register_chunks(chunks: list[str], source_url: str | None = None, page_map: dict[int, int] | None = None, source_text: str | None = None) -> list[ChunkRecord]` ¶

Register an ordered list of text chunks.

Parameters:

Name	Type	Description	Default
`chunks`	`list[str]`	Text chunks (e.g. output of `ClauseAwareChunker`).	required
`source_url`	`str \| None`	Override the tracker-level source URL for this batch.	`None`
`page_map`	`dict[int, int] \| None`	Optional mapping of chunk index → 1-based page number.	`None`
`source_text`	`str \| None`	The document text the chunks were cut from. When given, each chunk is located in it (in order; overlapping chunks are fine), so `char_start` / `char_end` and every resolved `SourceSpan` are true offsets into source_text. Raises `ValueError` if a chunk is not a substring of it. Without it, offsets assume the chunks tile the document separated by one character each, which is not true of any chunker that trims, joins or overlaps.	`None`

Returns:

Type	Description
`list[ChunkRecord]`	The list of `ChunkRecord` objects created for these chunks.

`register_chunk(text: str, source_url: str | None = None, page: int | None = None) -> ChunkRecord` ¶

Register a single chunk. Convenience wrapper around register_chunks.

`find_span(extracted_text: str) -> SourceSpan | None` ¶

Resolve extracted_text to a SourceSpan.

Returns None if no chunk matches above the similarity threshold.

Pass 1 — exact substring search (fast, preferred). Pass 2 — Jaccard token overlap (fallback for paraphrased values).

`annotate(document: LegalDocument, field_name: str, extracted_text: str) -> bool` ¶

Resolve extracted_text to a span and attach it to document.

Sets document.provenance[field_name] if a match is found.

Returns:

Type	Description
`bool`	True if provenance was attached, False if no match was found.

`annotate_all(document: LegalDocument) -> dict[str, bool]` ¶

Annotate every field in document.extracted_fields.

Values must be strings (or string-coercible).

Returns:

Type	Description
`dict[str, bool]`	Dict mapping field_name → whether provenance was found.

`coverage(document: LegalDocument) -> dict[str, float]` ¶

Provenance coverage statistics for document.

Returns:

Type	Description
`dict[str, float]`	`{"fields_with_provenance": int, "total_fields": int, "coverage_ratio": float}`

`get_chunk(chunk_id: str) -> ChunkRecord | None` ¶

Retrieve a chunk by its ID. Returns None if not found.

`clear() -> None` ¶

Reset tracker state (call between documents).

`ChunkRecord` `dataclass` ¶

Metadata about a text chunk registered with the tracker.

Attributes:

Name	Type	Description
`chunk_id`	`str`	Stable, deterministic identifier (index + content hash).
`text`	`str`	Raw chunk text.
`source_url`	`str \| None`	Canonical URL of the source document.
`page`	`int \| None`	1-based page number this chunk belongs to, if known.
`char_start`	`int`	Character offset of this chunk's first char in the full doc.
`char_end`	`int`	Character offset after this chunk's last char.
`content_hash`	`str`	SHA-256 of `text`.

Chunking and provenance¶

ClauseAwareChunker ¶

__init__(max_chunk_size: int = 4000, overlap: int = 200, preserve_sentences: bool = True) ¶

chunk(text: str) -> list[str] ¶

SemanticChunker ¶

__init__(max_chunk_size: int = 4000, overlap: int = 200, split_on: str = 'paragraph') ¶

chunk(text: str) -> list[str] ¶

ChunkingStrategy ¶

chunk(text: str) -> list[str] abstractmethod ¶

count_tokens(text: str) -> int ¶

ProvenanceTracker ¶

Resolution strategy¶

chunks: list[ChunkRecord] property ¶

register_chunks(chunks: list[str], source_url: str | None = None, page_map: dict[int, int] | None = None, source_text: str | None = None) -> list[ChunkRecord] ¶

register_chunk(text: str, source_url: str | None = None, page: int | None = None) -> ChunkRecord ¶

find_span(extracted_text: str) -> SourceSpan | None ¶

annotate(document: LegalDocument, field_name: str, extracted_text: str) -> bool ¶

annotate_all(document: LegalDocument) -> dict[str, bool] ¶

coverage(document: LegalDocument) -> dict[str, float] ¶

get_chunk(chunk_id: str) -> ChunkRecord | None ¶

clear() -> None ¶

ChunkRecord dataclass ¶

`ClauseAwareChunker` ¶

`init(max_chunk_size: int = 4000, overlap: int = 200, preserve_sentences: bool = True)` ¶

`chunk(text: str) -> list[str]` ¶

`SemanticChunker` ¶

`init(max_chunk_size: int = 4000, overlap: int = 200, split_on: str = 'paragraph')` ¶

`chunk(text: str) -> list[str]` ¶

`ChunkingStrategy` ¶

`chunk(text: str) -> list[str]` `abstractmethod` ¶

`count_tokens(text: str) -> int` ¶

`ProvenanceTracker` ¶

`chunks: list[ChunkRecord]` `property` ¶

`register_chunks(chunks: list[str], source_url: str | None = None, page_map: dict[int, int] | None = None, source_text: str | None = None) -> list[ChunkRecord]` ¶

`register_chunk(text: str, source_url: str | None = None, page: int | None = None) -> ChunkRecord` ¶

`find_span(extracted_text: str) -> SourceSpan | None` ¶

`annotate(document: LegalDocument, field_name: str, extracted_text: str) -> bool` ¶

`annotate_all(document: LegalDocument) -> dict[str, bool]` ¶

`coverage(document: LegalDocument) -> dict[str, float]` ¶

`get_chunk(chunk_id: str) -> ChunkRecord | None` ¶

`clear() -> None` ¶

`ChunkRecord` `dataclass` ¶