fabricatio_rag.capabilities.clean

Corpus cleanup capability for pre-embedding hygiene.

Provides deterministic, composable cleaning steps that operate on a corpus (strings or document-like objects) before it is chunked, embedded, and stored.

Pipeline order (each step is opt-in via its own guard in the config):

raw -> normalize -> strip_markup -> mask_pii -> lang_filter

-> split -> dedup -> length_filter -> cleaned

LLM-based enrichment (summaries, hypothetical questions, etc.) is intentionally NOT part of this class — it belongs in a separate, opt-in capability.

Classes

CleanText

Clean text using LLM-guided hashline edits.

Module Contents

class fabricatio_rag.capabilities.clean.CleanText

Bases: fabricatio_diff.capabilities.hashline_edit.HashlineEdit

Clean text using LLM-guided hashline edits.

Each call to clean() delegates to HashlineEdit.hashline_diff(), which runs a self-correcting LLM loop: the LLM emits hashline-anchored edits, they are applied via Rust primitives, and the result is judged against the guideline. If the judge rejects it, the loop retries with the error context.

async clean(clean_guideline: str, text: str) → str

async clean(clean_guideline: str, text: List[str]) → List[str]

async clean(clean_guideline: str, text: str | List[str]) → str | List[str]

Clean text(s) until they satisfy the given guideline.

Parameters:

• clean_guideline – Natural-language description of the target clean state (e.g. "Remove all HTML tags").

• text – Single string or list of strings to clean.

Returns:

the cleaned string. For a list of str: a list of cleaned strings, preserving order.

Return type:

For a single str

Raises:

HashlineEditExhaustedError – If the LLM cannot satisfy the guideline within the configured iteration limit.