fabricatio_rag.capabilities.chunk

Semantic chunking capability using LLM-guided split-point indices.

Splits each input text into mini-chunks of roughly rag_config.mini_chunk_size characters, asks the LLM to emit the starting mini-chunk index of each output chunk, then merges mini-chunks by those indices. Supports both single-string and batch (list of strings) input.

Classes

PreciseChunkText

LLM-guided text chunker.

Module Contents

class fabricatio_rag.capabilities.chunk.PreciseChunkText(/, **data: Any)

Bases: fabricatio_core.capabilities.usages.UseLLM

LLM-guided text chunker.

Splits texts into semantically coherent chunks by combining a deterministic mini-chunker (Rust split_into_chunks) with an LLM that emits the starting mini-chunk index of each output chunk.

async precise_chunk(chunk_guideline: str, text: str, max_size: int = 5, min_size: int = 2, mini_chunk_size: int | None = None) → List[str]
async precise_chunk(chunk_guideline: str, text: List[str], max_size: int = 5, min_size: int = 2, mini_chunk_size: int | None = None) → List[List[str]]
async precise_chunk(chunk_guideline: str, text: List[str] | str, max_size: int = 5, min_size: int = 2, mini_chunk_size: int | None = None) → List[List[str]] | List[str]

Split text into semantically coherent chunks using LLM-guided split points.

Parameters:

  • chunk_guideline – Natural-language instruction for how to chunk.

  • text – Single text or list of texts to chunk.

  • max_size – Maximum mini-chunks per output chunk.

  • min_size – Minimum mini-chunks per output chunk.

  • mini_chunk_size – Character size of mini-chunks (defaults to rag_config.mini_chunk_size).

Returns:

list of chunk strings. For a list of str: list of chunk-lists (one per input text).

Return type:

For a single str