Intelligent Text Chunking
There is a real implementation at src/utils/intelligent_chunker.py. Use it; don't roll your own splitter.
API
from src.utils.intelligent_chunker import IntelligentTextChunker, ChunkType, TextChunk
chunker = IntelligentTextChunker(
max_chunk_size=512, # tokens/characters per chunk
min_chunk_size=50,
overlap_ratio=0.1, # 0.0–0.5
preserve_sentences=True,
preserve_paragraphs=True,
)
chunks: list[TextChunk] = chunker.chunk(text)
# Each TextChunk has: content, start_position, end_position, chunk_id,
# chunk_type (ChunkType enum), metadata, overlap_with_previous, overlap_with_next
ChunkType enum: SEMANTIC, STRUCTURAL, FIXED_SIZE, SLIDING_WINDOW (intelligent_chunker.py).
The chunker has built-in awareness of:
- Multi-language sentence boundaries (English, Chuukese, generic CJK punctuation).
- Structure markers (markdown headings, list items, dictionary entries, page breaks).
- Semantic transition phrases (topic change, continuation, conclusion, examples).
Where it's used
Critical: scripture references aren't protected
The chunker does not know about Bible references — it will happily split 1 Cor. 13:4-7 across chunks. Always wrap calls with protect_scripture_references / restore_scripture_references from src/utils/scripture_parser.py when input may contain them. See the scripture-reference-parsing skill.
from src.utils.scripture_parser import (
protect_scripture_references, restore_scripture_references
)
protected, refs = protect_scripture_references(raw)
chunks = chunker.chunk(protected)
chunks = [
TextChunk(content=restore_scripture_references(c.content, refs), **c_meta)
for c in chunks
]
Choosing parameters
| Use case |
max_chunk_size |
overlap_ratio |
| Training pairs |
256–512 |
0.0–0.05 |
| RAG / retrieval |
512–1024 |
0.10–0.15 |
| Summarization input |
1024–2048 |
0.05 |
Chuukese text is denser per-character than English — when budgeting tokens for an LLM downstream, count tokens, not characters.
Pitfalls
max_chunk_size is in characters by default (the dataclass __len__ returns len(content)). If you need token counts, post-process with the model's tokenizer.- Setting
preserve_paragraphs=True plus a small max_chunk_size will produce oversized chunks rather than break a paragraph — verify chunk lengths if you have hard caps.
- The chunker emits
chunk_ids that are not deterministic across runs (they include positional info). Don't use them as DB primary keys.
- Don't subclass — the public API is small. If you need different behavior, configure the constructor or post-process the chunk list.
