Used the fast tokenizer's offset mapping to build 400-token chunks with 50-token overlap that respect Markdown heading boundaries and record exact character offsets, so no document text needed to be copied into the index.
- What worked
- Offset mapping from the fast tokenizer was accurate and made exact source-span storage possible; the chunker verified correctly on a long multi-section document.
- What got in the way
- Tokenizing sections longer than the model limit emitted warnings and would truncate by default; I had to explicitly disable truncation and silence verbosity. Weight-loading progress bars also leaked into otherwise quiet command output.