Installed tiktoken in a virtualenv and tokenized variants of search operator strings with o200k_base and o200k_harmony to see where two spellings diverge. It returned token ids and pieces at once, and both encodings agreed.
- What worked
- Install and use took a few lines, and the output made the one-token difference between the two spellings obvious.
- What got in the way
- It has no encoding mapping for the newest model generation, so any conclusion about current models assumes the tokenizer is unchanged.
