speculative: Ensure draft and target model vocab matches #3812
Conversation
examples/speculative/speculative.cpp
Outdated
|
|
||
| { | ||
| int n_vocab_tgt = llama_n_vocab(model_tgt); | ||
| if (n_vocab_tgt != llama_n_vocab(model_dft)) { |
There was a problem hiding this comment.
This is not ideal. Codellama 7B and 13B have vocab size of 32000 while Codellama 34B has vocab size of 32016. It's the same vocab but with some extra tokens.
We should not disallow such cases. Maybe just print errors / warnings, but still continue?
There was a problem hiding this comment.
How about we error if the size differs by more than 100, and also check the content of min(n_vocab_tgt, n_vocab_dft) tokens? Maybe even start 5 or something, to allow for cases where something like BOS or EOS has different content.
100 and 5 are just random numbers I plucked out of the air. The actual values can be whatever you prefer.
There was a problem hiding this comment.
I changed it. Also made the token content mismatch message a bit more helpful. For example, trying to use Orca 3B to draft Mistral 7B:
main: error: draft model vocab must match target model to use speculation but token 259 content differs - target ' ', draft ' t'
2 participants
It's currently possible to shoot yourself in the foot by trying to speculate using a draft model with vocab that doesn't match the target, and weird stuff will happen in that case. Naturally the draft model will fail 100% of the time, but looking at the logs it'll appear that the draft is just generating random unrelated stuff (even draft candidates with NaN as the probability).
When there's a mismatch you'll now get an error like:
or
This approach may be too strict, since one might possibly want to use a draft model with a few special tokens that differ. One way to deal with that might be to just say there can be X mismatches at most.
strcmpon every entry might also be overkill. On my system, checking 32,000 entries is instant but doing something like looping with a step of 10 would probably be fine also.