Server: completion_probabilities (tok_str and prob) seem to be broken

[reuank]

Hello,

I am using the llama.cpp server and noticed strange behavior in the server responses.

When starting a server on commit 637e9a8 using ./server -m ../models/llama-2-7b-chat.Q4_K_M.gguf -c 4096 -ngl 1000 -np 1 -cb, and using this curl command:

curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Choose between A, B and C.\n\n","n_predict": 1, "n_probs": 10, "temperature": 0}'

I get the following json response:

// commit hash 637e9a86
{
    "content": "A",
    "id_slot": 0,
    "stop": true,
    "model": "../models/llama-2-7b-chat.Q4_K_M.gguf",
    "tokens_predicted": 1,
    "tokens_evaluated": 12,
    "generation_settings":
    {
        ...
    },
    "prompt": "Choose between A, B and C.\n\n",
    "truncated": false,
    "stopped_eos": false,
    "stopped_word": false,
    "stopped_limit": true,
    "stopping_word": "",
    "tokens_cached": 12,
    "timings":
    {
        "prompt_n": 12,
        "prompt_ms": 280.894,
        "prompt_per_token_ms": 23.407833333333333,
        "prompt_per_second": 42.720741632074734,
        "predicted_n": 1,
        "predicted_ms": 1.734,
        "predicted_per_token_ms": 1.734,
        "predicted_per_second": 576.7012687427913
    },
    "completion_probabilities":
    [
        {
            "content": "A",
            "probs":
            [
                {
                    "tok_str": "A",
                    "prob": 0.6929230093955994
                },
                {
                    "tok_str": "Option",
                    "prob": 0.04242830350995064
                },
                {
                    "tok_str": "Wh",
                    "prob": 0.035371895879507065
                },
                {
                    "tok_str": "What",
                    "prob": 0.021582460030913353
                },
                {
                    "tok_str": "The",
                    "prob": 0.020988475531339645
                },
                {
                    "tok_str": "Your",
                    "prob": 0.009944385848939419
                },
                {
                    "tok_str": "In",
                    "prob": 0.007504411973059177
                },
                {
                    "tok_str": "You",
                    "prob": 0.0066000730730593204
                },
                {
                    "tok_str": "Question",
                    "prob": 0.006469167303293943
                },
                {
                    "tok_str": "If",
                    "prob": 0.006083796266466379
                }
            ]
        }
    ]
}

But when running the same command on the latest commit on master (f89fe27), I get

// commit hash f89fe273
{
    "content": "A",
    "id_slot": 0,
    "stop": true,
    "model": "../models/llama-2-7b-chat.Q4_K_M.gguf",
    "tokens_predicted": 1,
    "tokens_evaluated": 12,
    "generation_settings":
    {
        ...
    },
    "prompt": "Choose between A, B and C.\n\n",
    "truncated": false,
    "stopped_eos": false,
    "stopped_word": false,
    "stopped_limit": true,
    "stopping_word": "",
    "tokens_cached": 12,
    "timings":
    {
        "prompt_n": 12,
        "prompt_ms": 298.66,
        "prompt_per_token_ms": 24.888333333333335,
        "prompt_per_second": 40.17946829170294,
        "predicted_n": 1,
        "predicted_ms": 0.021,
        "predicted_per_token_ms": 0.021,
        "predicted_per_second": 47619.04761904762
    },
    "completion_probabilities":
    [
        {
            "content": "A",
            "probs":
            [
                {
                    "tok_str": "▅",
                    "prob": 1.0
                },
                {
                    "tok_str": "<s>",
                    "prob": 0.0
                },
                {
                    "tok_str": "</s>",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0000",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0001",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0002",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0003",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0004",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0005",
                    "prob": 0.0
                },
                {
                    "tok_str": "\u0006",
                    "prob": 0.0
                }
            ]
        }
    ]
}

The returned probs are strange, and the tokens seem to be the first n tokens of the tokenizer vocabulary.

What happened here?

Best
Leon

[ggerganov]

When temperature == 0.0f we don't compute probabilities. You can set temperature < 0.0f and it should work as expected

[reuank]

Thank you, @ggerganov. I just saw that the change happened in af0a5b6.

It is, however, a bit strange that the tokens for temperature == 0.0f are the first n tokens of the vocab, with the first token getting a prob of 1.0. Maybe it would be more intuitive to return an error, or to leave the probs in the completion_probabilities empty in this case?

[ggerganov]

Yes, it's confusing. Should improve this - PRs welcome

[JohannesGaessler]

Sorry, I must have made a mistake while testing.

It is, however, a bit strange that the tokens for temperature == 0.0f are the first n tokens of the vocab, with the first token getting a prob of 1.0. Maybe it would be more intuitive to return an error, or to leave the probs in the completion_probabilities empty in this case?

The intended behavior is that with temperature 0 the top tokens are still being returned but that all tokens other than the top one have 0 probability. Essentially you should be getting the same thing as with a temperature of 0.001.

[reuank]

@JohannesGaessler, I see.
I just opened #7202, but that is obsolete then, right?

[reuank]

I don't understand why the top token should have 100% probability, and the others 0% probability assigned. I mean their appearance and position in the top n tokens is defined by their respective logits. I would expect the server response to reflect the actual model output.

Is the reason for your approach that you save the softmax calculation?