Eval bug: It is not possible to catch LLM output anymore

[andrew-aladev]

Name and Version

Any llama.cpp version starting from b7348.

Operating systems

Linux

GGML backends

CPU

Hardware

any

Models

any

Problem description & steps to reproduce

I can't catch LLM output using llama-cli anymore, because there is no way to disable interactive mode for llama-cli anymore. I can use llama-completion, but it is a legacy binary, and I don't want to use it because in future releases it may be removed.

So I want to ask: how to catch the LLM output after you have merged #17824? Thank you.

First Bad Commit

#17824 @ngxson

Relevant log output

█ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄                                                         
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██                                                                                                                                                                  
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀                                                                                                                                                                  
                                    ██    ██                                                                                                                                                                     
                                    ▀▀    ▀▀                                                                                                                                                                     
                                                                                                                                                                                                                 
build      : b1062-34ce48d9                                                                                                                                                                                      
model      : unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF:Q8_K_XL                                                                                                                                                    
modalities : text                                                                                                                                                                                                
using custom system prompt                                                                                                                                                                                       
                                                                                                                                                                                                                 
available commands:                                                                                                                                                                                              
  /exit or Ctrl+C     stop or exit                                                                                                                                                                               
  /regen              regenerate the last response                                                                                                                                                               
  /clear              clear the chat history                                                                                                                                                                     
  /read               add a text file

[bandoti]

Hi @andrew-aladev are you trying to write the output to a file?

Please provide an explicit example on how you would invoke llama-cli previously. There are many way to invoke this tool and lots of technical debt built up, which is driving the need to consolidate (hence the new llama-cli).

The intention is not to remove llama-completions so I think you're safe to keep using that.

But it is important for us to understand how you're intending to use this exactly in order for us to establish general user needs. Thank you!

[andrew-aladev]

Hello @bandoti, I am using the complex command using llama-cli, but you can just use:

llama-cli --hf-repo 'unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF:Q8_K_XL' --prompt 'Test' > output.txt

The intention is not to remove llama-completions so I think you're safe to keep using that.

Sorry, but I think you are not right here.

[+] Add messages to warn user to move to llama-completion for legacy features

@ngxson already marked this binary as legacy. So I am sure it will be removed in the future. I am sure nobody wants to build a system on top of a legacy tool.

But it is important for us to understand how you're intending to use this exactly in order for us to establish general user needs.

I am sure the user needs a way to capture LLM output. Why do you think that this super basic feature is not required for the general user?

Please refer to the following code in the pull request:

// TODO: maybe support it later?
if (params.conversation_mode == COMMON_CONVERSATION_MODE_DISABLED) {
  console::error("--no-conversation is not supported by llama-cli\n");
  console::error("please use llama-completion instead\n");
}

How is it possible to accept pull requests that break super basic functionality? I just kind understand, because it is so obvious to click the 'reject' button in that case.

[ngxson]

If you have time to write these comments, you should have had time to write a PR to fix it too. There is no need to be passive-aggressive towards contributors.

There is nowhere mentioning about removing llama-completion. If you find where a maintainer say it, please quote and give a link.

[andrew-aladev]

Ok, for now I spend my time setting the llama.cpp version to b7348. Maybe I will implement this todo on the next week or on the weekend. Thank you.

[andrew-aladev]

@ngxson, Sorry, Please don't keep my thoughts too close to the heart.

You have posted a discussion and referenced it inside the code itself (also I saw this reference in the CLI output itself).

I've reviewed the changes you described, and there was no criminal in it. Please remember the changes you mentioned:

• Multimodal support
• Fully jinja support (the old llama-cli fails in certain cases)
• Conversation control (regenerate and roll back messages)
• Speculative decoding
• etc

But after your pull request, the core functionality became broken. And I am sure that this functionality is critical for everyone who uses llama.cpp. So, it means you haven't followed your described guidelines, and now your pull request is incomplete because it is missing core functionality.

@bandoti Sorry for interrupting you, I am going to implement the described todo in the next week. Can you please revert commits from #17824 now? I am sure @ngxson will approve that his PR breaks core functionality. I am going to add my commit on the top of @ngxson commits, then test core functionality and after that we will be able to prepare new PR. Thank you.

[bandoti]

@andrew-aladev No problem it's just a misunderstanding.

It is still not clear to me what the high level goal is though. The command provided to me says:

• Running a model with a basic prompt, writing the output to a file.
• Reasoning model is conversational usually

So how is the input handled when no --single-turn specified, for example?

Before implementing anything it is important to establish the high level goals otherwise we can spend hours implementing the wrong problem.

Please add more details and I can get back in a couple hours as I have some other engagements at the moment!

[andrew-aladev]

@bandoti, Well, it is very easy. We can archive the raw output without interactive input by replacing while (true) { ... } in main with do { ... } while (false) for now. The output is the following:

./bin/llama-cli --hf-repo 'unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL' --prompt 'Test'
common_download_file_single_online: using cached file: /home/puchuu/.cache/llama.cpp/unsloth_Qwen3-Coder-30B-A3B-Instruct-GGUF_Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf

Loading model...  

build      : b7359-c6f6e4f9
model      : unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read               add a text file


> Test

Hello! This is a test response. 

I'm functioning properly and ready to help with whatever you'd like to discuss or work on. What would you like to do?
[ Prompt: 20.9 t/s | Generation: 15.0 t/s ]
Exiting...

Now we can pipe ./bin/llama-cli to other processes like ./bin/llama-cli ... | parser. The only problem is the output now is not parsable, because it doesn't include <think></think> tags. But it can also be easily fixed by applying something from the grammars folder.

So from my view implementing this todo will be as follows:

1. Replace while (true) { ... } with do { ... } while (flag)
2. Copy is_interacting logic from completion (that includes proper isatty checks for WIN32 and UNIX)
3. Apply some default grammar (like json or other) when is_interacting is false.
4. Add llama-completion in .devops folder so users can use it inside docker container.

If you are fine about it that I will be able to implement it on the next week, thank you.

[andrew-aladev]

Hello @CISC, Sorry for interrupting. I've just received an email notification about we need to include llama-completion in Docker images. Please review my full plan that is required (from my point of view) for finishing #17824. Thank you!

[bandoti]

Piping output is a possibility of course but the thing is the tags are expected to be stripped by the template. I don't think we want to apply any default grammars.

I recommend using llama-completions for now while we figure out next steps. The application is not going to be deleted, and if it has a feature you rely on then it's best to keep using it.

We don't want to stuff all the old llama-cli features into the new application so we must deliberate on the big picture before adding capabilities.

I am going to try and collect user journeys and I'll make note of the piping capability there.