llama : auto download HF models if URL provided

[ggerganov]

QoL improvement - pass HF URL to auto-download the model if it does not exist:

./main \
  -m https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf \
  -p "Hello world"

Will attempt to download via wget or curl if they are available in your PATH

[slaren]

I could see this in a shell script, but this seems too hacky to include in llama.cpp.

Additionally, this may cause security issues. Consider this:

./main \
  -m "https://huggingface.co/.../mixtral-8x7b-instruct-v0.1.Q2_K.gguf;rm -fr /;" \
  -p "Hello world"

Applications that want to load a model based on a user request would need to be careful to sanitize the input.

[ggerganov]

Yes, probably not a good idea to merge like this

[cebtenzzre]

Yeah, system and popen should not be used with arbitrary user input. Better to use fork/execlp or posix_spawn (Unix) and _spawnlp or CreateProcess (Windows). This is somewhat tedious to do safely and portably in C or C++ without third-party dependencies such as glib.

[ggerganov]

Yes, I dropped the idea. Will probably add a bash script that can be used something like this:

./main -m $(./examples/hf.sh --model-name TheBloke/Mixtral-8x7B-v0.1-GGUF --quant Q4_K_M) ...

and it will do all the curl and wget calls

[staviq]

server includes httplib.h which does support client mode, so this should be doable without invoking any system commands

Usage seems easy enough (from httplib project): https://github.com/yhirose/cpp-httplib#client

[ggerganov]

The hf.sh script idea should be better