Ettore Di Giacinto

@mudler_it

dad, creator of LocalAI(localai.io) and Kairos (kairos.io) , ex @SUSE/@Rancher, ex-Gentoo Dev. vllm.cpp and APEX quants

Italy

github.com/mudler

Replies Reposts Media

Pinned
Ettore Di Giacinto
@mudler_it
Aug 13
a week ago I shared what I've been working on, vllm.cpp, vLLM's serving stack in C++ with no python in it. I got really a lot of feedback, and it seems I'm not the only one who wanted this to exist. We reached 800+ commits, 280 stars in just so little time. but where is it
9
Ettore Di Giacinto
@mudler_it
Sep 2
You won't need python to run high speed concurrency LLMs on your machines, and you will be able to embed it everywhere. vllm.cpp is getting better day by day. what's currently moving, and what not: - EXL3 quants works natively. On the DGX Spark 16.7 tok/s is our base decode,
8
Ettore Di Giacinto
@mudler_it
Sep 1
AMD Strix just joined the fleet! So here it is, what the harnesses use to pick up a box and when needed. I had to build something to manage the boxes, so agents can use the boxes freely during sessions (if you are interested in the tool, link in the replies). I've called it
2
Ettore Di Giacinto
@mudler_it
Sep 1
vllm.cpp is now getting native exl3 support.
Mia
@MiaAI_lab
Aug 31
Rejoice RTX 3090/4090/5090 owners💫 Qwen3.8-27B EXL3 is here. This is an *experimental* release so there might be some bugs along the way! As far as I know, this is the only way to serve Qwen3.8-27B with 200k+ context while using dflash2 on a 24GB VRAM, without sacrificing
Ettore Di Giacinto
@mudler_it
Aug 30
Hey @github . Can you provide some clarity on your ToS? It's so _silly_ that in the AI era we can't have separate accounts for operating with agents. This is so dumb that I can't take more of it. I don't complain about your uptime status, but this is very nonsense. I got
4