Track allocated buffers in rpc-server

[rgerganov]

Currently the rpc-server doesn't perform any input validation and this may have security implications. Another problem is that it may create memory leaks if clients do not free allocated buffers before disconnect (PR #7378 tries to address this).

I think we can address both issues by using std::unordered_set to track allocated buffers and perform some additional checks without any noticeable performance degradation.

[chraac]

Have some other thought maybe off topic:
Should we make a dedicated session liked structure for each client to organize the resource allocatd by connection? we can also hold ggml_backend_t there.

[rgerganov]

Should we make a dedicated session liked structure for each client to organize the resource allocatd by connection?

Right now the rpc-server can serve only one client at a time (I should add this to the README). I prefer to keep it that way because the code is simple as we don't have to deal with multiple threads, synchronization, etc. Users can still run multiple instances using the same backend and overcommit backend memory if this is what they want.