Loading Llama3 in Electron #212
Description
Issue description
Electron crashes when loadModel finishes loading (beta)
Expected Behavior
After loading a model using this code and trying to create a context, I'd expect there not to be a crash in llama-addon.node. I've tried with and without Metal enabled.
If a crash happens, I'd expect to be an error log from node-llama-cpp, but no error logs show up from node-llama-cpp either.
Actual Behavior
When creating context with llama 3 in Electron v28, electron crashes on runtime with an EXC_CRASH error (and no console logs)
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------
Process: Electron [76706]
Path: /Users/USER/Documents/*/Electron.app/Contents/MacOS/Electron
Identifier: com.github.Electron
Version: 28.2.9 (28.2.9)
Code Type: ARM-64 (Native)
Parent Process: Exited process [76691]
Responsible: Terminal [710]
User ID: 501
Date/Time: 2024-05-03 13:33:12.9471 -0700
OS Version: macOS 14.2.1 (23C71)
Report Version: 12
Anonymous UUID: 9118B97C-7A81-646B-40E0-2AE653F176A4
Sleep/Wake UUID: FE4FA3B9-0313-4913-85C1-72873A23C9D1
Time Awake Since Boot: 630000 seconds
Time Since Wake: 353677 seconds
System Integrity Protection: enabled
Crashed Thread: 0 CrBrowserMain Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGTRAP)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Termination Reason: Namespace SIGNAL, Code 5 Trace/BPT trap: 5
Terminating Process: Electron [76706]
Thread 0 Crashed:: CrBrowserMain Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x185749af8 __kill + 8
1 Electron Framework 0x10b600508 uv_kill + 12
2 Electron Framework 0x11226441c node::OnFatalError(char const*, char const*) + 460292
3 ??? 0x157e4fb74 ???
4 ??? 0x157e4d728 ???
5 ??? 0x157e4d728 ???
6 ??? 0x150091aa8 ???
7 ??? 0x157e4b108 ???
8 ??? 0x157e4adf8 ???
9 Electron Framework 0x10c950d80 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 488
Steps to reproduce
- start default Electron project.
- Download model
- Use this code in
src/main.tsto load a model.
(code below works fine in terminal node.js, which may suggest this is an issue on Electron's side, but I don't know enough to be sure, please let me know if you have ideas on where to start debugging.)
// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and import them here.
async function main() {
let { getLlama, LlamaChatSession, Llama3ChatWrapper } = await import(
'node-llama-cpp'
);
const llama = await getLlama();
try {
console.log('Loading model');
let model = await llama.loadModel({
modelPath: path.join(
app.getPath('sessionData'),
`models`,
'Meta-Llama-3-8B-Instruct-Q4_K_M.gguf'
),
onLoadProgress(loadProgress: number) {
console.log(`Load progress: ${loadProgress * 100}%`);
},
});
const context = await model.createContext(); // crash happens here
const session = new LlamaChatSession({
contextSequence: context.getSequence(),
chatWrapper: new Llama3ChatWrapper(),
});
const a1 = await session.prompt('test ');
console.log(a1);
} catch (e) {
console.log(e);
}
}
main();
- run the application
My Environment
| Dependency | Version |
|---|---|
| Operating System | Sonoma 14.2.1 |
| CPU | Apple M1 Max |
| Node.js version | v18.19 |
| Typescript version | 5.4.2 |
node-llama-cpp version |
3.0.0-beta.17 |
Additional Context
No response
Relevant Features Used
- Metal support
- CUDA support
- Grammar
Are you willing to resolve this issue by submitting a Pull Request?
Yes, I have the time, but I don't know how to start. I would need guidance.
