New File System Implementation

[ethanalee]

Objective: Write a high-performance multi-threaded file system layer that replaces the current implementation in library_fs.js.

Goals:

• Faster file system operations
• Multi-threaded without relying on proxying
• Smaller code size footprint
• Per-file persistence
• Support multiple async and sync backends
• Support for large files
• POSIX compatibility via syscalls

Design Sketch Link

Click here to view/comment

[kripken]

Thanks @ethanalee !

I'd add

• Full multithreading support (no constant proxying to the main thread). edit: optional, that is only in multithreaded builds

(no need to actually edit the top comment).

[hcldan]

I'd like to help however I can in moving this forward.

[kripken]

@hcldan Help would certainly be appreciated here!

For now, it's still early for writing code, but we've been reading library_fs.js and related files in order to understand the current capabilities, so that we know what not to regress. If you or others want to help later, reading that code now might end up useful.

Another issue is testing. Reviewing the current tests and seeing if there is any lack of coverage would be good, though that might not be easy to do. One possible way to combine this with the previous point is to find interesting/important areas in library_fs.js and then to see if intentionally breaking them (adding a throw, or returning a wrong value, etc.) causes a test to break - if no test breaks, that's missing coverage. (For doing that, testing ./tests/runner.py wasm0 other should be enough.)

[curiousdannii]

Emscripten's FS shouldn't require multithreading though.

[kripken]

@curiousdannii

Yes, definitely!

The singlethreaded case should continue to behave as it has been. Perhaps it will be smaller, but that's about it.

The multithreaded case should get a lot faster (by avoiding constant proxying to the main thread all the time).

[hcldan]

I've been looking at library_idbfs.js as it shows a way to do async storage operations... does anyone have any experience with this that I could chat with? I'd like to understand better what's going on here.

It looks like FS is a MEMFS that lives in the main browser thread, and we rely on these syncfs calls to keep the data current in the memfs as it's stored in the indexeddb. But that just seems to only take care of persistence, and not actually offload the storage to disk and out of memory... am I correct here?

Right now the system calls in emscripten proxy to the main ui thread (with pthreads)... that means these calls are already async... Where are they async? Can we push that async interface down to the FS layer... and then perhaps shim in the same async adaptation that's being done for proxying in the case where it's a single threaded app?

I started all this because the SF apis dont have sync counterparts in the main browser thread, and I can plug in the async SF apis inside the library_idbfs.js and put data in FS and metadata in idbfs... but that won't help us if all the file data stays in memory.

sidebar: also looking for help understanding this module architecture... any good place to start looking?
idbfs impl looks entirely different in surface area than the SFfs module... how are things hooked up? It's very confusing.

[hcldan]

I think I found some of the answers to my question here: https://github.com/emscripten-core/emscripten/blob/main/system/lib/pthread/library_pthread.c#L485

[hcldan]

would always proxying fs calls deadlock a single threaded program? I don't know how this wait works.
I think if we could turn the fs proxy always on, and make FS naturally async, it would give us more options. perhaps if the proxy layer for fs were moved into the fs module itself.

I'm having a hard time grokking the pthread code that looks at __proxy. it's a lot of code writing code.
I can't tell if it's general purpose or custom built to only work with syscalls

[hcldan]

As far as requirements go:

• Do not keep FS data in memory, delegate all IO to mounted filesystems. (eliminate syncfs)
  • FS metadata, might be fine, as I think SF doesn't have a good way to implement file locking, readonly/readwrite, and opening a file more than once.

[hcldan]

I actually think it might be a nice IDBFS enhancement to leave all file metadata in IDBFS and put file content in SF (if available). But we would need to eliminate syncfs and the MEMFS copy of FS data and that would mean needing an async FS layer.

[kripken]

Right now the system calls in emscripten proxy to the main ui thread (with pthreads)... that means these calls are already async...

I think that's not accurate. The filesystem operations are proxied synchronously to the main thread. We wait until they finish, so that they complete (write to memory, etc.), and also we need the return value from them.

Any proxying is not good, but synchronous is especially bad, and proxying to main thread is also very bad. The new version should avoid all that in most cases. Most of the code and metadata (directory structure) could be in wasm, and so it's accessible from all threads. Actual file data might be present in JS on a particular thread, and proxied to, but we can cache those in memory as well.