panic: unexpected error during closedir "Bad file descriptor" for unix::fs::Dir

View all comments

At Zed we see 2-5 panics per week across our userbase from dropping a Dir. It is possible that this issue is macOS specific (as all of our telemetry is from macOS today), but we are unfortunately not able to reproduce this panic internally.

The problem for us is that this assertion sometimes fails with unexpected error during closedir: Os { code: 9, kind: Uncategorized, message: "Bad file descriptor" }.

I expect that the standard library should not panic on situations that do happen in reality. In particular if you are trying to close a bad file descriptor, that seems "fine". For comparison a failure to close a file during the drop of a file descriptor is silently ignored as I would expect.

Not panicking is obviously not ideal; but in the context of a long-running interactive app like a text editor, we want to be able to take the trade off of higher uptime over lower visibility.

For more discussion of possible solutions to fallible drops, #98338 has some good ideas, and I think if any are implemented we should do an equivalent thing for directories too (I particularly like the idea of a manual close that returns a result, and which causes the Drop to become a no-op). In the meantime, I think we should make rust programs crash less by ignoring this failure.

Meta

rustc --version --verbose:

rustc 1.77.2 (25ef9e3d8 2024-04-09)
binary: rustc
commit-hash: 25ef9e3d85d934b27d9dada2f9dd52b1dc63bb04
commit-date: 2024-04-09
host: aarch64-apple-darwin
release: 1.77.2
LLVM version: 17.0.6

(Though this was happening in earlier versions of rust too)

Ignore errors in Drop for unix::fs::Dir

@ConradIrwin

@ConradIrwin Can you discuss the tracebacks for this, or does your telemetry lack sufficiently sophisticated crash reporting to make that possible?

I expect that the standard library should not panic on situations that do happen in reality.

I know you meant something slightly aside from the reading of this, but I do expect the standard library to panic on indexing a slice with its length, or greater than. And people write code like that every day. So clearly the criteria has to be something more like

whether it is reasonable to proceed anyways (arguable?)
whether it is fixable (arguably not, if we get an error as joyously "descriptive" as EBADF)

Fair: I do expect panics on programmer error or in a situation where it's unsafe to continue. I am relatively (but it's obviously hard to be 100%) sure this is not programmer error, and I'm convinced that it'd be just fine if the program kept running (c.f. how this error is ignored when dropping a file descriptor).

The full back-trace is below. All line numbers in Zed code are relative to v0.131.6 and in rust 1.77.1. I'm not sure why the top line number is missed by symbolicate, but the panic message narrows it down to this assertion

I believe the reason this panics at on line 4018 of worktree.rs in Zed's code is that child_paths goes out of scope, which was retaining a ReadDir returned from std::fs::read_dir() (via smol) here. I can't see any shenanigans on this codepath that would lead us to to invalidate the file descriptor, which leads me to believe that "this just happens sometimes" when interacting with the filesystem. (Though we are holding this across async/await, so it seems possible that something else is affecting the system's state, but it's very unclear as to what would cause this kind of a problem).

Panic `unexpected error during closedir: Os { code: 9, kind: Uncategorized, message: "Bad file descriptor" }` on thread 38 (com.apple.root.user-initiated-qos)
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/std/src/sys/pal/unix/fs.rs:0:	_$LT$std..sys..pal..unix..fs..Dir$u20$as$u20$core..ops..drop..Drop$GT$::drop::h84b244fb776f88f3
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/ptr/mod.rs:507:	core::ptr::drop_in_place::<std::path::PathBuf>
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/ptr/mod.rs:507:	core::ptr::drop_in_place::<std::sys::pal::unix::fs::InnerReadDir>
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/alloc/src/sync.rs:1756:	<alloc::sync::Arc<std::sys::pal::unix::fs::InnerReadDir>>::drop_slow
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/mem/mod.rs:394:	core::mem::size_of_val_raw::<dyn futures_core::stream::Stream<Item = core::result::Result<std::path::PathBuf, anyhow::Error>> + core::marker::Send>
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/alloc/layout.rs:201:	<core::alloc::layout::Layout>::for_value_raw::<dyn futures_core::stream::Stream<Item = core::result::Result<std::path::PathBuf, anyhow::Error>> + core::marker::Send>
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/alloc/src/boxed.rs:1241:	<alloc::boxed::Box<dyn futures_core::stream::Stream<Item = core::result::Result<std::path::PathBuf, anyhow::Error>> + core::marker::Send> as core::ops::drop::Drop>::drop
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/ptr/mod.rs:507:	core::ptr::drop_in_place::<alloc::boxed::Box<dyn futures_core::stream::Stream<Item = core::result::Result<std::path::PathBuf, anyhow::Error>> + core::marker::Send>>
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/ptr/mod.rs:507:	core::ptr::drop_in_place::<core::pin::Pin<alloc::boxed::Box<dyn futures_core::stream::Stream<Item = core::result::Result<std::path::PathBuf, anyhow::Error>> + core::marker::Send>>>
crates/worktree/src/worktree.rs:4018:	<worktree::BackgroundScanner>::scan_dir::{closure#0}
crates/worktree/src/worktree.rs:3754:	<worktree::BackgroundScanner>::scan_dirs::{closure#0}::{closure#0}::{closure#0}
crates/gpui/src/executor.rs:457:	<gpui::executor::Scope>::spawn::<<worktree::BackgroundScanner>::scan_dirs::{closure#0}::{closure#0}::{closure#0}>::{closure#0}
/rustc/7cf61ebde7b22796c69757901dd346d0fe70bd97/library/core/src/task/poll.rs:51:	<core::task::poll::Poll<()>>::map::<core::result::Result<(), alloc::boxed::Box<dyn core::any::Any + core::marker::Send>>, core::result::Result<(), alloc::boxed::Box<dyn core::any::Any + core::marker::Send>>::Ok>
/Users/administrator/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.7.0/src/raw.rs:557:	<async_task::raw::RawTask<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = ()> + core::marker::Send>>, (), <gpui::executor::BackgroundExecutor>::spawn_internal<()>::{closure#0}, ()>>::run
libdispatch.dylib	_dispatch_client_callout
libdispatch.dylib	_dispatch_root_queue_drain
libdispatch.dylib	_dispatch_worker_thread2
libsystem_pthread.dylib	_pthread_wqthread
libsystem_pthread.dylib	start_wqthread

Thank you for sharing!

I am not necessarily opposed to this change, I just felt it would be useful to know a bit more context.

Hmm. I wonder if, in another thread that is not bound by the executor, but does wait until after the worktree code has started up, destroying one of the directories in question would do it? That would make sense, and would explain why we get such a lovely "error" from the OS. That would be a classic case of "force majeure" that we typically do want to ignore, yes.

For comparison a failure to close a file during the drop of a file descriptor is silently ignored as I would expect.

This is different because closing regular files can emit IO errors for which there is no good recovery strategy because it is unclear after calling close whether the FD is already closed successfully or not. The recommended strategy for error handling in those cases is to call sync_all on files first before closing them.

EBADF is very different. It indicates that something closed a file descriptor it did not own, which is a violation of IO safety. Unless you have some evidence that this is just macos weirdness I would expect this to indicate a very serious bug in userspace which can lead to data corruption and UB due to file descriptor confusion.

This error should not be suppressed.

Arguably we should also not silently ignore EBADF for OwnedFd and only suppress other errors such as EIO.

And maybe we should add a fn close(self) -> Result<()> API to allow non-panicking error handling, though even then EBADF shouldn't be silently ignored and the application should be aborted to prevent data from being accidentally clobbered.

Caveat: closedir operates on a DIR*, not on a fd directly. So it's possible that the C library returns EBADF for something that's not an invalid file descriptor but some other invalid state in that wrapping struct.
Is the macos C library open source? Then we can check there. If not it'd require reproducing the issue and tracing syscalls to tell the difference.

It is, though as with everything Apple, it's a mess.

Errors can only occur from the close call, the rest of closedir just does some reclaiming. Most of the kernel is open-source as well, so one could try to figure out if anything except the fd lookup could cause the EBADF, but I'm not motivated enough to do that.

Thanks for looking it up. I think that is enough evidence that there's probably some library-UB going on in zed and we shouldn't try to silence it.

The only place in zed itself which calls as_raw_fd is this code for getting the PID of the child process running in the integrated terminal. While it's definitely very fishy (it's not tied to the lifetime of the PTY), it doesn't close anything so I doubt that this issue is caused there.

As for dependencies, it might help to get a complete backtrace to see which ReadDir::drop specifically is causing this.

It's not necessarily readdir that's causing this, it could be the victim of something else going rogue and closing the wrong fd numbers.

It must be... Though I'm hoping that there is enough locality between the close and the read_dir to point us in the direction.

I have an idea how to harden std types against these kinds of things, but it would be fairly expensive and imo only make sense for debug_assert builds. Randomization for file descriptor numbers. But I'm not sure if people would be on board with it.

If this replicates on macos and users have some janky fuse drivers that'd be a likely suspect. Though those drivers should have a bug filed against them.

A quick (though inconclusive) update: Although I'm not certain it doesn't reproduce, I was unable to reproduce with the example program (it fails before it gets to the end), and my testing with Zed itself, I could only reproduce libgit2/libgit2#6796, not this error with sshfs.

Still to try: updating the example program to run to completion, and setting up a Samba server to test that one.

I just tried compiling Zed on Linux with the rustc_codegen_cranelift backend. It crashes reliably with:

fatal runtime error: IO Safety violation: owned file descriptor already closed

Thread 1 "Zed" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: Bestand of map bestaat niet.
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff786fe8f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007ffff7820fb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff780b472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x0000555562b7e5d1 in std::sys::pal::unix::abort_internal () at library/std/src/sys/pal/unix/mod.rs:366
#5  0x0000555560e37664 in std::sys::pal::unix::fs::debug_assert_fd_is_open () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/std/src/rt.rs:43
#6  0x0000555560dffad6 in std::os::fd::owned::{impl#7}::drop () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/std/src/os/fd/owned.rs:181
#7  0x0000555560e38069 in core::ptr::drop_in_place<std::os::fd::owned::OwnedFd> () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#8  0x000055555b0f8dde in core::ptr::drop_in_place<std::sys::pal::unix::fd::FileDesc> () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#9  0x000055555b0f928c in core::ptr::drop_in_place<std::sys::pal::unix::process::process_common::Stdio> () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#10 0x000055555b0f8856 in core::ptr::drop_in_place<core::option::Option<std::sys::pal::unix::process::process_common::Stdio>> ()
    at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#11 0x000055555b0f930f in core::ptr::drop_in_place<std::sys::pal::unix::process::process_common::Command> () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#12 0x000055555b0f8d67 in core::ptr::drop_in_place<std::process::Command> () at /home/bjorn/Projects/cg_clif2/build/stdlib/library/core/src/ptr/mod.rs:542
#13 0x000055555b10be12 in alacritty_terminal::tty::unix::new () at /home/bjorn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alacritty_terminal-0.23.0/src/tty/unix.rs:285
#14 0x000055555afdc160 in terminal::{impl#7}::new () at crates/terminal/src/terminal.rs:377
#15 0x000055555a49c2fb in project::terminals::{impl#0}::create_terminal () at crates/project/src/terminals.rs:167

$ dist/rustc-clif -vV
rustc 1.80.0-nightly (b1ec1bd65 2024-05-18)
binary: rustc
commit-hash: b1ec1bd65f89c1375d2cf2fb733a87ef390276d3
commit-date: 2024-05-18
host: x86_64-unknown-linux-gnu
release: 1.80.0-nightly
Cranelift version: 0.108.0

Upon further investigation, I found alacritty/alacritty#7996, which fixes the crash I found here. I think it is very likely that the bug this fixes is responsible for the panic described in this issue too.

If you can reproduce this reliably, I’d love to debug it together - do you have time to talk the first week of July? If so please reach out to conrad@zed.dev

In my mind there’s quite a chance this is a bug in zed or its dependencies, but it’s also possible that this is a failure introduced by the filesystem.

things I’d like to try:

adding logging to open/close fds to see if we can find a double close
Knowing more about the file system you’re using

Nvm, I just read the attached issue. I’ll pull that into zed and see if it fixes out problems

I can confirm that using the current master branch of alacritty-terminal fixes the IO safety crash with rustc_codegen_cranelift. It does still crash, but that is because of some unimplemented things in rustc_codegen_cranelift (rust-lang/rustc_codegen_cranelift#1492), not a bug in Zed or any of it's dependencies.

diff --git a/crates/terminal/Cargo.toml b/crates/terminal/Cargo.toml
index ef6260730..3ce869597 100644
--- a/crates/terminal/Cargo.toml
+++ b/crates/terminal/Cargo.toml
@@ -14,7 +14,7 @@ doctest = false
 
 
 [dependencies]
-alacritty_terminal = "0.23"
+alacritty_terminal = { git = "https://github.com/alacritty/alacritty", version = "0.24.1-dev" }
 anyhow.workspace = true
 collections.workspace = true
 dirs = "4.0.0"

alacritty/alacritty#7996 hasn't been published to crates.io yet, so I had to use a git dependency.

@the8472

We've been running the fixed alacritty in preview for a week, and on stable for a few days, and have seen 0 instances of this - this either means we've been unusually lucky, or this is now solved.

Thank you very much to @the8472 for adding the debug asserts, and to @jakobhellermann for figuring out the problem.

I ran into this on notify's CI, except it showed up as an abort rather than the closedir panic:

test poll::tests::remove_file ... ok
fatal runtime error: IO Safety violation: owned file descriptor already closed, aborting
error: test failed, to rerun pass `-p notify --lib`

Caused by:
  process didn't exit successfully: `.../deps/notify-f7d57847dd78bfa8` (signal: 6, SIGABRT: process abort signal)

Full job: https://github.com/notify-rs/notify/actions/runs/30158197016/job/89679165708

I'm fairly sure it's the same underlying problem. debug_assert_fd_is_open in library/std/src/sys/fs/unix.rs does an fcntl(fd, F_GETFD) and aborts on EBADF. It has two callers, OwnedFd::drop and DirStream::drop, and in DirStream::drop it runs immediately before the closedir this issue is about:

let fd = unsafe { libc::dirfd(self.0) };
debug_assert_fd_is_open(fd);

let r = unsafe { libc::closedir(self.0) };
assert!(
    r == 0 || crate::io::Error::last_os_error().is_interrupted(),
    "unexpected error during closedir: {:?}",
    ...
);

Environment:

rustc 1.97.1 (8bab26f4f 2026-07-14) stable
aarch64-apple-darwin
GitHub Actions macos-26-arm64
cargo test --workspace --exclude examples --no-default-features --features macos_fsevent --features tokio
debug profile

The reproducer isn't deterministic, but it is narrow. The test that provokes it does three things:

creates an FSEvents watcher (FSEventStreamCreate plus FSEventStreamScheduleWithRunLoop on a dedicated run loop thread)
creates 4097 paths of the form dir_N/subdir, so 8194 directories, and asks FSEvents to watch all 4097 at once, which fails because a stream seems to cap out around 4096 paths (Filewatching fails silently with >4096 paths fsnotify/fsevents#48)
deletes that whole tree with remove_dir_all while the stream is still running

The abort is rarer than the panic. I found one occurrence across the CI runs I looked at, whereas the panic recurred often enough that someone added the workaround.

Two things we ruled out on our side:

notify doesn't close any raw file descriptors on this platform. The inotify and kqueue backends are the only ones touching RawFd/OwnedFd and neither is compiled in the macOS FSEvents configuration. The FSEvents backend only holds CoreFoundation objects. The descriptors notify owns here all come from plain std::fs::File.

remove_dir_all's own ownership looks right to me. fdreaddir passes the fd to fdopendir and then calls into_raw_fd on the OwnedFd, so closedir becomes the only closer, and remove_dir_all_recursive just borrows that raw fd for openat and unlinkat. The chmod_harder code that does hold an OwnedFd is cfg'd to Linux and Android.

That leaves something else in the process closing the descriptor. Both reports I know of, this one and the Zed report that opened the issue, are macOS applications using FSEvents.

So, two questions:

Is a stray close() coming from a system framework a known failure mode here? Is there a supported way to instrument it, something like interposing on close or a descriptor guard equivalent to Guard Malloc?
Would a standalone reproducer with no notify dependency be useful, just an FSEventStream over a large tree plus a concurrent remove_dir_all? I can write that if it would help, since I gather this hasn't been reproducible internally so far.
`

Is a stray close() coming from a system framework a known failure mode here?

We have had no such reports.

If there are things that close FDs that they don't own it does not necessarily trigger the abort anywhere near the code or on the file descriptor that's responsible because when the numbers start getting reused it can become hopelessly confused which object owns which FD, especially in concurrent programs.
If you get into this situation you may already be facing data corruption from stray writes into the wrong file descriptors. It is irrecoverable global state corruption, which is why we abort the process.

Is there a supported way to instrument it, something like interposing on close or a descriptor guard equivalent to Guard Malloc?

A macos equivalent of strace is what I'd reach for to find double-closes.

Would a standalone reproducer with no notify dependency be useful, just an FSEventStream over a large tree plus a concurrent remove_dir_all? I can write that if it would help, since I gather this hasn't been reproducible internally so far.

Wrong issue? This is the first time that FSEvents has been brought up here. If you think there's an issue in Rust or want input from us please open a new one.
But sure, a minimal reproducer will help anyone looking into it.

Add a standalone reproducer attempt for rust-lang#124105

This is a bug in fsevents, has nothing to do with rust, apologies for the noise

I tracked down the bug and posted a reproducer: https://github.com/daandemeyer/fsevents-fd-repro, nothing to do with rust, but posting it here in case someone else finds this thread while looking into similar issues.

That's at least the 2nd time that this assertion found an actual bug :)

Incomplete list of trophies from searching public github issues:

fix IO Safety violation from dropping OwnedFd multiple times alacritty/alacritty#7996 (this issue that started it)
IO Safety violation runtime error in debug mode stack-rs/rattan#80
fatal runtime error: IO Safety violation: owned file descriptor already closed kata-containers/kata-containers#11054
Panic while calling unregister vorner/signal-hook#200
Hotfix: rustc 1.80.0 fatal runtime error esa-tu-darmstadt/tapasco#404
Segfault w/ zbus D-Bus interface when setting a key in the kernel keyring stratis-storage/stratisd#3946
CloudHypervisor crashed after shutting down VM cloud-hypervisor/cloud-hypervisor#6485

Navigation Menu

Repository navigation

panic: unexpected error during closedir "Bad file descriptor" for unix::fs::Dir #124105

Meta

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

panic: unexpected error during closedir "Bad file descriptor" for unix::fs::Dir #124105

Description

Meta

Activity

workingjubilee commented on Apr 18, 2024

workingjubilee commented on Apr 18, 2024

ConradIrwin commented on Apr 18, 2024

workingjubilee commented on Apr 18, 2024

the8472 commented on Apr 18, 2024

the8472 commented on Apr 18, 2024

joboet commented on Apr 18, 2024

the8472 commented on Apr 18, 2024

joboet commented on Apr 18, 2024

the8472 commented on Apr 18, 2024

joboet commented on Apr 18, 2024

the8472 commented on Apr 18, 2024

15 remaining items

the8472 commented on Apr 18, 2024

ConradIrwin commented on Apr 26, 2024

bjorn3 commented on May 25, 2024

ConradIrwin commented on May 26, 2024

ConradIrwin commented on May 26, 2024

bjorn3 commented on May 26, 2024

ConradIrwin commented on Jun 14, 2024

daandemeyer commented on Jul 25, 2026

the8472 commented on Jul 25, 2026

daandemeyer commented on Jul 25, 2026

daandemeyer commented on Jul 25, 2026

RalfJung commented on Jul 25, 2026

the8472 commented on Jul 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions