Add std::fs::{Home|Media}Dirs

[CAD97]

View all comments

• New ACP: ACP: std::fs::UserDirs or similar libs-team#830

Replacement for std::os::unix::xdg as suggested by libs-api in #157515 (comment). Exposes media directories common between the three big OSes in addition to the cache/config/data/state directories under a separate feature gate. API summary:

// mod std::fs
struct HomeDirs { /* ... */ }
impl HomeDirs {
    fn empty() -> Self;
    fn take(&mut self) -> Self;

    fn config_home(&self) -> Option<&Path>;
    fn data_home(&self) -> Option<&Path>;
    fn state_home(&self) -> Option<&Path>;
    fn cache_home(&self) -> Option<&Path>;

    fn set_config_home(&mut self, path: PathBuf) -> &mut Self;
    fn set_data_home(&mut self, path: PathBuf) -> &mut Self;
    fn set_state_home(&mut self, path: PathBuf) -> &mut Self;
    fn set_cache_home(&mut self, path: PathBuf) -> &mut Self;
}

struct MediaDirs { /* ... */ }
impl MediaDirs {
    fn empty() -> Self;
    fn take(&mut self) -> Self;

    fn desktop(&self) -> Option<&Path>;
    fn documents(&self) -> Option<&Path>;
    fn downloads(&self) -> Option<&Path>;
    fn music(&self) -> Option<&Path>;
    fn pictures(&self) -> Option<&Path>;
    fn videos(&self) -> Option<&Path>;

    fn set_desktop(&mut self, path: PathBuf) -> &mut Self;
    fn set_documents(&mut self, path: PathBuf) -> &mut Self;
    fn set_downloads(&mut self, path: PathBuf) -> &mut Self;
    fn set_music(&mut self, path: PathBuf) -> &mut Self;
    fn set_pictures(&mut self, path: PathBuf) -> &mut Self;
    fn set_videos(&mut self, path: PathBuf) -> &mut Self;
}

// mod std::os::darwin::fs
trait HomeDirsExt for HomeDirs {
    fn sysdir() -> io::Result<Self>;
}

trait MediaDirsExt for MediaDirs {
    fn sysdir() -> io::Result<Self>;
}

// mod std::os::unix::fx
trait HomeDirsExt for HomeDirs {
    fn xdg() -> io::Result<Self>;

    fn runtime_home(&self) -> Option<&Path>;
    fn config_dirs(&self) -> Option<env::SplitPaths<'_>>;
    fn data_dirs(&self) -> Option<env::SplitPaths<'_>>;

    fn set_runtime_home(&mut self, path: PathBuf) -> &mut Self;
    fn set_config_dirs(&mut self, paths: impl IntoIterator<Item: AsRef<OsStr>>) -> Result<&mut Self, env::JoinPathsError>;
    fn set_data_dirs(&mut self, paths: impl IntoIterator<Item: AsRef<OsStr>>) -> Result<&mut Self, env::JoinPathsError>;
}

trait MediaDirsExt for MediaDirs {
    fn xdg() -> io::Result<Self>;

    fn templates(&self) -> Option<&Path>;
    fn set_templates(&mut self, path: PathBuf) -> &mut Self;
}

// mod std::os::windows::fs
trait HomeDirsExt for HomeDirs {
    fn known_folders() -> io::Result<Self>;
}

trait MediaDirsExt for MediaDirs {
    fn known_folders() -> io::Result<Self>;
}

---

This implementation diverges from the directories crate's mapping in that we set state_dir in the non-unix constructors (to ~/Library/Application Support on Darwin and %APPDATA% on Windows). This mapping is derived from the idea that "state" files are application support files that are not important nor portable enough to the user to be "data" files.

---

The XDG paths are as described in the XDG Base Directories Specification and the xdg-user-dirs tool. $XDG_CONFIG_DIR/user-dirs.dirs is parsed directly to avoid delegating to potentially arbitrary shell execution.

The Darwin paths are loaded via the sysdir(3) API from libSystem.dylib (introduced in macOS 10.12 with a similar timeline for other Darwin OSes, deprecating the earlier NSSystemDirectories.h API). Using the File System Effectively points to preferring the Foundation framework's NSSearchPathForDirectoriesInDomain(_:_:_:) or NSFileManager.URLForDirectory instead, but calling those correctly requires an active Objective C autorelease pool, IIUC. The Library/Application Support directory is used for config_home, data_home, and state_home; the Apple documentation The Library Directory Stores App-Specific Files directly calls out placing data and configuration files in Library/Application Support, and state files are just less user-meaningful data files.

The Windows paths are loaded via the Known Folders API (introduced in Vista). config_home and data_home are placed in AppData\Roaming as files intended to be important and portable to the user, while cache_home and state_home are placed in AppData\Local as files that aren't.

---

I'm not fully confident about the handling of the XDG base directory paths which don't have good cross-platform analogs, as well as the exact API for the search path dealing functions, but I'm confident that the shape of the rest of the API does match the stdlib API style. Common paths are platform-independent enough of a needed concept to be exposed by std, IMHO, but platform-specific that a struct with public fields (even #[non_exhaustive]) seems incorrect, specifically because of platform-specific paths that we may want to expose like is already done for XDG.

The one API change I could see doing is moving state_dir into the XDG UserDirsExt. I chose not to do this for this initial implementation, though, as getting the ideal choice of fallback for both Darwin and Windows can't be achieved in an OS-agnostic way:

-	Darwin	Windows
cache	~/Library/Caches	~/AppData/Local
config	~/Library/Application Support	~/AppData/Roaming
data	~/Library/Application Support	~/AppData/Roaming
state	~/Library/Application Support	~/AppData/Local

A more drastic change would be to move all four onto the unix UserDirsExt, adding caches/application_support to the Darwin UserDirsExt and roaming_app_data/local_app_data to the Windows UserDirsExt. This would be more "correct" but seems a bit heavy-handed, as it would mean applications need to pull in OS-specific extension traits just to place their support files in something more appropriate than a ~/.appname directory.

We could also separate the "home" directory API from the "media" directory API. I'm neutral on this with one relevant note: the app-specific cache/config/data/state files need a subdirectory named after the application, so "ProjectDirs" would exclude the media directories; it could make sense to have a type with just those and a push_application_subdir method.

Switching the impl to using a pal imp::UserDirs could be reasonable, but seems at odds with the desire to have the target agnostic way to "build your own" UserDirs. An ExtraUserDirs instead of the #[allow(dead_code)] fields would make sense, I just didn't know how to best set up that in the pal layer.

---

Disclaimer: This was worked on as part of my employment at Canonical. I initially proposed it independently of my employment, but improving std's functionality is part of my job description, so Canonical told me I should use work time on it.

AI Disclosure: I did not use AI to generate any of the code, with a partial exception for VSCode's AI-assisted smart autocomplete helping with the repetitive parts of the code. All nontrivial code was handwritten. As an experiment, I did use some AI to assist in exploring the problem and API design spaces.

I tested locally on my Ubuntu developer machine, but am relying on CI for Darwin and Windows tests. 🤞

• Unanswered question: Should macOS and Windows use the same system API to set user_home (to NSHomeDirectory and FOLDERID_Profile respectively) instead of env::home_dir? (Should env::home_dir be changed to call those?)
• Unanswered question: Should Windows avoid eagerly linking SHGetKnownFolderPath eagerly? If so, how? Add std::fs::{Home|Media}Dirs #158936 (comment)
• Unanswered question: Should the set_* methods do any kind of validation, such as ensuring the path is non-empty or even absolute?
• Unanswered question: Should the set_* methods take impl AsRef<Path> instead of PathBuf? (Would introduce needless copies without a separate ownership-taking option like replace_* below.)
• Unanswered question: Do we want fn replace_*(&mut self, x: Option<PathBuf>) -> Option<PathBuf> style methods to allow transferring ownership and setting paths back to None?
• Unanswered question: The lookup APIs could theoretically return non-absolute paths. Is this something we should check for and defend against more than this already does?
• Future work: What other NSSearchPathDirectory make sense to expose in the Darwin UserDirsExt?
• Future work: What other KNOWNFOLDERID make sense to expose in the Windows UserDirsExt?

cc @joshtriplett @nia-e

[madsmtm]

Reviewed the Darwin parts.

calling those correctly requires an active Objective C autorelease pool, IIUC

Not that hard though, you can push and pop it with objc_autoreleasePoolPush/objc_autoreleasePoolPop. The bigger problem is that it requires linking Foundation, which has a startup cost we'd rather avoid.

This implementation diverges from the directories crate's mapping [...]

It seems to me that for something as nuanced as these user dirs (with a lot of platform-specific details that are not readily apparent), it might make sense to implement the desires std API in directories first? And once it stabilizes more there, we could upstream it to std?

View changes since this review

[CAD97]

It seems to me that for something as nuanced as these user dirs (with a lot of platform-specific details that are not readily apparent), it might make sense to implement the desires std API in directories first?

This is mostly already the case. The only API-facing changes from directories here are:

• Trimming the less cleanly portable API surface
• Returning Option everywhere (requested by T-libs-api)
• Setter methods for custom setups (requested by T-libs-api)
• Explicit constructors for each convention (requested by T-libs-api)
• Making state_dir fall back to what directories calls data_local_dir for non-XDG constructors
• Combining UserDirs/BaseDirs (probably going to revert this)

The ideal API shape inside std and in a crate often differ slightly. This approved impl experiment is to determine if a form of this API that fits std's goals exists.

---

I'm going to split the base directory discovery and the user/media directories into different types to better represent that the existence of these sets is not strongly correlated and fix the things @madsmtm pointed out w.r.t. docs and the darwin impl, then this should be good for proper libs-api review.

The use of shlex for shell-unquote for the XDG user dirs needs a resolution, but doing the work to give shlex a rustc-dep-of-std feature can wait until we know whether that's the direction we want to take.

[madsmtm]

Huh, why?

View changes since the review

[CAD97]

I'm assuming, tbqh; the manpage doesn't really specify. Note that here libc incorrectly translates sysdir_search_path_domain_mask_t as an enum when it's a bitmask, so there's no potential for unsafety currently, but I'm being conservative.

[madsmtm]

I did see that libc issue, yeah.

I'm confident that it's not an issue though, the only way I could imagine it would work differently is if they decided in the future to use the rest of the integer for flags, and one of those flags doing something unsound. But that sounds improbable given that we're using the API in exactly the intended / documented fashion.

(It sounds a lot more likely to me that they'd add a new API if they really needed some new functionality).

[madsmtm]

r=me on the Darwin impl now (haven't tested it this time around, but pretty sure CI will catch it if it doesn't work), thanks!

Don't have much of an opinion on the API design and the other platforms, I'll leave that to someone else.

View changes since this review

[ChrisDenton]

For Windows I think we should ideally be delay loading at least SHGetKnownFolderPath and probably CoTaskMemFree too. SHGetKnownFolderPath is a GUI shell function and we try to avoid those in std (e.g. library/std/src/sys/args/windows.rs#L45). For local_app_data and roaming_app_data I think we should prefer the environment variables and only fallback to using the gui shell functions if that fails. That allows both allows users to override them and for it to work without the shell. That also implies that it should be possible to get the dirs.home.* directories without caching the dirs.media.* directories.

Tbh, I'm not personally convinced that the media directories are a good fit for std, at least for Windows, and not just for the reason above. Mostly CLI tools are going to be using either the current directory or otherwise explicit directories and GUI tools are usually going to want the GUI file/directory picker so I think this is fairly niche unless the standard library grows to encompass more GUI stuff. But that's ultimately a libs-api decision.

View changes since the review

[CAD97]

You got in right before I split UserDirs properly (76ddaa3), so it's now possible to load the HomeDirs without MediaDirs. I had forgotten the fact that linking into shell32.dll marks the application as graphical; that's reason enough to prefer using the %APPDATA% environment variables if they're present.

(I'm not sure how we'd do the delayed DLL load in std.)

[ChrisDenton]

We do have some helper macros, although they are more geared towards compatibility shims. But as I said, I'm happy to leave that to later. It might be enough that they're separate because now they'll be optimised out if they're never used. So that just leaves the case where they're conditionally used. But I doubt many CLI apps will have much use for media directories (as output paths are typically given explicitly or else the current directory is used).

[CAD97]

Well, I believe I've implemented lazy loading in 23bd91d; I'd appreciate a quick review of how I did so.

[CAD97]

I consider this fully ready for review now.

r? @rust-lang/libs-api

[rustbot]

⚠️ Warning ⚠️

• There are issue links (such as #123) in the commit messages of the following commits.
  Please move them to the PR description, to avoid spamming the issues with references to the commit, and so this bot can automatically canonicalize them to avoid issues with subtree.
  • fed84bc

[CAD97]

I did a clean rebase onto upstream main (no commit edits) on prompting by rustbot (not initially realizing that it's a "once needed" reminder).

Initial feedback from libs-api addressed, so putting this back for review. Continuing API review is happening on the ACP, so passing back to libs review.

@rustbot ready

r? rust-lang/libs

[ChrisDenton]

I don't think we need our own waiting mechanism here. It should be fine to call LoadLibraryEx from multiple threads since it does its own locking when loading modules. It doesn't really matter which thread wins the race, they'll all get back the module handle. Losing threads could call FreeLibraryEx to decrement the counter but I'm not even sure that's necessary seeing as we never intend to unload it.

View changes since the review

[jhpratt]

@rustbot reroll