Use memfd_create when available

[am11]

Closes #92905

[am11]

Armel failure is unrelated, fixed by dotnet/dotnet-buildtools-prereqs-docker#1141.

[stephentoub]

If the information in flags and perms isn't factored in here, where does it get incorporated?

[am11]

memfd_create flags do not have direct equivalents for read-only or read-write permissions. The flags used with memfd_create are mainly related to file descriptor behavior (e.g., closing on exec and allowing sealing), not the memory protection levels. Therefore, it makes sense to keep MFD_CLOEXEC hardcoded in C.

It was missing mmap call to set the protection, which I have just added. Inheritance is set the same way as with shm_open (default: CLOEXEC, clear flag if Inheritable is requested from line 244).

[stephentoub]

If flags and perms aren't relevant to the if block, should they be moved to the else block? They're only ever used there. I realize it's inside of a retry loop, but we expect retries to be rare bordering on non-existent.

[stephentoub]

That said, would there be any hardening benefits to using seals as a stand-in for what perms was being used for?

[am11]

When using shm_open with read-only permissions (e.g., O_RDONLY) and mapping it with mmap with read-only protections (e.g., PROT_READ), the resulting memory mapping will not allow writing through that specific file descriptor and mapping. However, if another process has opened the same shared memory object with read-write permissions (e.g., O_RDWR), it can still write to the shared memory, and those changes will be visible to the read-only mappings.

With memfd_create there is no protection on fd by default. We can write(fd) unless we implement write sealing: am11@f421782. This will make it readonly for current process (same as shm_open) as well as other processes (different than shm_open).

While it is not exactly the drop-in replacement, but I think it is a goodness that we will be more hardened than shm_open?

[stephentoub]

Yeah, other than the extra syscall, there doesn't appear to be a downside to setting seals and it will help to harden the permissions. I suggest we add it in. At that point, since there's then multiple interop calls involved, having completely separate code paths for memfd_create vs shmopen, including error handling, would seem to make sense.

[am11]

CI is using older glibc 2.23 (Ubuntu 16.04), which doesn't have the API.

  -- Looking for memfd_create
  -- Looking for memfd_create - not found

so it's using shm_open fallback.

I'll change it to make syscall directly (which is what libc 2.27 onwards are doing).

[adamsitnik]

@am11 Once again big thanks for your contribution. We have missed the Preview 7 snap by very little and now we unfortunately have to wait until main branch becomes .NET 10.

I added the "NO MERGE" label and set the 10.0 milestone to express that. The PR is good in the current shape, we just need to wait.

Thanks!

[am11]

Branch is now opened.

[adamsitnik]

LGTM, thank you again @am11 !