[API Proposal]: Zero-overhead member access with suppressed visibility checks

[jkotas]

Background and motivation

Number of existing .NET serializers depend on skipping member visibility checks for data serialization. Examples include System.Text.Json or EF Core. In order to skip the visibility checks, the serializers typically use dynamically emitted code (Reflection.Emit or Linq.Expressions) and classic reflection APIs as slow fallback. Neither of these two options are great for source generated serializers and native AOT compilation. This API proposal introduces a first class zero-overhead mechanism for skipping visibility checks.

API Proposal

namespace System.Runtime.CompilerServices;

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
public class UnsafeAccessorAttribute : Attribute
{
    public UnsafeAccessorAttribute(UnsafeAccessorKind kind);

    public UnsafeAccessorKind Kind { get; }

    // The name defaults to the annotated method name if not specified.
    // The name must be null for constructors
    public string? Name { get; set; }
}

public enum UnsafeAccessorKind
{
    Constructor, // call instance constructor (`newobj` in IL)
    Method, // call instance method (`callvirt` in IL)
    StaticMethod, // call static method (`call` in IL)
    Field, // address of instance field (`ldflda` in IL)
    StaticField // address of static field (`ldsflda` in IL)

    // Potential additions to handle niche cases
    // FieldGet, // get value of instance field (`ldfld` in IL). Required for `ref` fields.  
    // FieldSet, // get value of instance field (`stfld` in IL). Required for `ref` fields.  
    // NonVirtualMethod, // call instance method non-virtually (`call` in IL).
};

This attribute will be applied on extern static method. The implementation of the extern static method annotated with this attribute will be provided by the runtime based on the information in the attribute and the signature of the method that the attribute is applied to. The runtime will try to find the matching method or field and forward the call to it. If the matching method or field is not found, the body of the extern method will throw MissingFieldException or MissingMethodException.

For UnsafeAccessorKind.{Static}Method and UnsafeAccessorKind.{Static}Field, the type of the first argument of the annotated extern method identifies the owning type. The value of the first argument is treated as @this pointer for instance fields and methods. The first argument must be passed as ref for instance fields and methods on structs. The value of the first argument is not used by the implementation for static fields and methods.

The generic parameters of the extern static method are concatenation of the type and method generic arguments of the target method. For example, extern static void Method1<T1, T2>(Class1<T1> @this) can be used to call Class1<T1>.Method1<T2>(). The generic constraints of the extern static method must match generic constraints of the target type, field or method.

Return type is considered for the signature match. modreqs and modopts are not considered for the signature match.

API Usage

class UserData {
  private UserData() { }
  public string Name { get; set; }
}

[UnsafeAccessor(UnsafeAccessorKind.Constructor)]
extern static UserData CallPrivateConstructor();

// This API allows accessing backing fields for auto-implemented properties with unspeakable names
[UnsafeAccessor(UnsafeAccessorKind.Field, Name = "<Name>k__BackingField")]
extern static ref string GetName(UserData @this);

UserData ud = CallPrivateConstructor();
GetName(ud) = "Joe";

Alternative Designs

• Expand unsafe accessors in Roslyn The primary disadvantages is that compile time decision can be invalidated at runtime because the assembly can change. For example, Roslyn could be looking at a ref assembly where privates are stripped, NuGet version changes could cause compiler to see version 1 of an assembly at one phase in the build but at runtime we end up using version 2, etc.
• Allow suppressing visibility checks in Roslyn. This option was proposed and rejected in Add public methods to supress visibility checks for compilation roslyn#11149. It violates fundamental language design principles.
• Sidecar .dll with suppressed visibility checks. Runtime allows suppressing visibility checks in IL via IgnoresAccessChecksToAttribute. Build can generate sidecar .dll with trivial wrappers to access the private fields and methods with visibility checks suppressed. The source generated serializer would then call these wrappers to access the private members. It would be zero-cost way to circumvent visibility checks, assuming that the trivial wrapper methods get inlined by the JIT. The challenge with this option is integration into the overall build flow.
• Better Reflection APIs. We have work planned on new faster reflection APIs in .NET 8. The challenge with this option is that reflection APIs are always going to be significantly slower than a direct method invocation or a simple field access. We would be leaving performance on the table.
• Optimize reflection invoke call via pattern-matching The challenge with this option is that reflection invoke tends to be a complex code. Pattern matching them in the JIT or some other dedicated optimization step would be complex and fragile. For example, consider what it would take to pattern match typeof(MyType).GetMethod("set_IntProperty", BindingFlags.Public | BindingFlags.Instance).Invoke(BindingFlags.DoNotWrapExceptions , ptr, new object[] { (object)intValue });, optimize it to p.IntProperty = intValue;.
• Do nothing Live with the fact that there is no cheap way to circumvent visibility checks without Reflection.Emit. Encourage serializers that want to be lean, fast and AOT friendly to avoid these patterns.

Risks

• The proposed API has to be recognized in all linker, AOT compilers and runtimes.
• The proposed API addressed skipping member visibility checks only. Skipping of type visibility checks can be added later (look for UnsafeAccessorType in the discussion below for the potential design).

Tagging subscribers to this area: @dotnet/area-system-runtime-compilerservices
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Number of existing .NET serializers depend on skipping member visibility checks for data serialization. Examples include System.Text.Json or EF Core. In order to skip the visibility checks, the serializers typically use dynamically emitted code (Reflection.Emit or Linq.Expressions) and classic reflection APIs as slow fallback. Neither of these two options are great for source generated serializers and native AOT compilation. This API proposal introduces a first class zero-overhead mechanism for skipping visibility checks.

API Proposal

namespace System.Runtime.CompilerServices;

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
public class UnsafeAccessorAttribute : Attribute
{
    public LibraryImportAttribute(string name, UnsafeAccessorKind kind)
    {
        Name = name;
        Kind = kind;
    }
}

public enum UnsafeAccessorKind
{
    Constructor = 1,
    InstanceMethod = 2,
    StaticMethod = 3,
    FieldGetter = 4,
    FieldSetter = 5
};

This attribute will be applied on extern static method. The implementation of the extern static method annotated with this attribute will be provided by the runtime based on the information in the attribute and the signature of the method that the attribute is applied to. The runtime will try to find the matching method or field and forward the call to it. If the matching method or field is not found, the body of the extern method will throw MissingFieldException or MissingMethodException.

API Usage

class UserData {
  private UserData() { }
  public string Name { get; set; }
}

[UnsafeAccessor(".ctor", UnsafeAccessorKind.Constructor)]
extern UserData CallPrivateConstructor()

// This API allows accessing backing fields for auto-implemented properties with unspeakable names
[UnsafeAccessor("<Name>k__BackingField", UnsafeAccessorKind.FieldSetter)]
extern static void SetName(UserData ud, string v);

UserData ud = CallPrivateConstructor();
ud.SetName(ud, "Joe");

Alternative Designs

• Expand unsafe accessors in Roslyn The primary disadvantages is that compile time decision can be invalidated at runtime because the assembly can change. For example, Roslyn could be looking at a ref assembly where privates are stripped, NuGet version changes could cause compiler to see version 1 of an assembly at one phase in the build but at runtime we end up using version 2, etc.
• Allow suppressing visibility checks in Roslyn. This option was proposed and rejected in Add public methods to supress visibility checks for compilation roslyn#11149. It violates fundamental language design principles.
• Sidecar .dll with suppressed visibility checks. Runtime allows suppressing visibility checks in IL via IgnoresAccessChecksToAttribute. Build can generate sidecar .dll with trivial wrappers to access the private fields and methods with visibility checks suppressed. The source generated serializer would then call these wrappers to access the private members. It would be zero-cost way to circumvent visibility checks, assuming that the trivial wrapper methods get inlined by the JIT. The challenge with this option is integration into the overall build flow.
• Better Reflection APIs. We have work planned on new faster reflection APIs in .NET 8. The challenge with this option is that reflection APIs are always going to be significantly slower than a direct method invocation or a simple field access. We would be leaving performance on the table.
• Optimize reflection invoke call via pattern-matching The challenge with this option is that reflection invoke tends to be a complex code. Pattern matching them in the JIT or some other dedicated optimization step would be complex and fragile. For example, consider what it would take to pattern match typeof(MyType).GetMethod("set_IntProperty", BindingFlags.Public | BindingFlags.Instance).Invoke(BindingFlags.DoNotWrapExceptions , ptr, new object[] { (object)intValue });, optimize it to p.IntProperty = intValue;.
• Do nothing Live with the fact that there is no cheap way to circumvent visibility checks without Reflection.Emit. Encourage serializers that want to be lean, fast and AOT friendly to avoid these patterns.

Risks

• The proposed API has to be recognized in all linker, AOT compilers and runtimes.
• The proposed API addressed skipping member visibility checks only. It does not address skipping type visibility checks.

Author:	jkotas
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.CompilerServices
Milestone:	-

[DaZombieKiller]

Is there an advantage in providing FieldGetter/FieldSetter accessor kinds over a single Field kind that returns a ref? Keeping them separate allows for the system to error if you try to mutate a readonly field, but that ability is needed in many serializers so I'm not sure if it's useful.

public enum UnsafeAccessorKind
{
    Constructor = 1,
    InstanceMethod = 2,
    StaticMethod = 3,
    Field = 4,
};

[UnsafeAccessor("<Name>k__BackingField", UnsafeAccessorKind.Field)]
extern static ref int GetName(ref StructType @this);

I'm also curious, should the return type need to match the field type? If not, how would this work for fixed buffers or fields of non-public types?

[jkotas]

single Field kind that returns a ref?

Thanks. Applied your suggestion.

how would this work for fixed buffers or fields of non-public types?

The proposal does not work for types that you do not have access to. It is called out in the risks section.

[DaZombieKiller]

The proposal does not work for types that you do not have access to. It is called out in the risks section.

Do fixed buffers also fall under that, since the type is unspeakable and "hidden" by the compiler?

Since fixed buffer fields are annotated with [FixedBuffer(typeof(T), length)], maybe a Span could be used?

That would extend the proposal to:

public enum UnsafeAccessorKind
{
    Constructor = 1,
    InstanceMethod = 2,
    StaticMethod = 3,
    Field = 4,
    FixedBufferField = 5,
};

public struct Example
{
    private unsafe fixed int _buffer[5];
}

[UnsafeAccessor("_buffer", UnsafeAccessorKind.FixedBufferField)]
static extern Span<int> GetBuffer(ref Example @this);

Alternatively, allowing the element type to be used as the accessor's return type could work. Callers would then need to query the length with reflection and use Unsafe.Add. I suppose this also applies to [InlineArray] if that is implemented in the future.

[EgorBo]

What if my type has a private field, and the type of that field is also private (e.g. private struct MyStruct or private enum, etc) ?

[DaZombieKiller]

I was thinking about how non-public and unspeakable types could be handled here, and the idea of using TypedReference came to mind. However, there's a blocker: methods are seemingly not allowed to return TypedReference.

It can be passed through a pointer though. So if TypedReference is leveraged and UnsafeAccessor supports providing a fully-qualified type name, it should be possible to handle private types to some extent:

// ExampleAssembly
internal struct CustomString
{
    public string ToSerializedString() => throw null;
}

public class UserData
{
    private UserData() { }
    internal CustomString Name { get; set; }
}

// Generated
[UnsafeAccessor("<Name>k__BackingField", UnsafeAccessorKind.Field)]
extern static unsafe void GetName(UserData @this, TypedReference* outReference);

[UnsafeAccessor("ToSerializedString", UnsafeAccessorKind.InstanceMethod, DeclaringTypeName = "CustomString, ExampleAssembly")]
extern static string ToSerializedString(TypedReference @this);

TypedReference field;
UserData ud = CallPrivateConstructor();
GetName(ud, &field);
string name = ToSerializedString(field);

Something about this feels really nasty though, I'm not sure that I like it. Even if it's only intended to be used by generated code, it seems like "too much".

[kant2002]

Isn't this would be fragile if used without source generator. what if we have class defined like this.

// v1
class UserData {
  private UserData() { }
  public string Name { get; set; }
}

[UnsafeAccessor("<Name>k__BackingField", UnsafeAccessorKind.Field)]
extern static ref int GetName(UserData @this);

and then change it to

// v2
class UserData {
  private string name;
  private UserData() { }
  public string Name { get => this.name; set => this.name = value; }
}

[UnsafeAccessor("<Name>k__BackingField", UnsafeAccessorKind.Field)]
extern static ref int GetName(UserData @this);

Now it's easy to forget to update "<Name>k__BackingField"

[kant2002]

I initially miss namespace where this would be living, and probably I less worry about wrong people misusing stuff. anyway what about multiple construtors and constructors with parameters?

V1. Custom parameter

class UserData {
  private UserData(string name) { }
  public string Name { get; set; }
}

[UnsafeAccessor(".ctor", UnsafeAccessorKind.Constructor)]
extern UserData CallPrivateConstructor(string param)

Is this possible?

V2. Multiple construtors

class UserData {
  private UserData() { }
  private UserData(string name) { }
  public string Name { get; set; }
}

[UnsafeAccessor(".ctor", UnsafeAccessorKind.Constructor)]
extern UserData CallPrivateConstructor(string param)

how to disambiguate between construtors?

[BreyerW]

Not sure how feasible/desirable but another alternative would be to extend unsafer unsafe proposal like i proposed here dotnet/csharplang#6476 (comment) namely to allow unsafe to skip visibility checks.

[lambdageek]

/cc @vargaz @SamMonoRT FYI

[jkotas]

@DaZombieKiller

Do fixed buffers also fall under that, since the type is unspeakable and "hidden" by the compiler?

Yes, fixed buffers fall under that. I do not think it makes sense to create a special solution for fixed buffers. If we do anything here, it should be a general solution for all unaccessible types.

if TypedReference is leveraged and UnsafeAccessor supports providing a fully-qualified type name, it should be possible to handle private types to some extent

Yes, there are ways to naturally extend this design to support inaccessible types. It is not necessary for the first iteration. The serializers scenarios that are motivating this proposal do not need it.

TypedReference is one of the tools that can be part of the solution. The design you have proposed would only work for fields, it would not work for methods that may have multiple types in signatures.

@kant2002

Isn't this would be fragile if used without source generator

Yes, this has the exact same problems as private reflection has today.

how to disambiguate between constructors?

The accessor method has to have signature that matches the target method. It means that accessor method with UnsafeAccessorKind.Constructor with one string argument is always going to match a constructor with one string argument and no other constructors.

@BreyerW

Not sure how feasible/desirable but another alternative would be to extend unsafer unsafe proposal like i proposed here dotnet/csharplang#6476 (comment) namely to allow unsafe to skip visibility checks.

This is not feasible. It is variant of "Allow suppressing visibility checks in Roslyn" and "Expand unsafe accessors in Roslyn" mentioned in the alternative designs.