Runtime error when loading and runnning assembly with big types

[vzarytovskii]

Please, read the first comment for more details and TL;DR

When compiling assembly with one namespace, which contains 3 types, and each type contains 40_000 instance methods, running the assembly fails with:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at Program+Main.main(System.String[])
NativeCommandExitException: Program "dotnet.exe" ended with non-zero exit code: -1073741819.

Script to generate Program.fs (which then can be just built using normal .NET):

open System.Text
open System.IO

#nowarn "20"

let generateTypes (x:int) (y:int) =
    let sb = StringBuilder()
    for i in 1..x do
        sb.Append("[<NoComparison; NoEquality>]\n")
        sb.Append(sprintf "type Type%d() =\n" i)
        for j in 1..y do
            sb.Append(sprintf "    member _.Method%d() = printfn \"Type%d Method%d\"\n" j i j)
        sb.Append('\n')
    sb

let generatedCode = generateTypes 1 (65533)

generatedCode.Append("module Main =\n")
generatedCode.Append("    [<EntryPoint>]\n")
generatedCode.Append("    let main _ =\n")
generatedCode.Append("        printfn \"Fin\"\n")
generatedCode.Append("        0\n")

File.WriteAllText("Program.fs", generatedCode.ToString())

Update: so, 1 type with 65532 methods causes no issues 65533 methods (+1 more) shows the issue.

It seems we overflow somewhere above 65535 (65532 + main + .ctor(s)), MethodDef table shows 65535 in both cases, however in the second case (case of failure) it seems all the methods' type is the same (AssemblyAttribute), meaning when we have more than 65k methods in total, all of them go to one type int the table.

Update 2: if we have much more methods (70k) we get AccessViolation, if we have exactly 65535/65536 - we get te initial issue:

Update 3: My initial testing methodology is incorrect - it is obvious that the limitation for one type is 65k (known CLR limitation), however if you will generate 2 types with 32766 methods each, it will still fail,

Codegen for 2 types:

open System.Text
open System.IO

#nowarn "20"

let generateTypes (x:int) (y:int) =
    let sb = StringBuilder()
    for i in 1..x do
        sb.Append("[<NoComparison; NoEquality>]\n")
        sb.Append(sprintf "type Type%d() =\n" i)
        for j in 1..y do
            sb.Append(sprintf "    member _.Method%d() = printfn \"Type%d Method%d\"\n" j i j)
        sb.Append('\n')
    sb

let generatedCode = generateTypes 2 (32766)

generatedCode.Append("module Main =\n")
generatedCode.Append("    [<EntryPoint>]\n")
generatedCode.Append("    let main _ =\n")
generatedCode.Append("        printfn \"Fin\"\n")
generatedCode.Append("        0\n")

File.WriteAllText("Program.fs", generatedCode.ToString())

Unhandled exception. System.TypeLoadException: Type '<StartupCode$helloworld>.$.NETCoreApp,Version=v7.0.AssemblyAttributes' from assembly 'helloworld, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' contains more methods than the current implementation allows.

Provide any related information (optional):

• Win 11, latest .NET8 RC

[vzarytovskii]

Issue is the same as #16024 (I will leave this one since it has more repro and investigations).

The issue was there forever (I can reproduce it on 7.0.101, 6.0.121, 5.0.408 and even 3.1.426).

It seems that that when @aaronmu hit the initial issue, they have had some near-threshold amount of method, bindings, and types.

The reason that it was/is working for them in 7.0.2xx and doesn't work in 7.0.3xx is that something changed in our codegen (e.g. we started generating more methods for types by default), which is hitting the threshold for them in the newer SDK.

We should get to the bottom of it, but at least it seems that it's not a recent regression introduced in NET7.

TL;DR: the bug was there forever, repro is easy - have > 65535 methods in assembly combined (incl. auto-generated from equality, comparison, DUs, etc) and it will result in incorrect assembly, which will fail in runtime in one of two wasy:

1. AccessViolation, when there are much more methods than 65535 (reproduces on 70k)
2. TypeLoadException, when there are exactly 65535+-1 method (probably how loader works).

Workarounds, until we have a fix, are: well, reduce amount of methods in one dll:

• Annotating methods which don't need comparison and equality with [<NoComparison;NoEquality>], this will allow not to generate non-needed/non-used methods.
• Move some shared code to separate project.

[T-Gro]

Another tip:

If you do want structural equality and comparison (they are useful), another way to reduce methods is to apply [<DefaultAugmentation(false)>]. This reduces the codegen of DU methods which are typically consumed by C# code.

[cataggar]

This does look like it may be the cause of the AccessViolation exceptions (internal Teams thread). Any executable or test that includes a reference to our Domain project ends up with an AccessViolation when built without <Optimize>true</Optimize>, which Debug configuration normally does and is important for debugging.

We are using an F# JSON library and my guess is that we exceeded the 65535 barrier described above for methods on a type. I wonder if there is an easy way to inspect the assembly to see. I wonder why it is not impacting optimized assemblies. Will they be impacted when more serialization is added?

For us, I would say this is high impact.

[cataggar]

It would be great if there was a utility that could count the methods in an assembly to see if it exceeds 65535. This misses all of the internal and compiler generated method. It would be nice to have something similar that worked.

let dllPath = @"C:\Users\cataggar\ms\Domain.dll"
open Mono.Cecil
let mdl = ModuleDefinition.ReadModule dllPath
let mutable methodCount = 0
for typ in mdl.Types do
    methodCount <- methodCount + typ.Methods.Count
printfn "Types: %d, Methods: %d" mdl.Types.Count methodCount

[cataggar]

I figured it out. We have 86249 methods in our assembly, which exceeds 65535.

// System.Reflection.Metadata 8.0.0
// https://learn.microsoft.com/en-us/dotnet/api/system.reflection.metadata.metadatareader

open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable

let main() =
    use fs = new FileStream(dllPath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)
    use peReader = new PEReader(fs)
    let mr = peReader.GetMetadataReader()

    let mutable methodCount = 0
    for tdefh in mr.TypeDefinitions do
        let tdef = mr.GetTypeDefinition tdefh
        let ns = mr.GetString(tdef.Namespace);
        let name = mr.GetString(tdef.Name);
        let methods = tdef.GetMethods()
        //printfn "%d %s.%s" methods.Count ns name
        methodCount <- methodCount + methods.Count
    printfn "Types: %d, Methods: %d" mr.TypeDefinitions.Count methodCount

main()