Can AsyncMethodBuilder trust MoveNext?

[benaadams]

AsyncXMethodBuilder Start<> can't inline as it uses a try+finally to restore the ExecutionContext

However, the generated state machine will always capture any exception and set it in the Task.

So... Question is can the MoveNext be trusted and the the try+finally removed?

Part of investigating https://github.com/dotnet/corefx/issues/18481

/cc @stephentoub

[stephentoub]

Interesting question.

Technically, no, as any IAsyncStateMachine implementation could be used with it, including hand-written ones, ones that purposefully rely on the lack of a finally there to mess with the EC state, etc.

That said, while in desktop I think that's crucial as there's security-related state stored in the EC, in core the only state stored is that for async locals, so the impact is significantly lessened. And the type is in the compiler services namespace, is really meant for compiler use, and thus should be able to be targeted at the patterns we expect of the compilers; folks (in particular me) do use ATMB manually sometimes as a struct-based TCS, but I've never used it for Start (other than in ValueTask's builder) and have trouble envisioning a real case where someone would want to.

I know Start shows up as a measurable cost in some workloads. How much is that lessened in, say, plaintext, if you remove the finally? This may be a case where we need to weigh the perfect against practice.

cc: @kouvel, @vancem

[benaadams]

Change would be something like this dotnet/coreclr#11155

Will measure

[AndyAyersMS]

We could also enhance the inliner so it could inline methods with EH. I've been looking for examples where this would be beneficial.

Initially we probably wouldn't have the heuristics right for these methods and so might not be able to find profitable cases automatically, but it would unblock aggressive inlining.

[stephentoub]

Andy, that would be excellent.

[benaadams]

The dotnet/coreclr#11155 changes are significant; though split function checking for completed and not awaiting is a factor of 10 better. However my tests on that path don't deal with ExecutionContext.

Pre

Sync Chain Awaited (TP)         42.960 M    38.551 M    38.145 M    44.092 M    43.414 M
- Depth    2                    45.323 M    44.743 M    45.622 M    45.640 M    45.760 M
- Depth   16                    27.019 M    27.701 M    28.664 M    27.366 M    27.778 M
- Depth   64                    26.210 M    26.123 M    25.907 M    24.435 M    24.762 M
- Depth  512                    23.774 M    23.402 M    23.538 M    24.187 M    23.240 M

Sync Chain Awaited              45.061 M    67.391 M    85.997 M    65.965 M     3.664 M
- Depth    2                    45.597 M    89.874 M    61.895 M    29.144 M     4.262 M
- Depth   16                    26.697 M    55.823 M    51.840 M    87.253 M     4.045 M
- Depth   64                    24.803 M    42.027 M    46.296 M    27.263 M     3.733 M
- Depth  512                    22.570 M    43.876 M    63.615 M    31.899 M     3.346 M

Post

Sync Chain Awaited (TP)         53.784 M    55.157 M    55.556 M    56.010 M    53.125 M
- Depth    2                    57.475 M    58.418 M    57.174 M    57.034 M    54.887 M
- Depth   16                    44.613 M    45.206 M    45.441 M    45.169 M    43.569 M
- Depth   64                    42.485 M    42.726 M    42.627 M    42.346 M    40.628 M
- Depth  512                    41.049 M    41.723 M    41.588 M    41.480 M    39.786 M

Sync Chain Awaited              53.961 M   102.299 M    97.172 M    32.529 M     4.121 M
- Depth    2                    55.467 M   106.942 M   108.917 M    33.131 M     3.675 M
- Depth   16                    43.523 M    86.100 M    83.682 M    34.969 M     3.757 M
- Depth   64                    40.600 M    81.042 M    86.022 M    31.465 M     3.964 M
- Depth  512                    38.052 M    74.648 M    79.673 M    27.042 M     4.128 M

Pre

CachedTask Chain Await (TP)     40.848 M    40.176 M    40.816 M    39.404 M    40.033 M
- Depth    2                    38.474 M    41.460 M    42.022 M    44.130 M    41.990 M
- Depth   16                    26.572 M    27.443 M    28.260 M    27.699 M    25.754 M
- Depth   64                    24.796 M    26.042 M    25.668 M    25.435 M    23.398 M
- Depth  512                    22.138 M    21.985 M    23.241 M    21.709 M    22.541 M

CachedTask Chain Await          40.766 M    83.720 M    82.680 M    21.114 M     3.368 M
- Depth    2                    43.358 M    79.131 M    50.984 M    34.616 M     4.524 M
- Depth   16                    26.677 M    53.619 M    74.960 M    44.416 M     3.708 M
- Depth   64                    24.115 M    48.153 M   109.730 M    75.999 M     3.649 M
- Depth  512                    22.211 M    41.060 M    69.486 M    30.070 M     3.102 M

Post

CachedTask Chain Await (TP)     48.700 M    49.654 M    48.861 M    49.153 M    47.376 M
- Depth    2                    53.677 M    53.917 M    53.908 M    53.631 M    51.225 M
- Depth   16                    44.168 M    44.775 M    44.396 M    44.031 M    42.487 M
- Depth   64                    40.736 M    41.158 M    41.348 M    40.852 M    39.682 M
- Depth  512                    40.008 M    40.389 M    40.203 M    40.174 M    38.657 M

CachedTask Chain Await          47.808 M    93.415 M    99.891 M    35.489 M     4.107 M
- Depth    2                    52.487 M   103.841 M    82.150 M    31.886 M     4.254 M
- Depth   16                    43.525 M    85.042 M    96.839 M    37.454 M     4.135 M
- Depth   64                    41.321 M    80.286 M    90.187 M    34.551 M     3.810 M
- Depth  512                    39.434 M    71.079 M    97.535 M    32.088 M     3.843 M

[benaadams]

Removed the try and EC switching, inline all that can be and is a little faster again but no where near the 10 fold difference by doing a completion check

[benaadams]

Dropping the EC switching; and try finally as it was suggested that might be the main cost dotnet/roslyn#10449 (comment) and inlining everything for one section (AsyncTaskMethodBuilder``1:Start(byref):this doesn't want to inline even with aggressive)

It seems the main cost is memory traffic?

Successfully inlined AsyncValueTaskMethodBuilder`1:Create():struct (22 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined AsyncTaskMethodBuilder`1:Create():struct (10 IL bytes) (depth 2) [below ALWAYS_INLINE size]
Successfully inlined AsyncValueTaskMethodBuilder`1:Start(byref):this (13 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined AsyncValueTaskMethodBuilder`1:get_Task():struct:this (27 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined ValueTask`1:.ctor(int):this (15 IL bytes) (depth 2) [aggressive inline attribute]
**************** Inline Tree
Inlines into 0600001D OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this
  [1 IL=0018 TR=000015 06000018] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:Create():struct
    [2 IL=0010 TR=000063 06003F8D] [below ALWAYS_INLINE size] AsyncTaskMethodBuilder`1:Create():struct
  [3 IL=0047 TR=000039 06000019] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:Start(byref):this
    [0 IL=0007 TR=000107 06003F8E] [FAILED: unprofitable inline] AsyncTaskMethodBuilder`1:Start(byref):this
  [4 IL=0059 TR=000046 0600001E] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:get_Task():struct:this
    [0 IL=0009 TR=000150 0600001F] [FAILED: noinline per IL/cached result] AsyncValueTaskMethodBuilder`1:GetTaskFromMethodBuilder():struct:this
    [5 IL=0021 TR=000133 06000001] [aggressive inline attribute] ValueTask`1:.ctor(int):this
Budget: initialTime=255, finalTime=359, initialBudget=2550, currentBudget=2648
Budget: increased by 98 because of force inlines
Budget: initialSize=1613, finalSize=1613
; Assembly listing for method OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this
; Emitting BLENDED_CODE for X64 CPU with AVX
; optimized code
; rsp based frame
; partially interruptible
; Final local variable assignments
;
;  V00 this         [V00,T01] (  3,  3   )     ref  ->  rcx         this class-hnd
;  V01 RetBuf       [V01,T00] (  5,  5   )   byref  ->  rsi        
;  V02 arg1         [V02,T02] (  3,  3   )     int  ->   r8        
;  V03 loc0         [V03    ] (  9,  8   )  struct (72) [rsp+0x70]   do-not-enreg[XSFB] must-init addr-exposed ld-addr-op
;  V04 loc1         [V04    ] (  2,  2   )  struct (32) [rsp+0x50]   do-not-enreg[XSFB] must-init addr-exposed ld-addr-op
;  V05 loc2         [V05,T03] (  3,  3   )  struct (32) [rsp+0x30]   do-not-enreg[SFB] must-init ld-addr-op
;* V06 tmp1         [V06    ] (  0,  0   )  struct (24) zero-ref    do-not-enreg[SB] ld-addr-op
;  V07 tmp2         [V07    ] (  5,  3.50)  struct (16) [rsp+0x20]   do-not-enreg[XS] must-init addr-exposed
;* V08 tmp3         [V08    ] (  0,  0   )  struct (16) zero-ref   
;  V09 tmp4         [V09,T04] (  2,  2   )     int  ->  rdx         ld-addr-op
;  V10 tmp5         [V10    ] (  3,  2   )     ref  ->  [rsp+0x20]   do-not-enreg[X] addr-exposed V07._task(offs=0x00) P-DEP
;  V11 tmp6         [V11    ] (  3,  2   )     int  ->  [rsp+0x28]   do-not-enreg[X] addr-exposed V07._result(offs=0x08) P-DEP
;  V12 tmp7         [V12,T05] (  2,  1   )     ref  ->  rcx         V08._task(offs=0x00) P-INDEP
;  V13 tmp8         [V13,T06] (  2,  1   )     int  ->  rdx         V08._result(offs=0x08) P-INDEP
;  V14 OutArgs      [V14    ] (  1,  1   )  lclBlk (32) [rsp+0x00]  
;
; Lcl frame size = 184

G_M45286_IG01:
       57                   push     rdi
       56                   push     rsi
       4881ECB8000000       sub      rsp, 184
       C5F877               vzeroupper 
       488BF1               mov      rsi, rcx
       488D7C2420           lea      rdi, [rsp+20H]
       B926000000           mov      ecx, 38
       33C0                 xor      rax, rax
       F3AB                 rep stosd 
       488BCE               mov      rcx, rsi
       488BF2               mov      rsi, rdx

G_M45286_IG02:
       48894C2470           mov      gword ptr [rsp+70H], rcx
       4489842480000000     mov      dword ptr [rsp+80H], r8d
       488D4C2430           lea      rcx, bword ptr [rsp+30H]

G_M45286_IG03:
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F01           vmovdqu  qword ptr [rcx], xmm0
       C4E17A7F4110         vmovdqu  qword ptr [rcx+16], xmm0

G_M45286_IG04:
       33C9                 xor      rcx, rcx
       488D542438           lea      rdx, bword ptr [rsp+38H]
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F02           vmovdqu  qword ptr [rdx], xmm0
       48894A10             mov      qword ptr [rdx+16], rcx

G_M45286_IG05:
       C4E17A6F442430       vmovdqu  xmm0, qword ptr [rsp+30H]
       C4E17A7F842488000000 vmovdqu  qword ptr [rsp+88H], xmm0
       C4E17A6F442440       vmovdqu  xmm0, qword ptr [rsp+40H]
       C4E17A7F842498000000 vmovdqu  qword ptr [rsp+98H], xmm0

G_M45286_IG06:
       C7442478FFFFFFFF     mov      dword ptr [rsp+78H], -1

G_M45286_IG07:
       C4E17A6F842488000000 vmovdqu  xmm0, qword ptr [rsp+88H]
       C4E17A7F442450       vmovdqu  qword ptr [rsp+50H], xmm0
       C4E17A6F842498000000 vmovdqu  xmm0, qword ptr [rsp+98H]
       C4E17A7F442460       vmovdqu  qword ptr [rsp+60H], xmm0

G_M45286_IG08:
       488D4C2458           lea      rcx, bword ptr [rsp+58H]
       488D542470           lea      rdx, bword ptr [rsp+70H]
       E8FBE6FFFF           call     AsyncTaskMethodBuilder`1:Start(byref):this
       80BC248C00000000     cmp      byte  ptr [rsp+8CH], 0
       7514                 jne      SHORT G_M45286_IG09
       488D8C2488000000     lea      rcx, bword ptr [rsp+88H]
       488D542420           lea      rdx, bword ptr [rsp+20H]
       E887E6FFFF           call     AsyncValueTaskMethodBuilder`1:GetTaskFromMethodBuilder():struct:this
       EB12                 jmp      SHORT G_M45286_IG10

G_M45286_IG09:
       8B942488000000       mov      edx, dword ptr [rsp+88H]
       33C9                 xor      rcx, rcx
       48894C2420           mov      gword ptr [rsp+20H], rcx
       89542428             mov      dword ptr [rsp+28H], edx

G_M45286_IG10:
       488D0E               lea      rcx, bword ptr [rsi]
       488B542420           mov      rdx, gword ptr [rsp+20H]
       E87E09D05F           call     CORINFO_HELP_CHECKED_ASSIGN_REF
       8B442428             mov      eax, dword ptr [rsp+28H]
       894608               mov      dword ptr [rsi+8], eax
       488BC6               mov      rax, rsi

G_M45286_IG11:
       4881C4B8000000       add      rsp, 184
       5E                   pop      rsi
       5F                   pop      rdi
       C3                   ret      

; Total bytes of code 262, prolog size 32 for method OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this
; ============================================================

Marking <GetTotalWorthValueTaskInt32AwaitAsync>d__17:MoveNext():this as NOINLINE because of has exception handling
**************** Inline Tree
Inlines into 06003F8E AsyncTaskMethodBuilder`1:Start(byref):this
  [0 IL=0028 TR=000022 06000066] [FAILED: has exception handling] <GetTotalWorthValueTaskInt32AwaitAsync>d__17:MoveNext():this
Budget: initialTime=159, finalTime=159, initialBudget=1590, currentBudget=1590
Budget: initialSize=883, finalSize=883
; Assembly listing for method AsyncTaskMethodBuilder`1:Start(byref):this
; Emitting BLENDED_CODE for X64 CPU with AVX
; optimized code
; rsp based frame
; fully interruptible
; Final local variable assignments
;
;* V00 this         [V00    ] (  0,  0   )   byref  ->  zero-ref    this
;  V01 arg1         [V01,T00] (  3,  3   )   byref  ->  rdx        
;* V02 tmp0         [V02    ] (  0,  0   )     ref  ->  zero-ref   
;# V03 OutArgs      [V03    ] (  1,  1   )  lclBlk ( 0) [rsp+0x00]  
;
; Lcl frame size = 0

G_M53769_IG01:

G_M53769_IG02:
       488BCA               mov      rcx, rdx
       48B8303D3EF0FA7F0000 mov      rax, 0x7FFAF03E3D30

G_M53769_IG03:
       48FFE0               rex.jmp  rax
; Total bytes of code 16, prolog size 0 for method AsyncTaskMethodBuilder`1:Start(byref):this
; ============================================================

Successfully inlined OrderBook:get_Orders():ref:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined List`1:get_Item(int):ref:this (27 IL bytes) (depth 1) [profitable inline]
Successfully inlined ValueTask`1:GetAwaiter():struct:this (12 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined ValueTaskAwaiter`1:.ctor(struct):this (8 IL bytes) (depth 2) [aggressive inline attribute]
Successfully inlined ValueTaskAwaiter`1:get_IsCompleted():bool:this (12 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined ValueTask`1:get_IsCompleted():bool:this (22 IL bytes) (depth 2) [profitable inline]
Successfully inlined Task:get_IsCompleted():bool:this (16 IL bytes) (depth 3) [below ALWAYS_INLINE size]
Successfully inlined Task:IsCompletedMethod(int):bool (11 IL bytes) (depth 4) [below ALWAYS_INLINE size]
Marking AsyncTaskMethodBuilder`1:AwaitUnsafeOnCompleted(byref,byref):this as NOINLINE because of has exception handling
Successfully inlined AsyncValueTaskMethodBuilder`1:AwaitUnsafeOnCompleted(byref,byref):this (21 IL bytes) (depth 1) [profitable inline]
Successfully inlined ValueTaskAwaiter`1:GetResult():int:this (32 IL bytes) (depth 1) [profitable inline]
Successfully inlined OrderBook:get_Orders():ref:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined List`1:get_Count():int:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined AsyncValueTaskMethodBuilder`1:SetResult(int):this (31 IL bytes) (depth 1) [profitable inline]
**************** Inline Tree
Inlines into 06000066 <GetTotalWorthValueTaskInt32AwaitAsync>d__17:MoveNext():this
  [1 IL=0044 TR=000107 0600000E] [below ALWAYS_INLINE size] OrderBook:get_Orders():ref:this
  [2 IL=0055 TR=000112 060038FA] [profitable inline] List`1:get_Item(int):ref:this
    [0 IL=0009 TR=000252 060001EE] [FAILED: does not return] ThrowHelper:ThrowArgumentOutOfRange_IndexException()
  [0 IL=0060 TR=000116 06000035] [FAILED: noinline per IL/cached result] Order:GetOrderWorthValueTaskInt32AwaitAsync():struct:this
  [3 IL=0069 TR=000123 06000010] [aggressive inline attribute] ValueTask`1:GetAwaiter():struct:this
    [4 IL=0006 TR=000271 0600002A] [aggressive inline attribute] ValueTaskAwaiter`1:.ctor(struct):this
  [5 IL=0078 TR=000132 0600002B] [below ALWAYS_INLINE size] ValueTaskAwaiter`1:get_IsCompleted():bool:this
    [6 IL=0006 TR=000298 06000009] [profitable inline] ValueTask`1:get_IsCompleted():bool:this
      [7 IL=0014 TR=000326 06002291] [below ALWAYS_INLINE size] Task:get_IsCompleted():bool:this
        [8 IL=0010 TR=000342 06002292] [below ALWAYS_INLINE size] Task:IsCompletedMethod(int):bool
  [9 IL=0111 TR=000162 06000021] [profitable inline] AsyncValueTaskMethodBuilder`1:AwaitUnsafeOnCompleted(byref,byref):this
    [0 IL=0015 TR=000372 06003F91] [FAILED: has exception handling] AsyncTaskMethodBuilder`1:AwaitUnsafeOnCompleted(byref,byref):this
  [10 IL=0152 TR=000049 0600002C] [profitable inline] ValueTaskAwaiter`1:GetResult():int:this
    [0 IL=0014 TR=000403 0600002D] [FAILED: noinline per IL/cached result] ValueTaskAwaiter`1:GetResultFromTask():int:this
  [11 IL=0202 TR=000087 0600000E] [below ALWAYS_INLINE size] OrderBook:get_Orders():ref:this
  [12 IL=0207 TR=000094 060038F4] [below ALWAYS_INLINE size] List`1:get_Count():int:this
  [0 IL=0265 TR=000233 0600001D] [FAILED: within catch region] AsyncValueTaskMethodBuilder`1:SetException(ref):this
  [13 IL=0287 TR=000208 0600001B] [profitable inline] AsyncValueTaskMethodBuilder`1:SetResult(int):this
    [0 IL=0010 TR=000439 0600001C] [FAILED: noinline per IL/cached result] AsyncValueTaskMethodBuilder`1:SetResultFromTask(int):this
Budget: initialTime=939, finalTime=1183, initialBudget=9390, currentBudget=9402
Budget: increased by 12 because of force inlines
Budget: initialSize=6811, finalSize=7526
; Assembly listing for method <GetTotalWorthValueTaskInt32AwaitAsync>d__17:MoveNext():this
; Emitting BLENDED_CODE for X64 CPU with AVX
; optimized code
; rbp based frame
; fully interruptible
; Final local variable assignments
;
;  V00 this         [V00,T02] ( 27,435   )   byref  ->  [rbp+0x10]   do-not-enreg[H] this
;  V01 loc0         [V01,T20] (  2,  2   )     int  ->  rcx        
;  V02 loc1         [V02,T07] (  4,161   )     ref  ->  rbx         class-hnd
;  V03 loc2         [V03,T19] (  3,  3   )     int  ->  r14        
;  V04 loc3         [V04,T12] (  4, 82   )     int  ->  r14        
;  V05 loc4         [V05,T03] (  5,224   )     int  ->  r15        
;  V06 loc5         [V06    ] ( 10,155   )  struct (16) [rbp-0x38]   do-not-enreg[XSFB] must-init addr-exposed ld-addr-op
;  V07 loc6         [V07    ] (  4, 64   )  struct (16) [rbp-0x48]   do-not-enreg[XS] must-init addr-exposed ld-addr-op
;* V08 loc7         [V08    ] (  0,  0   )     ref  ->  zero-ref    class-hnd
;* V09 tmp0         [V09    ] (  0,  0   )     int  ->  zero-ref   
;* V10 tmp1         [V10    ] (  0,  0   )     int  ->  zero-ref   
;  V11 tmp2         [V11,T01] (  2,512   )     int  ->  rcx        
;  V12 tmp3         [V12,T21] (  2,  0   )     ref  ->  rdx         class-hnd
;  V13 tmp4         [V13,T06] (  6,192   )     int  ->  rdx        
;  V14 tmp5         [V14,T08] (  5,160   )     ref  ->  rcx         class-hnd
;  V15 tmp6         [V15    ] (  3, 96   )  struct (16) [rbp-0x58]   do-not-enreg[XSFB] must-init addr-exposed
;* V16 tmp7         [V16    ] (  0,  0   )  struct (16) zero-ref    ld-addr-op
;  V17 tmp8         [V17,T04] (  3,208   )    bool  ->  rax        
;  V18 tmp9         [V18,T13] (  2, 64   )     ref  ->  rax         ld-addr-op class-hnd
;  V19 tmp10        [V19,T15] (  2, 32   )     int  ->  rax        
;  V20 tmp11        [V20,T18] (  4,  8   )   byref  ->  rcx        
;  V21 tmp12        [V21,T11] (  3, 88   )     int  ->  rax        
;  V22 tmp13        [V22,T00] (  2,512   )     ref  ->  rdx         class-hnd
;  V23 tmp14        [V23,T17] (  5, 10   )   byref  ->  rcx        
;  V24 tmp15        [V24    ] (  3, 32   )     ref  ->  [rbp-0x48]   do-not-enreg[X] addr-exposed V07._task(offs=0x00) P-DEP
;  V25 tmp16        [V25    ] (  3, 32   )     int  ->  [rbp-0x40]   do-not-enreg[X] addr-exposed V07._result(offs=0x08) P-DEP
;  V26 tmp17        [V26,T14] (  2, 32   )     ref  ->  rax         V16._task(offs=0x00) P-INDEP
;  V27 tmp18        [V27,T16] (  2, 32   )     int  ->  rdx         V16._result(offs=0x08) P-INDEP
;  V28 tmp19        [V28,T05] (  6,192   )     ref  ->  rcx        
;  V29 tmp20        [V29,T09] (  4,128   )     ref  ->  rcx        
;  V30 tmp21        [V30,T10] (  3, 96   )   byref  ->  rcx         stack-byref
;  V31 OutArgs      [V31    ] (  1,  1   )  lclBlk (32) [rsp+0x00]  
;  V32 PSPSym       [V32    ] (  1,  1   )    long  ->  [rbp-0x60]   do-not-enreg[X] addr-exposed
;* V33 tmp24        [V33    ] (  0,  0   )   byref  ->  zero-ref   
;
; Lcl frame size = 88

G_M63166_IG01:
       55                   push     rbp
       4157                 push     r15
       4156                 push     r14
       57                   push     rdi
       56                   push     rsi
       53                   push     rbx
       4883EC58             sub      rsp, 88
       C5F877               vzeroupper 
       488DAC2480000000     lea      rbp, [rsp+80H]
       488BF1               mov      rsi, rcx
       488D7DA8             lea      rdi, [rbp-58H]
       B90C000000           mov      ecx, 12
       33C0                 xor      rax, rax
       F3AB                 rep stosd 
       488BCE               mov      rcx, rsi
       488965A0             mov      qword ptr [rbp-60H], rsp
       48894D10             mov      bword ptr [rbp+10H], rcx

G_M63166_IG02:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       8B4908               mov      ecx, dword ptr [rcx+8]
       488B4510             mov      rax, bword ptr [rbp+10H]
       488B18               mov      rbx, gword ptr [rax]

G_M63166_IG03:
       85C9                 test     ecx, ecx
       0F84F0000000         je       G_M63166_IG14
       4533F6               xor      r14d, r14d
       E968010000           jmp      G_M63166_IG22

G_M63166_IG04:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       33D2                 xor      edx, edx
       89510C               mov      dword ptr [rcx+12], edx
       E946010000           jmp      G_M63166_IG21

G_M63166_IG05:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       44897114             mov      dword ptr [rcx+20], r14d
       488B4B08             mov      rcx, gword ptr [rbx+8]
       488B5510             mov      rdx, bword ptr [rbp+10H]
       8B520C               mov      edx, dword ptr [rdx+12]
       3B5118               cmp      edx, dword ptr [rcx+24]
       0F835C010000         jae      G_M63166_IG23

G_M63166_IG06:
       488B4908             mov      rcx, gword ptr [rcx+8]
       3B5108               cmp      edx, dword ptr [rcx+8]
       0F83A5000000         jae      G_M63166_IG12
       4863D2               movsxd   rdx, edx
       488B4CD110           mov      rcx, gword ptr [rcx+8*rdx+16]
       488D55B8             lea      rdx, bword ptr [rbp-48H]
       3909                 cmp      dword ptr [rcx], ecx
       E8E6B3FFFF           call     Order:GetOrderWorthValueTaskInt32AwaitAsync():struct:this
       488D45A8             lea      rax, bword ptr [rbp-58H]

G_M63166_IG07:
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F00           vmovdqu  qword ptr [rax], xmm0

G_M63166_IG08:
       488B45B8             mov      rax, gword ptr [rbp-48H]
       8B55C0               mov      edx, dword ptr [rbp-40H]
       488D4DA8             lea      rcx, bword ptr [rbp-58H]
       488901               mov      gword ptr [rcx], rax
       895108               mov      dword ptr [rcx+8], edx

G_M63166_IG09:
       C4E17A6F45A8         vmovdqu  xmm0, qword ptr [rbp-58H]
       C4E17A7F45C8         vmovdqu  qword ptr [rbp-38H], xmm0

G_M63166_IG10:
       48837DC800           cmp      gword ptr [rbp-38H], 0
       7465                 je       SHORT G_M63166_IG13
       488B45C8             mov      rax, gword ptr [rbp-38H]
       8B4034               mov      eax, dword ptr [rax+52]
       A900006001           test     eax, 0x1600000
       0F95C0               setne    al
       0FB6C0               movzx    rax, al

G_M63166_IG11:
       85C0                 test     eax, eax
       0F8582000000         jne      G_M63166_IG17
       488B4510             mov      rax, bword ptr [rbp+10H]
       33D2                 xor      edx, edx
       895008               mov      dword ptr [rax+8], edx
       488B4510             mov      rax, bword ptr [rbp+10H]
       488D7838             lea      rdi, bword ptr [rax+56]
       488D75C8             lea      rsi, bword ptr [rbp-38H]
       E85009D05F           call     CORINFO_HELP_ASSIGN_BYREF
       48A5                 movsq    
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       3909                 cmp      dword ptr [rcx], ecx
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4883C118             add      rcx, 24
       C6410501             mov      byte  ptr [rcx+5], 1
       3909                 cmp      dword ptr [rcx], ecx
       4883C108             add      rcx, 8
       488D55C8             lea      rdx, bword ptr [rbp-38H]
       4C8B4510             mov      r8, bword ptr [rbp+10H]
       E849E6FFFF           call     AsyncTaskMethodBuilder`1:AwaitUnsafeOnCompleted(byref,byref):this
       E9E4000000           jmp      G_M63166_IG27

G_M63166_IG12:
       E8779FAC5F           call     CORINFO_HELP_RNGCHKFAIL

G_M63166_IG13:
       B801000000           mov      eax, 1
       EBA6                 jmp      SHORT G_M63166_IG11

G_M63166_IG14:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4883C138             add      rcx, 56

G_M63166_IG15:
       C4E17A6F01           vmovdqu  xmm0, qword ptr [rcx]
       C4E17A7F45C8         vmovdqu  qword ptr [rbp-38H], xmm0

G_M63166_IG16:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4883C138             add      rcx, 56
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F01           vmovdqu  qword ptr [rcx], xmm0
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       C74108FFFFFFFF       mov      dword ptr [rcx+8], -1

G_M63166_IG17:
       48837DC800           cmp      gword ptr [rbp-38H], 0
       746C                 je       SHORT G_M63166_IG24
       488D4DC8             lea      rcx, bword ptr [rbp-38H]
       E8C8AFFFFF           call     ValueTaskAwaiter`1:GetResultFromTask():int:this

G_M63166_IG18:
       488D4DC8             lea      rcx, bword ptr [rbp-38H]

G_M63166_IG19:
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F01           vmovdqu  qword ptr [rcx], xmm0

G_M63166_IG20:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       448BF0               mov      r14d, eax
       44037114             add      r14d, dword ptr [rcx+20]
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       448B790C             mov      r15d, dword ptr [rcx+12]
       41FFC7               inc      r15d
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4489790C             mov      dword ptr [rcx+12], r15d

G_M63166_IG21:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       8B490C               mov      ecx, dword ptr [rcx+12]
       488B5308             mov      rdx, gword ptr [rbx+8]
       3B4A18               cmp      ecx, dword ptr [rdx+24]
       0F8CA6FEFFFF         jl       G_M63166_IG05

G_M63166_IG22:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       448B7910             mov      r15d, dword ptr [rcx+16]
       418D4FFF             lea      ecx, [r15-1]
       488B5510             mov      rdx, bword ptr [rbp+10H]
       894A10               mov      dword ptr [rdx+16], ecx
       4585FF               test     r15d, r15d
       0F8F7CFEFFFF         jg       G_M63166_IG04
       EB0A                 jmp      SHORT G_M63166_IG25

G_M63166_IG23:
       E81591E146           call     ThrowHelper:ThrowArgumentOutOfRange_IndexException()

G_M63166_IG24:
       8B45D0               mov      eax, dword ptr [rbp-30H]
       EB98                 jmp      SHORT G_M63166_IG18

G_M63166_IG25:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       C74108FEFFFFFF       mov      dword ptr [rcx+8], -2
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       3909                 cmp      dword ptr [rcx], ecx
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4883C118             add      rcx, 24
       80790500             cmp      byte  ptr [rcx+5], 0
       740A                 je       SHORT G_M63166_IG26
       418BD6               mov      edx, r14d
       E8F9E3FFFF           call     AsyncValueTaskMethodBuilder`1:SetResultFromTask(int):this
       EB07                 jmp      SHORT G_M63166_IG27

G_M63166_IG26:
       448931               mov      dword ptr [rcx], r14d
       C6410401             mov      byte  ptr [rcx+4], 1

G_M63166_IG27:
       488D65D8             lea      rsp, [rbp-28H]
       5B                   pop      rbx
       5E                   pop      rsi
       5F                   pop      rdi
       415E                 pop      r14
       415F                 pop      r15
       5D                   pop      rbp
       C3                   ret      

G_M63166_IG28:
       55                   push     rbp
       4157                 push     r15
       4156                 push     r14
       57                   push     rdi
       56                   push     rsi
       53                   push     rbx
       4883EC28             sub      rsp, 40
       C5F877               vzeroupper 
       488B6920             mov      rbp, qword ptr [rcx+32]
       48896C2420           mov      qword ptr [rsp+20H], rbp
       488DAD80000000       lea      rbp, [rbp+80H]

G_M63166_IG29:
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       C74108FEFFFFFF       mov      dword ptr [rcx+8], -2
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       3909                 cmp      dword ptr [rcx], ecx
       488B4D10             mov      rcx, bword ptr [rbp+10H]
       4883C118             add      rcx, 24
       E8AEE3FFFF           call     AsyncValueTaskMethodBuilder`1:SetException(ref):this
       488D05AFFFFFFF       lea      rax, G_M63166_IG27

G_M63166_IG30:
       4883C428             add      rsp, 40
       5B                   pop      rbx
       5E                   pop      rsi
       5F                   pop      rdi
       415E                 pop      r14
       415F                 pop      r15
       5D                   pop      rbp
       C3                   ret      

; Total bytes of code 622, prolog size 50 for method <GetTotalWorthValueTaskInt32AwaitAsync>d__17:MoveNext():this
; ============================================================

[benaadams]

C# for above

internal async ValueTask<int> GetTotalWorthValueTaskInt32AwaitAsync(int repeats)
{
    int total = 0;
    while (repeats-- > 0)
    {
        for (int i = 0; i < Orders.Count; i++)
        {
            total += await Orders[i].GetOrderWorthValueTaskInt32AwaitAsync(); // Maually Marked No-Inline
        }
    }
    return total;
}

[benaadams]

Adding a manual split with an inner function brings it more or less on par with the sync version (10-fold increase)

internal Task<int> GetTotalWorthTaskInt32ManualCompletedAsync(int repeats)
{
    // Main function
    {
        int total = 0;
        while (repeats-- > 0)
        {
            for (int i = 0; i < Orders.Count; i++)
            {
                var task = Orders[i].GetOrderWorthTaskInt32ManualCompletedAsync();
                if (!task.IsCompleted) return Awaited(task, total, i, repeats);
                total += task.Result;
            }
        }
        return Task.FromResult(total);
    }
    // Inner async function
    async Task<int> Awaited(Task<int> task, int total, int i, int r)
    {
        total += await task;
        while (++i < Orders.Count) // finish the inner loop
        {
            total += await Orders[i].GetOrderWorthTaskInt32ManualCompletedAsync();
        }
        while (r-- > 0) // finish the outer loop
        {
            for (i = 0; i < Orders.Count; i++)
            {
                total += await Orders[i].GetOrderWorthTaskInt32ManualCompletedAsync();
            }
        }
        return total;
    }
}

[benaadams]

With the following asm

Successfully inlined OrderBook:get_Orders():ref:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined List`1:get_Item(int):ref:this (27 IL bytes) (depth 1) [profitable inline]
Successfully inlined ValueTask`1:get_IsCompleted():bool:this (22 IL bytes) (depth 1) [profitable inline]
Successfully inlined Task:get_IsCompleted():bool:this (16 IL bytes) (depth 2) [below ALWAYS_INLINE size]
Successfully inlined Task:IsCompletedMethod(int):bool (11 IL bytes) (depth 3) [below ALWAYS_INLINE size]
Successfully inlined ValueTask`1:get_Result():int:this (22 IL bytes) (depth 1) [profitable inline]
Successfully inlined OrderBook:get_Orders():ref:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined List`1:get_Count():int:this (7 IL bytes) (depth 1) [below ALWAYS_INLINE size]
Successfully inlined ValueTask`1:.ctor(int):this (15 IL bytes) (depth 1) [aggressive inline attribute]
**************** Inline Tree
Inlines into 06000020 OrderBook:GetTotalWorthValueTaskInt32ManualCompletedAsync(int):struct:this
  [1 IL=0009 TR=000040 0600000E] [below ALWAYS_INLINE size] OrderBook:get_Orders():ref:this
  [2 IL=0015 TR=000044 060038FA] [profitable inline] List`1:get_Item(int):ref:this
    [0 IL=0009 TR=000134 060001EE] [FAILED: does not return] ThrowHelper:ThrowArgumentOutOfRange_IndexException()
  [0 IL=0020 TR=000048 06000038] [FAILED: noinline per IL/cached result] Order:GetOrderWorthValueTaskInt32ManualCompletedAsync():struct:this
  [3 IL=0028 TR=000055 06000009] [profitable inline] ValueTask`1:get_IsCompleted():bool:this
    [4 IL=0014 TR=000159 06002291] [below ALWAYS_INLINE size] Task:get_IsCompleted():bool:this
      [5 IL=0010 TR=000175 06002292] [below ALWAYS_INLINE size] Task:IsCompletedMethod(int):bool
  [0 IL=0040 TR=000085 06000025] [FAILED: unprofitable inline] OrderBook:<GetTotalWorthValueTaskInt32ManualCompletedAsync>g__Awaited20_0(struct,int,int,int):struct:this
  [6 IL=0049 TR=000066 0600000E] [profitable inline] ValueTask`1:get_Result():int:this
    [0 IL=0009 TR=000214 0600000F] [FAILED: noinline per IL/cached result] ValueTask`1:GetResultFromTask():int:this
  [7 IL=0062 TR=000029 0600000E] [below ALWAYS_INLINE size] OrderBook:get_Orders():ref:this
  [8 IL=0067 TR=000032 060038F4] [below ALWAYS_INLINE size] List`1:get_Count():int:this
  [9 IL=0084 TR=000108 06000001] [aggressive inline attribute] ValueTask`1:.ctor(int):this
Budget: initialTime=330, finalTime=472, initialBudget=3300, currentBudget=3316
Budget: increased by 16 because of force inlines
Budget: initialSize=2183, finalSize=2546
; Assembly listing for method OrderBook:GetTotalWorthValueTaskInt32ManualCompletedAsync(int):struct:this
; Emitting BLENDED_CODE for X64 CPU with AVX
; optimized code
; rsp based frame
; fully interruptible
; Final local variable assignments
;
;  V00 this         [V00,T06] (  6, 34.50)     ref  ->  rsi         this class-hnd
;  V01 RetBuf       [V01,T13] (  7,  4.50)   byref  ->  rbx        
;  V02 arg1         [V02,T10] (  5, 18.50)     int  ->  registers  
;  V03 loc0         [V03,T11] (  5, 18   )     int  ->  rbp        
;  V04 loc1         [V04,T02] ( 10, 74.50)     int  ->  registers  
;  V05 loc2         [V05    ] (  9, 57   )  struct (16) [rsp+0x40]   do-not-enreg[XS] must-init addr-exposed ld-addr-op
;  V06 loc3         [V06,T05] (  3, 48   )     int  ->  rdi        
;* V07 tmp1         [V07    ] (  0,  0   )  struct (16) zero-ref   
;  V08 tmp2         [V08,T01] (  5, 80   )     ref  ->  rcx         class-hnd
;  V09 tmp3         [V09,T08] (  2, 16   )    bool  ->  rcx        
;  V10 tmp4         [V10,T07] (  2, 32   )     ref  ->  rcx         ld-addr-op class-hnd
;  V11 tmp5         [V11,T12] (  2, 16   )     int  ->  rcx        
;  V12 tmp6         [V12,T09] (  3, 24   )     int  ->  rax        
;  V13 tmp7         [V13,T04] (  2, 64   )     ref  ->  rcx         class-hnd
;  V14 tmp8         [V14    ] (  7, 40.50)     ref  ->  [rsp+0x40]   do-not-enreg[X] addr-exposed V05._task(offs=0x00) P-DEP
;  V15 tmp9         [V15    ] (  5, 24.50)     int  ->  [rsp+0x48]   do-not-enreg[X] addr-exposed V05._result(offs=0x08) P-DEP
;  V16 tmp10        [V16,T15] (  2,  1   )     ref  ->  rax         V07._task(offs=0x00) P-INDEP
;  V17 tmp11        [V17,T16] (  2,  1   )     int  ->  rbp         V07._result(offs=0x08) P-INDEP
;  V18 tmp12        [V18,T00] (  6, 96   )     ref  ->  rcx        
;  V19 tmp13        [V19,T03] (  4, 64   )     ref  ->  rcx        
;  V20 tmp14        [V20    ] (  3,  4   )  struct (16) [rsp+0x30]   do-not-enreg[XSB] must-init addr-exposed
;  V21 tmp15        [V21,T14] (  3,  3   )   byref  ->  rcx         stack-byref
;* V22 tmp16        [V22,T17] (  0,  0   )     ref  ->  zero-ref   
;* V23 tmp17        [V23,T18] (  0,  0   )   byref  ->  zero-ref   
;  V24 OutArgs      [V24    ] (  1,  1   )  lclBlk (48) [rsp+0x00]  
;
; Lcl frame size = 80

G_M36646_IG01:
       4156                 push     r14
       57                   push     rdi
       56                   push     rsi
       55                   push     rbp
       53                   push     rbx
       4883EC50             sub      rsp, 80
       488BF1               mov      rsi, rcx
       488D7C2430           lea      rdi, [rsp+30H]
       B908000000           mov      ecx, 8
       33C0                 xor      rax, rax
       F3AB                 rep stosd 
       488BCE               mov      rcx, rsi
       488BF1               mov      rsi, rcx
       488BDA               mov      rbx, rdx
       418BF8               mov      edi, r8d

G_M36646_IG02:
       33ED                 xor      ebp, ebp
       E9CD000000           jmp      G_M36646_IG11

G_M36646_IG03:
       33FF                 xor      edi, edi
       E9B6000000           jmp      G_M36646_IG10

G_M36646_IG04:
       4987FE               xchg     rdi, r14
       488B4E08             mov      rcx, gword ptr [rsi+8]
       443B7118             cmp      r14d, dword ptr [rcx+24]
       0F83D9000000         jae      G_M36646_IG13

G_M36646_IG05:
       488B4908             mov      rcx, gword ptr [rcx+8]
       443B7108             cmp      r14d, dword ptr [rcx+8]
       0F83D1000000         jae      G_M36646_IG14
       4963D6               movsxd   rdx, r14d
       488B4CD110           mov      rcx, gword ptr [rcx+8*rdx+16]
       488D542440           lea      rdx, bword ptr [rsp+40H]
       3909                 cmp      dword ptr [rcx], ecx
       E880ADFFFF           call     Order:GetOrderWorthValueTaskInt32ManualCompletedAsync():struct:this
       488B4C2440           mov      rcx, gword ptr [rsp+40H]
       4885C9               test     rcx, rcx
       7457                 je       SHORT G_M36646_IG07
       488B4C2440           mov      rcx, gword ptr [rsp+40H]
       8B4934               mov      ecx, dword ptr [rcx+52]
       F7C100006001         test     ecx, 0x1600000
       0F95C1               setne    cl
       0FB6C9               movzx    rcx, cl
       85C9                 test     ecx, ecx
       753F                 jne      SHORT G_M36646_IG07
       488D4C2430           lea      rcx, bword ptr [rsp+30H]
       488B542440           mov      rdx, gword ptr [rsp+40H]
       488911               mov      gword ptr [rcx], rdx
       8B542448             mov      edx, dword ptr [rsp+48H]
       895108               mov      dword ptr [rcx+8], edx
       4489742420           mov      dword ptr [rsp+20H], r14d
       897C2428             mov      dword ptr [rsp+28H], edi
       488BCE               mov      rcx, rsi
       488BD3               mov      rdx, rbx
       4C8D442430           lea      r8, bword ptr [rsp+30H]
       448BCD               mov      r9d, ebp
       E83EA7FFFF           call     OrderBook:<GetTotalWorthValueTaskInt32ManualCompletedAsync>g__Awaited20_0(struct,int,int,int):struct:this
       488BC3               mov      rax, rbx

G_M36646_IG06:
       4883C450             add      rsp, 80
       5B                   pop      rbx
       5D                   pop      rbp
       5E                   pop      rsi
       5F                   pop      rdi
       415E                 pop      r14
       C3                   ret      

G_M36646_IG07:
       488B4C2440           mov      rcx, gword ptr [rsp+40H]
       4885C9               test     rcx, rcx
       740C                 je       SHORT G_M36646_IG08
       488D4C2440           lea      rcx, bword ptr [rsp+40H]
       E814ACFFFF           call     ValueTask`1:GetResultFromTask():int:this
       EB04                 jmp      SHORT G_M36646_IG09

G_M36646_IG08:
       8B442448             mov      eax, dword ptr [rsp+48H]

G_M36646_IG09:
       03E8                 add      ebp, eax
       41FFC6               inc      r14d
       4987FE               xchg     rdi, r14

G_M36646_IG10:
       488B4E08             mov      rcx, gword ptr [rsi+8]
       3B7918               cmp      edi, dword ptr [rcx+24]
       0F8C3EFFFFFF         jl       G_M36646_IG04
       418BFE               mov      edi, r14d

G_M36646_IG11:
       8D4FFF               lea      ecx, [rdi-1]
       448BF1               mov      r14d, ecx
       85FF                 test     edi, edi
       0F8F26FFFFFF         jg       G_M36646_IG03
       33C0                 xor      rax, rax
       488903               mov      gword ptr [rbx], rax
       896B08               mov      dword ptr [rbx+8], ebp
       488BC3               mov      rax, rbx

G_M36646_IG12:
       4883C450             add      rsp, 80
       5B                   pop      rbx
       5D                   pop      rbp
       5E                   pop      rsi
       5F                   pop      rdi
       415E                 pop      r14
       C3                   ret      

G_M36646_IG13:
       E81D8BE146           call     ThrowHelper:ThrowArgumentOutOfRange_IndexException()
       CC                   int3     

G_M36646_IG14:
       E8CF98AC5F           call     CORINFO_HELP_RNGCHKFAIL
       CC                   int3     

; Total bytes of code 298, prolog size 39 for method OrderBook:GetTotalWorthValueTaskInt32ManualCompletedAsync(int):struct:this
; ============================================================

[benaadams]

Statemachine captures

[benaadams]

il for pre-start

  .method assembly hidebysig instance class [System.Threading.Tasks.Extensions]System.Threading.Tasks.ValueTask`1<int32> 
    GetOrderWorthValueTaskInt32AwaitAsync() cil managed 
  {
    .custom instance void [mscorlib]System.Runtime.CompilerServices.AsyncStateMachineAttribute::.ctor(class [mscorlib]System.Type) 
      = (
        01 00 32 4f 72 64 65 72 2b 3c 47 65 74 4f 72 64 // ..2Order+<GetOrd
        65 72 57 6f 72 74 68 56 61 6c 75 65 54 61 73 6b // erWorthValueTask
        49 6e 74 33 32 41 77 61 69 74 41 73 79 6e 63 3e // Int32AwaitAsync>
        64 5f 5f 31 36 00 00                            // d__16..
      )
      // MetadataClassType(Order+<GetOrderWorthValueTaskInt32AwaitAsync>d__16)
    .maxstack 2
    .locals init (
      [0] valuetype Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16' V_0,
      [1] class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32> V_1
    )

    IL_0000: ldloca.s     V_0
    IL_0002: ldarg.0      // this
    IL_0003: stfld        class Order Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'::'<>4__this'
    IL_0008: ldloca.s     V_0
    IL_000a: call         instance class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<!0/*int32*/> class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32>::Create()
    IL_000f: stfld        class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32> Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'::'<>t__builder'
    IL_0014: ldloca.s     V_0
    IL_0016: ldc.i4.m1    
    IL_0017: stfld        int32 Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'::'<>1__state'
    IL_001c: ldloc.0      // V_0
    IL_001d: ldfld        class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32> Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'::'<>t__builder'
    IL_0022: stloc.1      // V_1
    IL_0023: ldloca.s     V_1
    IL_0025: ldloca.s     V_0
    IL_0027: call         instance void class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32>::Start<valuetype Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'>(!!0/*valuetype Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'*/&)
    IL_002c: ldloca.s     V_0
    IL_002e: ldflda       class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32> Order/'<GetOrderWorthValueTaskInt32AwaitAsync>d__16'::'<>t__builder'
    IL_0033: call         instance class [System.Threading.Tasks.Extensions]System.Threading.Tasks.ValueTask`1<!0/*int32*/> class [System.Threading.Tasks.Extensions]System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1<int32>::get_Task()
    IL_0038: ret          

  } // end of method Order::GetOrderWorthValueTaskInt32AwaitAsync

and asm

Successfully inlined AsyncValueTaskMethodBuilder`1:Create():struct (10 IL bytes) (depth 1) [aggressive inline attribute]
Marking AsyncTaskMethodBuilder`1:Start(byref):this as NOINLINE because of has exception handling
Successfully inlined AsyncValueTaskMethodBuilder`1:Start(byref):this (13 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined AsyncValueTaskMethodBuilder`1:get_Task():struct:this (27 IL bytes) (depth 1) [aggressive inline attribute]
Successfully inlined ValueTask`1:.ctor(int):this (15 IL bytes) (depth 2) [aggressive inline attribute]
**************** Inline Tree
Inlines into 0600001D OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this
  [1 IL=0018 TR=000015 06000018] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:Create():struct
  [2 IL=0047 TR=000039 06000019] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:Start(byref):this
    [0 IL=0007 TR=000080 06003F96] [FAILED: has exception handling] AsyncTaskMethodBuilder`1:Start(byref):this
  [3 IL=0059 TR=000046 0600001E] [aggressive inline attribute] AsyncValueTaskMethodBuilder`1:get_Task():struct:this
    [0 IL=0009 TR=000123 0600001F] [FAILED: noinline per IL/cached result] AsyncValueTaskMethodBuilder`1:GetTaskFromMethodBuilder():struct:this
    [4 IL=0021 TR=000106 06000001] [aggressive inline attribute] ValueTask`1:.ctor(int):this
Budget: initialTime=255, finalTime=329, initialBudget=2550, currentBudget=2624
Budget: increased by 74 because of force inlines
Budget: initialSize=1613, finalSize=1613
; Assembly listing for method OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this
; Emitting BLENDED_CODE for X64 CPU with AVX
; optimized code
; rsp based frame
; partially interruptible
; Final local variable assignments
;
;  V00 this         [V00,T01] (  3,  3   )     ref  ->  rcx         this class-hnd
;  V01 RetBuf       [V01,T00] (  5,  5   )   byref  ->  rsi        
;  V02 arg1         [V02,T02] (  3,  3   )     int  ->   r8        
;  V03 loc0         [V03    ] (  9,  8   )  struct (72) [rsp+0x50]   do-not-enreg[XSFB] must-init addr-exposed ld-addr-op
;  V04 loc1         [V04    ] (  2,  2   )  struct (32) [rsp+0x30]   do-not-enreg[XSFB] must-init addr-exposed ld-addr-op
;* V05 loc2         [V05    ] (  0,  0   )  struct (32) zero-ref    do-not-enreg[SB] ld-addr-op
;  V06 tmp1         [V06    ] (  5,  3.50)  struct (16) [rsp+0x20]   do-not-enreg[XS] must-init addr-exposed
;* V07 tmp2         [V07    ] (  0,  0   )  struct (16) zero-ref   
;  V08 tmp3         [V08,T03] (  2,  2   )     int  ->  rdx         ld-addr-op
;  V09 tmp4         [V09    ] (  3,  2   )     ref  ->  [rsp+0x20]   do-not-enreg[X] addr-exposed V06._task(offs=0x00) P-DEP
;  V10 tmp5         [V10    ] (  3,  2   )     int  ->  [rsp+0x28]   do-not-enreg[X] addr-exposed V06._result(offs=0x08) P-DEP
;  V11 tmp6         [V11,T04] (  2,  1   )     ref  ->  rcx         V07._task(offs=0x00) P-INDEP
;  V12 tmp7         [V12,T05] (  2,  1   )     int  ->  rdx         V07._result(offs=0x08) P-INDEP
;  V13 OutArgs      [V13    ] (  1,  1   )  lclBlk (32) [rsp+0x00]  
;
; Lcl frame size = 152

G_M45285_IG01:
       57                   push     rdi
       56                   push     rsi
       4881EC98000000       sub      rsp, 152
       C5F877               vzeroupper 
       488BF1               mov      rsi, rcx
       488D7C2420           lea      rdi, [rsp+20H]
       B91E000000           mov      ecx, 30
       33C0                 xor      rax, rax
       F3AB                 rep stosd 
       488BCE               mov      rcx, rsi
       488BF2               mov      rsi, rdx

G_M45285_IG02:
       48894C2450           mov      gword ptr [rsp+50H], rcx
       4489442460           mov      dword ptr [rsp+60H], r8d
       488D4C2468           lea      rcx, bword ptr [rsp+68H]
       C4E17957C0           vxorpd   xmm0, xmm0
       C4E17A7F01           vmovdqu  qword ptr [rcx], xmm0
       C4E17A7F4110         vmovdqu  qword ptr [rcx+16], xmm0
       C7442458FFFFFFFF     mov      dword ptr [rsp+58H], -1

G_M45285_IG03:
       C4E17A6F442468       vmovdqu  xmm0, qword ptr [rsp+68H]
       C4E17A7F442430       vmovdqu  qword ptr [rsp+30H], xmm0
       C4E17A6F442478       vmovdqu  xmm0, qword ptr [rsp+78H]
       C4E17A7F442440       vmovdqu  qword ptr [rsp+40H], xmm0

G_M45285_IG04:
       488D4C2438           lea      rcx, bword ptr [rsp+38H]
       488D542450           lea      rdx, bword ptr [rsp+50H]
       E88BE6FFFF           call     AsyncTaskMethodBuilder`1:Start(byref):this
       807C246C00           cmp      byte  ptr [rsp+6CH], 0
       7511                 jne      SHORT G_M45285_IG05
       488D4C2468           lea      rcx, bword ptr [rsp+68H]
       488D542420           lea      rdx, bword ptr [rsp+20H]
       E81DE6FFFF           call     AsyncValueTaskMethodBuilder`1:GetTaskFromMethodBuilder():struct:this
       EB0F                 jmp      SHORT G_M45285_IG06

G_M45285_IG05:
       8B542468             mov      edx, dword ptr [rsp+68H]
       33C9                 xor      rcx, rcx
       48894C2420           mov      gword ptr [rsp+20H], rcx
       89542428             mov      dword ptr [rsp+28H], edx

G_M45285_IG06:
       488D0E               lea      rcx, bword ptr [rsi]
       488B542420           mov      rdx, gword ptr [rsp+20H]
       E8D708D05F           call     CORINFO_HELP_CHECKED_ASSIGN_REF
       8B442428             mov      eax, dword ptr [rsp+28H]
       894608               mov      dword ptr [rsi+8], eax
       488BC6               mov      rax, rsi

G_M45285_IG07:
       4881C498000000       add      rsp, 152
       5E                   pop      rsi
       5F                   pop      rdi
       C3                   ret      

; Total bytes of code 189, prolog size 32 for method OrderBook:GetTotalWorthValueTaskInt32AwaitAsync(int):struct:this

[vancem]

I wanted to comment on the original question (about the try-finally in AsyncMethodBuilder inhibiting inlineing). (I realize there is this other issue with quick checks that seems to be much bigger (but also roughly unrelated), My comment only applies if after resolving that we still care about the question of inlining AsyncMethodBuilder.

While solving it by allowing the JIT to inline things with try-finally, I must admit that for me this is not a second best solution. Handling complexity is great, but even better is achieving the same value without the complexity. It seems to me that this boils down to what the contract is between AsyncMethodBuilder and the state machine. By simply stating that exceptions can't escape, we can avoid the need of the try-finally, and things get simpler. From what Stephen indicates, this seems OK (even custom state machines could achieve this without losing interestingly flexibility). I think that should be one the table (which is basically what Ben's original question was). I think that should be the proposal to beat (we should work through any details to get comfortable with it).

[benaadams]

ValueTask's custom builder uses AMB to achieve the correct EC semantics; which then the C# compiler builds its tasklike statemachine with; when again the finally is irrelevant.

Issue would arise if the statemachine was manually built (rather than via compiler so MoveNext didn't always catch) and AMB was used to achieve some behvaiour. However, you'd have to be violating the async contract - where failures are returned via failed tasks rather than directly throwing methods.

While the removal was obvious to me at the start; I now have a slight concern as @stephentoub highlights the SecurityContext.

People can and do store the current user in an AsyncLocal by adding HttpContextAccessor in ASP.NET and then using HttpContextAccessor.HttpContext.User.Identity. (Do they not perf? 😢)

Also people (accidentally) write async violating Async methods by throwing direct exceptions rather than rather than returning failed tasks. e.g.

Task NonAsyncMyAsync(int index)
{
    if (index < 0)
    {
        throw new NakedException();
    }
    return RealAsync();
}

So I wonder if that could fall off the statemachine rails at some point? Though any true await should wrap that regardless in the compiler's catch?

[benaadams]

Actually the Threadpool handling of EC should always guard against that.

Also user code is always wrapped by a compiler generated statemachine via an await.