Mullapudi2016 experimental GPU scheduling support: the tech debt

[antonysigma]

• GPU autoscheduling with Mullapudi2016: the reference implementation #7787
• Enable experimental GPU features for the pipelines in apps/*/; manual tuning of the last_level_cache_size estimates for complex pipelines. Enable experimental Mullapudi2016 GPU scheduler for test-bench #8650
• Mullapudi2016-GPU: Reorder to avoid for-loops to be sandwiched between gpu_blocks. #8647
• Intercept the split operation in Partitioner::split_dim. Make it a deferred operation by GPUTilingDedup::reorder. Mullapudi2016-GPU: Reorder to avoid for-loops to be sandwiched between gpu_blocks. #8647 (comment)
• (Optional) Resolve the apps/iir_blur/ error having vectorized width larger than the channel count (=3). Always ensure gpu_threads count >= warp size of 32 #8656
• For target darwin-64-metal, investigate why a gpu_tile(..., {32, 32, 1}) exceeds the maximum thread count per block. Enable experimental Mullapudi2016 GPU scheduler for test-bench #8650 (comment) https://developer.apple.com/documentation/metal/calculating-threadgroup-and-grid-sizes?utm_source=chatgpt.com#Calculate-Threads-per-Threadgroup
• Write manual GPU schedules for the test cases at test/autoscheduler/mullapudi2016/*.cpp to pass performance regression tests; GPU autoscheduling with Mullapudi2016: the reference implementation #7787 (comment)
• (Optional) Resolve other unknown issues not anticipated by the original Mullapudi paper, e.g. CodeGen_NVPTX does not support more than 4 gpu_blocks; needing fuse() to fit the 4 gpu block limit imposed by the hardware.

cc'ed @mcourteaux for context.

[antonysigma]

To those who are interested: the so-called reordering code generation bug is triggered whenever a for loop is sandwiched between gpu block loops, such as:

Error: for loop over r$1.s1.dom$x is inside GPU block loop over r$1.s1.y.y_o but outside GPU block loop over r$1.s1.x.x_vo. Funcs cannot be scheduled in between GPU block loops. The loop nest is:
compute_at for r$1:
 for r$1.s1.dom$y:
  gpu_block r$1.s1.c.c_o:
   gpu_block r$1.s1.y.y_o:
    for r$1.s1.dom$x:
     gpu_block r$1.s1.x.x_vo:

Test case: https://github.com/halide/Halide/blob/main/test/autoschedulers/mullapudi2016/reorder.cpp#L124