Running a System.Diagnostics.Process with both async stdout/stderr redirection blocks a ThreadPool thread

[drauch]

Description

If you run a System.Diagnostics.Process and redirect both stdout and stderr in an async way (using the BeginOutputReadLine/BeginErrorReadLine methods) it blocks a thread pool thread.

Reproduction Steps

The following C# interactive notebook:

using System.Diagnostics;
using System.Threading;

bool stop = false;
var threadPoolMonitoringThread = new Thread(() => { while (!stop) { Console.WriteLine($"{DateTime.Now}: ThreadCount: {ThreadPool.ThreadCount}, PendingWorkItemCount: {ThreadPool.PendingWorkItemCount}"); Thread.Sleep(1000); } });
threadPoolMonitoringThread.Start();

var tasks = Enumerable.Range(0, 10).Select(async _ => {
  using (Process process = new Process())
    {
      process.StartInfo.FileName = "pwsh.exe";
      process.StartInfo.Arguments = "-Command Start-Sleep -Seconds 10";
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = true;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;
      
      process.OutputDataReceived += (s, a) => { };
      process.ErrorDataReceived += (s, a) => { };

      process.Start();    

      process.BeginOutputReadLine();      
      process.BeginErrorReadLine();

      await process.WaitForExitAsync();      
    }
});

await Task.WhenAll(tasks);
stop = true;

reproduces the problem. You can immediately see the PendingWorkItemCount raising to very high numbers, while actually all the WaitForExitAsync calls should not block threads on the thread pool at all.

If you either remove the BeginOutputReadLine or the BeginErrorReadLine call the PendingWorkItemCount stays 0 as expected. So it must have something to do with redirecting both streams.

Expected behavior

The thread pool should not be filled by 10 parallel async calls.

Actual behavior

The thread pool is filled up, because each call blocks a thread pool thread.

Regression?

No response

Known Workarounds

No response

Configuration

Reproduced on .NET 6 and .NET 7

Other information

I'm pretty confident that I'm using the Process API correctly, also various other Process-class-Wrappers like MedallionShell or CliWrap are having the same problem.

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

If you run a System.Diagnostics.Process and redirect both stdout and stderr in an async way (using the BeginOutputReadLine/BeginErrorReadLine methods) blocks a thread pool thread.

Reproduction Steps

The following C# interactive notebook:

using System.Diagnostics;
using System.Threading;

bool stop = false;
var threadPoolMonitoringThread = new Thread(() => { while (!stop) { Console.WriteLine($"{DateTime.Now}: ThreadCount: {ThreadPool.ThreadCount}, PendingWorkItemCount: {ThreadPool.PendingWorkItemCount}"); Thread.Sleep(1000); } });
threadPoolMonitoringThread.Start();

var tasks = Enumerable.Range(0, 10).Select(async _ => {
  using (Process process = new Process())
    {
      process.StartInfo.FileName = "pwsh.exe";
      process.StartInfo.Arguments = "-Command Start-Sleep -Seconds 10";
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = true;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;
      
      process.OutputDataReceived += (s, a) => { };
      process.ErrorDataReceived += (s, a) => { };

      process.Start();    

      process.BeginOutputReadLine();      
      process.BeginErrorReadLine();

      await process.WaitForExitAsync();      
    }
});

await Task.WhenAll(tasks);
stop = true;

reproduces the problem. You can immediately see the PendingWorkItemCount raising to very high numbers, while actually all the WaitForExitAsync blocks should not block threads on the thread pool at all.

If you either remove the BeginOutputReadLine or the BeginErrorReadLine call the PendingWorkItemCount stays 0 as expected.

Expected behavior

The thread pool should not be filled by 10 parallel async calls.

Actual behavior

The thread pool is filled up, because each call blocks a thread pool thread.

Regression?

No response

Known Workarounds

No response

Configuration

Reproduced on .NET 6 and .NET 7

Other information

I'm pretty confident that I'm using the Process API correctly, also various other Process-class-Wrappers like MedallionShell are having the same problem.

Author:	drauch
Assignees:	-
Labels:	area-System.Diagnostics.Process
Milestone:	-

[drauch]

Here is a Dropbox link to the Polyglot Notebook I've posted above:
https://www.dropbox.com/s/qjnuwmv7zu1306q/Repro81896.ipynb?dl=0

[stephentoub]

If you look in the Parallel Stacks window in VS, you'll see something like:

That's because Process is creating anonymous pipes for the child process's stdout/stderr, and anonymous pipes on Windows don't support overlapped I/O. As such, all reads are blocking.

[drauch]

Thank you for your quick reply.

That's because Process is creating anonymous pipes for the child process's stdout/stderr, and anonymous pipes on Windows don't support overlapped I/O. As such, all reads are blocking.

I understand, is there a suitable workaround? We want to run a process as part of an ASP.NET Core request and blocking thread pool threads is a no-go in that area.

Requirements:

• Get live stdout/stderr output from the process
• Being able to set a maximum timeout for the process

Would really appreciate it :-)

Best regards,
D.R.

[stephentoub]

is there a suitable workaround?

My expectation is that you wouldn't hit such an issue on Linux, so if deploying on Linux is an option, you could try that.

Other than that, I can't think of any workarounds.

Someone could experiment with using something other than anonymous pipes, e.g. maybe named pipes could be used instead (e.g. with the server pipe created to expect overlapped I/O). I don't know what the ramifications of that would be for launched processes, though, and whether such a difference would be observable in an impactful way.

[drauch]

We're stuck on Windows unfortunately. Thanks for trying anyways.

[fschmied]

So, if we want to avoid ThreadPool threads from being blocked, the best thing we can do is to avoid the Begin... methods and instead use something like this, right?

var t1 = Task.Factory.StartNew (() => process.StandardOutput.ReadToEnd(), TaskCreationOptions.LongRunning);
var t2 = Task.Factory.StartNew (() => process.StandardError.ReadToEnd(), TaskCreationOptions.LongRunning);
var t3 = process.WaitForExitAsync();

await Task.WhenAll(t1, t2, t3);

[fschmied]

Should this be documented on the Process class docs? (https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.process?view=net-7.0)

[stephentoub]

the best thing we can do is to avoid the Begin... methods and instead use something like this, right?

To my knowledge that's currently the way you'd avoid blocking pool threads.

[davidfowl]

I understand, is there a suitable workaround? We want to run a process as part of an ASP.NET Core request and blocking thread pool threads is a no-go in that area.

Is this a remote console via the browser (using websockets to marshal the std/in/out/err)?

[drauch]

I understand, is there a suitable workaround? We want to run a process as part of an ASP.NET Core request and blocking thread pool threads is a no-go in that area.

Is this a remote console via the browser (using websockets to marshal the std/in/out/err)?

No, requests trigger the execution of various different Windows executables. The output is used to return suitable responses to the clients. Unfortunately, we can't change the executables either (i.e., to read the output from, e.g., a file).

[davidfowl]

I see, my only advice would be to limit concurrency of those requests or use a separate pool of threads.