echo benchmark for Websockets

[kondratyev-nv]

Contributes to dotnet/runtime#55293.

Projects

• New server app - src/BenchmarksApps/Websocket
• New client app - src/WebsocketClient

I haven't added a new worker as I haven't found any information on how it's used and how to test it. If the current code is ok, I can add it in the following PR if necessary.

Scenarios

• echo - Server responds with the same content received from a client

Tests

• Run benchmark locally via crank. Replaced links to yaml files to my fork and run crank

crank.exe --config https://raw.githubusercontent.com/kondratyev-nv/Benchmarks/main/scenarios/websocket.benchmarks.yml --scenario websocket --profile local --variable scenario=echo

Output

| application           |                           |
| --------------------- | ------------------------- |
| CPU Usage (%)         | 39                        |
| Cores usage (%)       | 156                       |
| Working Set (MB)      | 79                        |
| Private Memory (MB)   | 88                        |
| Build Time (ms)       | 31,798                    |
| Start Time (ms)       | 1,637                     |
| Published Size (KB)   | 87,593                    |
| .NET Core SDK Version | 5.0.301                   |
| ASP.NET Core Version  | 5.0.7+67acc3d             |
| .NET Runtime Version  | 5.0.0-rtm.20509.6+cca8c88 |


| warmup                |               |
| --------------------- | ------------- |
| CPU Usage (%)         | 53            |
| Cores usage (%)       | 213           |
| Working Set (MB)      | 48            |
| Private Memory (MB)   | 28            |
| Build Time (ms)       | 14,074        |
| Start Time (ms)       | 870           |
| Published Size (KB)   | 66,597        |
| .NET Core SDK Version | 5.0.301       |
| ASP.NET Core Version  | 5.0.7+67acc3d |
| .NET Runtime Version  | 5.0.7+556582d |
| Max RPS               | 41,633        |
| Requests              | 625,245       |
| Mean latency (ms)     | 2             |


| load                  |               |
| --------------------- | ------------- |
| CPU Usage (%)         | 43            |
| Cores usage (%)       | 173           |
| Working Set (MB)      | 44            |
| Private Memory (MB)   | 26            |
| Build Time (ms)       | 7,077         |
| Start Time (ms)       | 1,437         |
| Published Size (KB)   | 66,597        |
| .NET Core SDK Version | 5.0.301       |
| ASP.NET Core Version  | 5.0.7+67acc3d |
| .NET Runtime Version  | 5.0.7+556582d |
| Max RPS               | 25,665        |
| Requests              | 391,466       |
| Mean latency (ms)     | 4             |

P.S. Link in CONTRIBUTING.md leads to 404 😅

[BrennanConroy]

+@CarnaViire

[kondratyev-nv]

For the new benchmark, I'll keep links in yaml files to my fork while this PR is not ready to be merged, so I won't need to change those every time.

[kondratyev-nv]

@BrennanConroy @sebastienros Any comments on this?

[BrennanConroy]

Looks good overall, we'll wait for Seb to be back so he can take a look at the yml files etc.

[stephentoub]

How close is this to being merged?
cc: @davidfowl

[kondratyev-nv]

How close is this to being merged?

From my end, after PR is approved all I need to do is to change references to the correct remote repo in yml files.

[stephentoub]

@sebastienros? I was hoping to use this against citrine and some of the other servers to help validate dotnet/runtime#56282.

[sebastienros]

@stephentoub Since the code is already in a PR, the yml files are available to pass to a crank command line even if it's not merged yet. So the command line that was shared already works, it's just pulling the apps from the branch. Here I updated it to point to the perf-lin machine for instance.

crank.exe --config https://raw.githubusercontent.com/kondratyev-nv/Benchmarks/main/scenarios/websocket.benchmarks.yml --scenario websocket --profile aspnet-perf-lin --variable scenario=echo

Then to use custom dlls, add --application.options.outputFiles c:\bin\*.dll if the "application" tier.

[stephentoub]

@sebastienros, I'd tried that previously (and again now), but get this:

The specified endpoint url 'http://asp-perf-lin:5001' for 'application' is invalid or not responsive: "No such host is known. (asp-perf-lin:5001)"

[sebastienros]

@stephentoub Try to access it in a browser (http://asp-perf-lin:5001) you need to be in the VPN. I just tried it an it worked.

[stephentoub]

Ah, somehow my VPN refuses to connect.

[adityamandaleeka]

@stephentoub Is it working for you now?

Also, I just looked at your other question about throughput figures for this benchmark. @sebastienros I see client side figures for requests and requests/sec emitted by this benchmark (under "warmup" and "load"). Just to confirm, those are successful/completed requests right? Not just requests sent? In other words, is that number a good representation of the throughput of the application?

[sebastienros]

@adityamandaleeka I actually added a comment on the PR to always display the number of failures. And to not display the warmup job as an independent result.

[BrennanConroy]

Just to confirm, those are successful/completed requests right? Not just requests sent?

These numbers are generated by the client code at https://github.com/aspnet/Benchmarks/pull/1686/files#diff-2b8caa6c2f3e1c51ab39e188615427641d0906941c8cb85b80aea66987fadbb9R176 for every end of message received.

I'm not sure what an unsuccessful request would look like for WebSockets, do you just mean it throws?

[sebastienros]

Merged to see if that runs successfully on the CI. Would be useful to add it to the scenarios/readme.md for visibility. If you don't do it I will.

[kondratyev-nv]

I've already updated Readme in this PR - https://github.com/aspnet/Benchmarks/tree/main/scenarios#websockets-benchmarks.