How to use EventCounters?

[felipepessoto]

The lack of Performance Counters in .NET Core is blocking me to use ASP.NET Core for Enterprise applications.

I found many issues telling us to use EventCounters, but I couldn't find docs or even code in the repository using it.

There is any docs?

Anyway, ETW events doesn't replace Performance Counters, I use it to monitor Req/sec and I can't see how to do it easily with ETW, or ask the support team to monitor or collect it. Would be nice to have a Windows only implementation.

Thanks

[karelz]

For PerformanceCounters please vote on #15468

@vancem used to have extensive documentation of EventSource, with EventCounters added as well. I can't find the source, but it should be part of the NuGet package. @vancem any pointer?

[felipepessoto]

#15468 is only to read the values. I'm more interested in writing them.

[karelz]

For the producer scenario, why cannot you use EventCounters?
Even PerfCounters (v2) use ETW under the hood, just slightly different events format. Everything should be achievable with just EventCounters.

The problem is that .NET Framework uses PerfCounters v1. It is fragile, old technology. It requires DLL in system32, so admin machine-wide setup and machine-wide registration. It is hard to do multiple products on the same machine (side-by-side). It is buggy.
Machine-wide registration is something we want to avoid in .NET Core. It leads to more and more troubles.

If you still need PerfCounters, I would suggest to use EventCounters and write your own out-of-process consumer of EventCounters which translates the events into PerfCounters (v2). It will be work, it will require you to solve the machine-wide registration, but it could work if you truly believe you need that.

[felipepessoto]

I still need to read more about EventCounters e ETW to say it won't work in my scenario. I'm doing some tests

In particular, it's very easy to use PerfCounter. For devs, we just need to call Increment() and it translates to absolute numbers, rate per second, average, etc. And the support team just need to open Perfmon and watch the numbers.

Another great feature is the MultiInstance support

[karelz]

In particular, it's very easy to use PerfCounter. For devs, we just need to call Increment() and it translates to absolute numbers, rate per second, average, etc.

EventCounters have similar metrics automatically AFAIR.

And the support team just need to open Perfmon and watch the numbers.

That one is tricky - you pay for it by having .NET Framework installed machine wide and registered machine wide. It's a hidden HUGE cost. You can achieve that, but you have to pay the cost of installing & registering.
It shouldn't be that difficult to write the code. The installation in each app will be giant pain IMO.

[felipepessoto]

I find out how to generate the events and read them in other process. But just the events written by EventSource.WriteEvent, I can't read those written by EventCounter.WriteMetric.

I created a project based on this PR: aspnet/Hosting#886. But it's curious nobody else is using EventCounters, even this PR wasn't merged.

[karelz]

EventCounters are based on top of EventSource, so the consumption should be the same.

I let @vancem chime in on how wide-spread EventCounters are these days and where to find the best examples (we had them in pretty extensive internal documentation 2-3 years ago when we designed them, so I hope it is available somewhere - @vancem typically does that, so if it's not available, we should fix that ;)).

[vancem]

• @cshung

Note that EventCounters are work in progress at the moment. Andrew was able to put a certain amount of work to get something working enough so that that direction/architecture was clear, and we could build upon it. The tooling is certainly not there yet. You could imagine building a bridge to windows PerfCounters so that PerfMon works (however it has all the disadvantages of that, including windows-only and admin-install requirements).

Andrew: did we publish your blog on how to use it? I am noticing that none of the docs we wrote up have made it externally. We should massage your blog.html into markdown and at least check in into the CoreFX repo (we can have a blog entry point at it). We should also get the spec that we wrote out so it is visiable. We can use this issue to track that.

The source code for EventCounters is at

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/src/System/Diagnostics/Tracing/EventCounter.cs

And here are the tests, which can be helpful in actual usage.

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/tests/BasicEventSourceTest/TestEventCounter.cs

This is no substitute for proper docs (which we will work on), but I can give you a quick sketch of what EventCounters are. Bascially they are a layer on top of EventSource that allow you to have a simple API (WriteMetric) that gets accumulated LOCALLY (calculating sum, average ...). Consumers (e.g. tools), can then ask to turn these counters on and have their values dumped into an event at a given rate (e.g. once every X seconds or on demand).

So from an instrumentation perspective, it is pretty relatively simple to use EventCounters, (you create an EventSource (which defines a 'namespace' for the data), define counters (give the various values names), and then do WriteMetric() calls on it every time one of these values changes. The EventCounter infrastructure calculates stats locally. All of this is off unless some tool asks for it. When the tool turns it on it gives a update rate, and the stats are dumped at that rate as events. As mentioned there is rudimentary support for this in the PerfView tool, but otherwise I know of no other tooling.

To answer the obvious question: why don't you just support PerfCounters? First we are not averse for someone doing this. We just don't think it is the best approach for support in CoreFX because

1. Architecturally Perfcounters are a separate 'pipe' of diagnostic information. This is pretty unfortunate, as there is no real value in having two pipes. Arguably you should pass the counter data through the same 'pipe' as any other telementry. This is in fact what WIndows does under the hood (perfcounters use ETW as a substrate), however from the users's point of view they are not integrated. Fixing this means that you need to change the PerfCounter APIs.

2. PerfCounters are really never enough alone. They are for monitoring, but when things go wrong, you quickly want to get more detailed data (which require events). This is just another argument that counters are just a special kind of low-overhead eventing, and you want the two systems integrated (so you only have to instrument once).

3. The only existing code for PerfCounters is on Windows, and has a fair bit of complexity to it (there are many counter types, and other subtleties) so making this work cross platform is problematic (and it does not fix point (1). Thus making something that is 'drop in compatible' is dubious. Once you decide that you should break the API, then fixing the other architectural problems (it is not integrated with eventing), make even more sense.

4. Existing PerfCounters require admin install. Again this really is a manifestation of point (1), because if you just used the existing telemetry pipe the problem is already solved.

We do understand that we are not at parity with what exists on windows, and it will take some time to get to that parity (we need to make it easy). However even here it is not clear how exactly to do that (arguably we should just be part of a bigger monitoring system like App Insights). We are making progress, but it is slower than we would like.

[felipepessoto]

I tried to use Perview to collect data from EventCounter, but it just logs events from WriteEvent, it seems EventCounter.WriteMetric is doing nothing.

[vancem]

Exactly what did you do to try to turn on the EventCounter using PerfView? You have to do more than just turn on the provider, you have to tell it the sampling frequency. For example to turn on EventCounters associated with the EventSource called 'SomeEventSource' a PerfView command line that would do that is

• PerfView /onlyProviders=*SomeEventSource:EventCounterIntervalSec=1 collect

Did you do something like this?

[felipepessoto]

I only set the provider, without the interval (and I used the GUI).

I will try your suggestion. Thanks

[felipepessoto]

It worked on PerfView.

How can I collect EventCounters using TraceEventSession?

Thanks.

[felipepessoto]

About the admin privileges. Perfmon requires me to be admin to create the definition, what is ok for most people. Does ETW require me to be admin to collect the data?

[karelz]

AFAIK ETW collection requires admin :(

[cshung]

Some time ago I wrote a PerfView extension that allow visualizing the EventCounter data, hope it helps.
microsoft/perfview#146