Capture userAgent in URL metric when debugging

[westonruter]

Important

This is a follow-up PR to #1489. Merge that PR before this one so that this PR will merge into trunk.

I discovered on my site when looking at URL metrics that there was a crazy tall viewport:

"viewport": {
    "width": 1024,
    "height": 12140
}

Apparently this is because Googlebot is the crawler for the URL, as I found from https://www.seroundtable.com/googlebot-9000px-high-viewport-24727.html (although I need to confirm this).

In any case, it isn't currently possible to determine the user agent responsible for a given URL metric. So I think it will be very helpful when debugging URL metrics data to have this information included.

Separately, we'll need to compute the aspect ratio of the viewport and abort capturing the URL metric if it is not conceivably a regular device form factor!

[westonruter]

Actually, I don't like having to special case this optional property. This is probably better to make the schema extensible, and then a mini plugin can be installed which logs the user agent in the URL metric. This will pair very nicely with #1373 which introduced client-side extensions to amend the URL metrics prior to sending.

[adamsilverstein]

Apparently this is because Googlebot is the crawler for the URL

Ah, I wonder if crawler bots could be a problem generally as some may not render as well as others. Having the metrics would be helpful to see for sure; could we exclude bots entirely if we find they cause issues with collected metrics?

[westonruter]

@adamsilverstein I think we could exclude known bots by user agent, like Googlebot. But simply excluding by extreme aspect ratios should solve the problem too, right?

[adamsilverstein]

@adamsilverstein I think we could exclude known bots by user agent, like Googlebot. But simply excluding by extreme aspect ratios should solve the problem too, right?

Probably for most sites; just thinking about sites with JS front ends for example that may not get rendered properly by a crawler, potentially leading to saving of invalid metrics. I think its fine to leave for now, unless we discover this issue is more widespread than we expect.

I guess the good part about allowing the crawlers is we are more likely to get and update metrics even for low traffic sites.

[westonruter]

Closing in favor of another PR to make the schema extensible.