Fleet version: v4.73.1
Web browser and operating system: N/A
💥 Actual behavior
The following routes had long processing times resulting in 502 errors after upgrading to v4.73.1:
/api/latest/fleet/os_versions
/api/fleet/orbit/software_install/result
/api/latest/fleet/device/<uuid>/software/install/532769
The following happened to the DB load when these routes were hit:
This also looked to cause high disk io/iops on the database at the same time:
Note: /api/latest/fleet/os_versions was on a team with over 60k hosts and the behavior was not present in v4.72.0. I am currently trying to see if this is causative or if the software installers URIs that were hit 15 minutes earlier at the time of high io were the underlying problem.
It's possible something else caused this as well. During the time of high impact, this was the query using the most load:
SELECT `osv` . `cve` , `cm` . `cvss_score` , `cm` . `epss_probability` , `cm` . `cisa_known_exploit` , `cm` . `published` AS `cve_published` , `cm` . `description` , `osv` . `resolved_in_version` , `osv` . `created_at` FROM ( SELECT `v` . `cve` , MIN ( `v` . `created_at` ) `created_at` , GROUP_CONCAT ( DISTINCTROW `v` . `resolved_in_version` SEPARATOR ? ) `resolved_in_version` FROM `operating_system_vulnerabilities` `v` JOIN `operating_systems` `os` ON `os` . `id` = `v` . `operating_system_id` A...
Query summary:
During the time of the issue, container CPU looked fairly normal, but container memory increased significantly. Not to problematic levels but an abnormal amount:
🛠️ To fix
TBD
Product designer: _________________________
🧑💻 Steps to reproduce
- Perform a software install on v4.73.1. Ideally with a large number of historical installs given the id of the install
- look at the os_versions on a team with 60k+ hosts
It looks like one or both could be causes or just symptoms.
🕯️ More info (optional)
N/A
Fleet version: v4.73.1
Web browser and operating system: N/A
💥 Actual behavior
The following routes had long processing times resulting in 502 errors after upgrading to v4.73.1:
The following happened to the DB load when these routes were hit:
This also looked to cause high disk io/iops on the database at the same time:
Note:
/api/latest/fleet/os_versionswas on a team with over 60k hosts and the behavior was not present in v4.72.0. I am currently trying to see if this is causative or if the software installers URIs that were hit 15 minutes earlier at the time of high io were the underlying problem.It's possible something else caused this as well. During the time of high impact, this was the query using the most load:
Query summary:
During the time of the issue, container CPU looked fairly normal, but container memory increased significantly. Not to problematic levels but an abnormal amount:
🛠️ To fix
TBD
Product designer: _________________________
🧑💻 Steps to reproduce
It looks like one or both could be causes or just symptoms.
🕯️ More info (optional)
N/A