feat(datadog)!: Add SLO tables, housekeeping#11166

disq · 2023-06-01T11:58:38Z

Adds datadog_slos and datadog_slo_corrections tables for SLO info.

Housekeeping:

Updates the Datadog SDK used from 2.9.0 to 2.13.0
Updates all PKs to include account_name and id, plus parent id if exists
Updates plugin-sdk version to 3.9.0
Removes all manually added fields, now using transformers exclusively
Datadog NullableTypes mapped to proper types

Closes #10780

github-actions · 2023-06-01T11:58:59Z

This PR has the following changes to source plugin(s) tables:

Table datadog_dashboard_lists: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_dashboards: column type changed from json to utf8 for description
Table datadog_dashboards: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int32 for updater_id
⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for canceled
⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for end
⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for monitor_id
⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for parent_id
Table datadog_downtimes: column added with name notify_end_states and type list<item: utf8, nullable>
Table datadog_downtimes: column added with name notify_end_types and type list<item: utf8, nullable>
Table datadog_downtimes: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_global_variables: primary key constraint added to column account_name
Table datadog_global_variables: column order changed for id
Table datadog_hosts: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_incident_attachments: column added with name incident_id (PK) and type utf8
⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint added to column account_name
⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint added to column id
⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint removed from column _cq_id
Table datadog_incident_attachments: column added with name additional_properties and type json
Table datadog_incident_attachments: column order changed for type
Table datadog_incidents: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int32 for updater_id
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for canceled
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for end
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for monitor_id
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for parent_id
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint added to column account_name
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint added to column id
⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint removed from column _cq_id
Table datadog_monitor_downtimes: column added with name additional_properties and type json
Table datadog_monitor_downtimes: column added with name notify_end_states and type list<item: utf8, nullable>
Table datadog_monitor_downtimes: column added with name notify_end_types and type list<item: utf8, nullable>
Table datadog_monitors: column added with name matching_downtimes and type json
Table datadog_monitors: column order changed for deleted
Table datadog_monitors: column order changed for id
Table datadog_monitors: column order changed for priority
Table datadog_notebooks: column order changed for id
Table datadog_permissions: column order changed for id
⚠️ BREAKING CHANGE: Table datadog_role_permissions: column added with name role_id (PK) and type utf8
⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint added to column account_name
⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint added to column id
⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint removed from column _cq_id
⚠️ BREAKING CHANGE: Table datadog_role_users: column added with name role_id (PK) and type utf8
⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint added to column account_name
⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint added to column id
⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint removed from column _cq_id
Table datadog_roles: column order changed for id
Table datadog_slo_corrections was added
Table datadog_slos was added
Table datadog_synthetics: column order changed for public_id
⚠️ BREAKING CHANGE: Table datadog_user_permissions: column added with name user_id (PK) and type utf8
⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint added to column account_name
⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint added to column id
⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint removed from column _cq_id
Table datadog_users: column order changed for id

hermanschaaf · 2023-06-01T12:49:01Z

website/tables/datadog/datadog_slo_corrections.md

@@ -0,0 +1,19 @@
+# Table: datadog_slo_corrections
+
+This table shows data for Datadog Slo Corrections.


Minor, but would be nice to add an exception for SLO in the title here

hermanschaaf

LGTM - could you maybe just add some more details about the changes to the PR description for future readers?

plugins/source/datadog/client/resolvers.go

plugins/source/datadog/client/transformers.go

candiduslynx · 2023-06-01T13:04:29Z

plugins/source/datadog/resources/services/downtimes/downtimes.go

 		Resolver:  fetchDowntimes,
 		Multiplex: client.AccountMultiplex,
-		Transform: transformers.TransformWithStruct(&datadogV1.Downtime{}),
+		Transform: client.TransformWithStruct(&datadogV1.Downtime{}, transformers.WithPrimaryKeys("Id")),


transformers.WithPrimaryKeys("Id") might be moved to client.TransformWithStruct

I don't think that's wise, we'd need to revert it all for a single struct later on :)

candiduslynx

just a note about WithTitleTransformer

website/tables/datadog/datadog_slo_corrections.md

plugins/source/datadog/resources/plugin/plugin.go

candiduslynx

only a smallish transformer comment (might be not required, just took k8s title replacer as source)

disq · 2023-06-01T14:17:26Z

only a smallish transformer comment (might be not required, just took k8s title replacer as source)

I don't think it's required as I don't see any rogue/trailing spaces in the SLO table markdown files.

disq · 2023-06-01T14:35:30Z

future readers?

@hermanschaaf Do you think that's enough? I've updated the PR title as well, but I don't think a "commit override" is necessary.

hermanschaaf · 2023-06-01T14:36:49Z

@disq Looks good, thanks. I agree, don't think a commit override is necessary here

🤖 I have created a release *beep* *boop* --- ## [3.0.0](plugins-source-datadog-v2.0.0...plugins-source-datadog-v3.0.0) (2023-06-06) ### ⚠ BREAKING CHANGES * **datadog:** Add SLO tables, housekeeping ([#11166](#11166)) ### This Release has the Following Changes to Tables - Table `datadog_dashboards`: column type changed from `json` to `utf8` for `description` (:warning: breaking) - Table `datadog_downtimes`: column type changed from `json` to `int32` for `updater_id` (:warning: breaking) - Table `datadog_downtimes`: column type changed from `json` to `int64` for `canceled` (:warning: breaking) - Table `datadog_downtimes`: column type changed from `json` to `int64` for `end` (:warning: breaking) - Table `datadog_downtimes`: column type changed from `json` to `int64` for `monitor_id` (:warning: breaking) - Table `datadog_downtimes`: column type changed from `json` to `int64` for `parent_id` (:warning: breaking) - Table `datadog_downtimes`: column added with name `notify_end_states` and type `list<item: utf8, nullable>` - Table `datadog_downtimes`: column added with name `notify_end_types` and type `list<item: utf8, nullable>` - Table `datadog_global_variables`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_incident_attachments`: column added with name `incident_id (PK)` and type `utf8` (:warning: breaking) - Table `datadog_incident_attachments`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_incident_attachments`: primary key constraint added to column `id` (:warning: breaking) - Table `datadog_incident_attachments`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `datadog_incident_attachments`: column added with name `additional_properties` and type `json` - Table `datadog_monitor_downtimes`: column type changed from `json` to `int32` for `updater_id` (:warning: breaking) - Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `canceled` (:warning: breaking) - Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `end` (:warning: breaking) - Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `monitor_id` (:warning: breaking) - Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `parent_id` (:warning: breaking) - Table `datadog_monitor_downtimes`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_monitor_downtimes`: primary key constraint added to column `id` (:warning: breaking) - Table `datadog_monitor_downtimes`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `datadog_monitor_downtimes`: column added with name `additional_properties` and type `json` - Table `datadog_monitor_downtimes`: column added with name `notify_end_states` and type `list<item: utf8, nullable>` - Table `datadog_monitor_downtimes`: column added with name `notify_end_types` and type `list<item: utf8, nullable>` - Table `datadog_monitors`: column added with name `matching_downtimes` and type `json` - Table `datadog_role_permissions`: column added with name `role_id (PK)` and type `utf8` (:warning: breaking) - Table `datadog_role_permissions`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_role_permissions`: primary key constraint added to column `id` (:warning: breaking) - Table `datadog_role_permissions`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `datadog_role_users`: column added with name `role_id (PK)` and type `utf8` (:warning: breaking) - Table `datadog_role_users`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_role_users`: primary key constraint added to column `id` (:warning: breaking) - Table `datadog_role_users`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `datadog_rum_events` was added - Table `datadog_slo_corrections` was added - Table `datadog_slos` was added - Table `datadog_user_permissions`: column added with name `user_id (PK)` and type `utf8` (:warning: breaking) - Table `datadog_user_permissions`: primary key constraint added to column `account_name` (:warning: breaking) - Table `datadog_user_permissions`: primary key constraint added to column `id` (:warning: breaking) - Table `datadog_user_permissions`: primary key constraint removed from column `_cq_id` (:warning: breaking) ### Features * Add `datadog_rum_events` table ([#11229](#11229)) ([e2a07a2](e2a07a2)), closes [#11044](#11044) * **datadog:** Add SLO tables, housekeeping ([#11166](#11166)) ([103a6bf](103a6bf)) ### Bug Fixes * **deps:** Update github.com/apache/arrow/go/v13 digest to e07e22c ([#11151](#11151)) ([5083cf7](5083cf7)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 20b0de9 ([#11199](#11199)) ([dc3565d](dc3565d)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 88d5dc2 ([#11226](#11226)) ([9f306bc](9f306bc)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to a7aad4c ([#11184](#11184)) ([8a0822e](8a0822e)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to c67fb39 ([#11169](#11169)) ([dcb0f92](dcb0f92)) * **deps:** Update golang.org/x/exp digest to 2e198f4 ([#11155](#11155)) ([c46c62b](c46c62b)) * **deps:** Update google.golang.org/genproto digest to e85fd2c ([#11156](#11156)) ([dbe7e92](dbe7e92)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.0.9 ([#11240](#11240)) ([f92cd4b](f92cd4b)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.3 ([#11150](#11150)) ([dc00994](dc00994)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.4 ([#11244](#11244)) ([8fceef6](8fceef6)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.7.0 ([#11113](#11113)) ([487bf87](487bf87)) * **pagination:** Fix pagination for `datadog_incidents` table ([#11230](#11230)) ([b0eae04](b0eae04)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

disq added 5 commits June 1, 2023 11:59

feat(datadog)!: Update Datadog SDK, fix types

4e81c33

Update SDK

5863831

PK changes, use transformers everywhere

172a37d

fix nullable resolvers

a3b24e5

Add slo

8a8aed9

disq requested review from a team and bbernays and removed request for a team June 1, 2023 11:58

cq-bot added datadog labels Jun 1, 2023

disq linked an issue Jun 1, 2023 that may be closed by this pull request

feat: datadog source plugin to include SLO data #10780

Closed

1 task

account_name in PKs

fd01243

vercel bot deployed to Preview June 1, 2023 12:06 View deployment

hermanschaaf reviewed Jun 1, 2023

View reviewed changes

hermanschaaf approved these changes Jun 1, 2023

View reviewed changes