Resource Manager: update API from `v1beta1` to `v3`

[enriquejosepadilla]

Python: 3.5.3
Operating System: Debian GNU
Pip Freeze:

google-api-core==1.3.0
google-api-python-client==1.7.4
google-auth==1.5.1
google-auth-httplib2==0.0.3
google-cloud-container==0.1.1
google-cloud-core==0.28.1
google-cloud-resource-manager==0.28.1
google-cloud-storage==1.11.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

Stacktrace:

Traceback (most recent call last):
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/bin/django_gke", line 11, in
load_entry_point('django-gke', 'console_scripts', 'django_gke')()
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/django_gke.py", line 253, in main
args.func(args.debug)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/django_gke.py", line 173, in new
project_id = goog_client.create_project_workflow()
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cli/google_client.py", line 111, in create_project_workflow
project_id, project_name)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cloudlib/project.py", line 40, in create_and_set_existing_project
self._create_project(project)
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/backoff/_sync.py", line 99, in retry
ret = target(*args, **kwargs)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cloudlib/project.py", line 55, in _create_project
project.create()
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/google/cloud/resource_manager/project.py", line 138, in create
data=data)
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/google/cloud/_http.py", line 293, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 POST https://cloudresourcemanager.googleapis.com/v1beta1/projects: The caller does not have permission

Steps to reproduce: Using Gcloud auth login with a non corp account and the attempting to project.create() will throw this error.

We are able to get this to work with corp accounts but will be doing external users soon and found that this fails.

I believe changing the version here to v1 would fix this.

[tseaver]

@theacodes Can you confirm that moving resourcemanager from v1beta1 to v1 is reasonable?

[theacodes]

Yeah totally reasonable.

…

On Thu, Sep 20, 2018 at 12:01 PM Tres Seaver ***@***.***> wrote: @theacodes <https://github.com/theacodes> Can you confirm that moving resourcemanager from v1beta1 to v1 is reasonable? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/GoogleCloudPlatform/google-cloud-python/issues/6039#issuecomment-423296387>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc_EP-7BWKbnhwmhywRplgKtBv890ks5uc-YMgaJpZM4Wyrke> .

[tseaver]

@enriquejosepadilla Can you retry with the current trunk?

[enriquejosepadilla]

Alas, if only the beta version of the API was the same. I got the following stacktrace:

Please wait while project Djangogke Project is being created
Traceback (most recent call last):
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/bin/django_gke", line 11, in
load_entry_point('django-gke', 'console_scripts', 'django_gke')()
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/django_gke.py", line 227, in main
args.func(args.debug)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/django_gke.py", line 153, in new
project_id = goog_client.create_project_workflow()
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cli/google_client.py", line 111, in create_project_workflow
project_id, project_name)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cloudlib/project.py", line 40, in create_and_set_existing_project
self._create_project(project)
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/backoff/_sync.py", line 99, in retry
ret = target(*args, **kwargs)
File "/usr/local/google/home/epad/Projects/django-gke2/django-gke/django_gke/cloudlib/project.py", line 55, in _create_project
project.create()
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/google/cloud/resource_manager/project.py", line 139, in create
self.set_properties_from_api_repr(resource=resp)
File "/usr/local/google/home/epad/.local/share/virtualenvs/django-gke-0KxaOTFM/lib/python3.5/site-packages/google/cloud/resource_manager/project.py", line 86, in set_properties_from_api_repr
self.number = resource['projectNumber']
KeyError: 'projectNumber'

Seems to me they changed the data structure in the API. I'll look further into it to see what differences there are.

[enriquejosepadilla]

Going to the rest docs here and comparing the v1 vs v1beta theres a clear difference:

v1:

{
"name": "operations/cp.5198783426586810065"
}

v1beta:

{
"projectNumber": "567866699074",
"projectId": "testing23452376",
"lifecycleState": "ACTIVE",
"name": "testing",
"createTime": "2018-09-21T18:14:10.072Z",
"parent": {
"type": "organization",
"id": "433637338589"
}
}

The code currently parses the results v1beta has.

Tried using the website with my non-corp account also throwed a 403, so without a doubt the beta is only meant for internal users.

[tseaver]

I just diffed pretty-printed versions of the v1beta1 and v1 discovery docs. There are a lot more changes there than I would expect:

• New resource types: folders, liens, operations
• organizations and projects have added a bunch of methods for munging org policies: clearOrgPolicy, getEffectiveOrgPolicy, getOrgPolicy, listAvailableOrgPolicyConstraints, listOrgPolicies, setOrgPolicy.
• organizations.list is gone: it looks to be replaced by organizations.search.
• projects.create now returns an LRO.

Moving resource_manager from supporting v1beta1 to v1 is definitely a project which needs prioritization, and not a simple bugfix.

And then there is v2, which drops all resources except folders !?

[enriquejosepadilla]

@tseaver Thanks for looking into that. And wow thats a huge diff between a beta and the released version.

And i have no idea why their V2 changed so drastically, can't find any documentation regarding why they dropped almost everything.

[tseaver]

@enriquejosepadilla it seems likely that the effort to support the new API version is significant enough to require prioritization in our backlog.

/cc @danoscarmike

[enriquejosepadilla]

Hey @tseaver sounds good. We currently aren't blocked because of this.

[zehauser]

+1 to this issue. Really hoping it gets prioritized! :)

[Elad-Laufer]

+1 to this issue

[W3Meyrick]

+1 to this issue