Intermittent DefaultCredentialsError on GCE

[dhermes]

Original issue: googleapis/google-cloud-python#4358

After successful use of credentials, _ = google.auth.default(), an application crashes when credentials cannot be detected:

...
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 212, in __init__
    Client.__init__(self, credentials=credentials, _http=_http)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 125, in __init__
    credentials, _ = google.auth.default()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

/cc @dmho418

[dhermes]

Also from @dmho418 (may be unrelated to this issue, but is probably the root cause):

...
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/upload.py", line 97, in transmit
    retry_strategy=self._retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/compute_engine/credentials.py", line 93, in refresh
    raise exceptions.RefreshError(exc)
RuntimeError: RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true

[dhermes]

@jonparrott Would there be a way to access the number of retries that were exceeded?

[theacodes]

It'll be the requests / urllib3 default

[dhermes]

• Failing method (invoked by / raised from refresh())
• Function throwing exception (actual line where thrown)
• Transport used
• Exception being raised (from urllib3)

Default retry strategy for the requests transport:

>>> import requests
>>> session = requests.Session()
>>> adapter1, adapter2 = session.adapters.values()
>>> adapter1.max_retries
Retry(total=0, connect=None, read=False, redirect=None, status=None)
>>> adapter2.max_retries
Retry(total=0, connect=None, read=False, redirect=None, status=None)

From the docstring:

By default, Requests does not retry failed connections.

[dhermes]

I'm trying to reproduce on GCE with:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install -y python-dev python-pip
$ sudo -H pip install --upgrade google-auth requests
$ cat << EOF > repro.py
> import pdb
> import sys
> import time
>
> import google.auth
>
>
> def main():
>     if len(sys.argv) < 2:
>         num_attempts = 100
>     else:
>         num_attempts = int(sys.argv[1])
>     for n in range(num_attempts):
>         print(n)
>         try:
>             credentials, project = google.auth.default()
>         except:
>             pdb.set_trace()
>             raise
>         time.sleep(0.5)
>
>
> if __name__ == '__main__':
>     main()
> EOF
$
$ python repro.py 10
$ python repro.py 100
$ python repro.py 1000

and am not having any luck.

If I had to guess, I'd say the only way to reproduce would be to stress out the instance (e.g. high CPU usage) so that the process handling the metadata server fails.

[antoineazar]

We see this issue in our setup as well. Python script connecting to BQ and running hundreds of queries in rapid fire sequence. We'll occasionally see:
A INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable.

followed by a crash:
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credential and re-run the application.

[dhermes]

Awesome data @antoineazar! Thanks for the confirmation.

Any code you could share so we could try to reproduce / strategize?

[antoineazar]

@dhermes any code that runs multiple (dozens usually suffice) queries on BQ in rapid sequence. Some simple sample code (pre-0.28 library, didn't test with 0.28), you can wrap this in a loop:

client = bigquery.Client()
query_job = client.run_async_query(str(uuid.uuid4()), query)

# Use standard SQL syntax.
query_job.use_legacy_sql = False

# Set a destination table.
dest_dataset = client.dataset(dest_dataset_id)
dest_table = dest_dataset.table(dest_table_id)
query_job.destination = dest_table

# Allow the results table to be overwritten.
query_job.write_disposition = 'WRITE_TRUNCATE'

query_job.begin()
query_job.result()  # Wait for query to finish.

[GEverding]

I'm seeing this issue in production. Is there a workaround/fix?

[theacodes]

@GEverding the recommendation right now is to use a service account keyfile instead of relying on the GCE metadata service.

It's possible that we could make retry failed connections to the metadata service, but I'm unsure on that at the moment.