Retry did not occur in case of google.auth.exceptions.TransportError

[nitishxp]

I am finding below error in one of my production environment using Cloud Functions. The sdk should retry by default but its not retrying.
I am trying to access a bucket in Cloud function python38 environment

Code Snippet

googleCredentials = service_account.Credentials.from_service_account_info(authentication)
gcs = storage.Client(project=project, credentials=googleCredentials)
stage_bucket = gcs.get_bucket('test-bucket') 

My package version

google-auth==1.24.0
google-cloud-storage==1.36.2
google-api-python-client==1.12.8

Error:

Error in app: Exception Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen retries = retries.increment( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment raise six.reraise(type(error), error, _stacktrace) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 182, in __call__ response = self.session.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/common/functions.py", line 55, in wrapper response = wrapped_func(*args, **kwargs) File "/workspace/main.py", line 970, in manifest **stage_bucket = gcs.get_bucket("test-bucket")** File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/client.py", line 402, in get_bucket bucket.reload( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 996, in reload super(Bucket, self).reload( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 218, in reload api_response = client._connection.api_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 78, in api_request return call() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 472, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 336, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 374, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 460, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/service_account.py", line 361, in refresh access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant response_data = _token_endpoint_request(request, token_uri, body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request response = request(method="POST", url=token_uri, headers=headers, body=body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 188, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[frankyn]

Adding context from @busunkim96:

It looks like requests.ConnectionError is in the list of retryable types for the storage library, so the storage library intends for it to be retried.

I think it isn't actually being retried because google-auth re-wraps that error as a google.auth.exceptions.TransportError. It looks like the retry configs needs tweaking in the storage library so I'm going to close this in favor of tracking at #414.

[nitishxp]

Could you also add ConnectionError ?

[frankyn]

Hi @nitishxp could you file another issue for this request. It might need more investigation because we currently retry ConnectionError

python-storage/google/cloud/storage/retry.py

Line 30 in 23a8db8

requests.ConnectionError,