Exception on import when AWS_PROFILE is invalid/null

Expected Behavior

import aws_lambda_powertools should still succeed even if the ENV contains AWS_PROFILE with a bad value.

Current Behavior

It appears that during import, under at least some conditions, aws_lambda_powertools will attempt to setup an aws client session? I'm not sure I understand why. So if the ENV contains an invalid AWS config, in particular AWS_PROFILE= or AWS_PROFILE=fakeprofile, then the import fails with an exception. Which, yes, it's invalid, and I now that I know aws_lambda_powertools has this behavior I can work around it easily by unsetting the envs. But it still seems like it shouldn't be setting up a session on import?

Possible Solution

Not sure on a solution, but a workaround is to make sure your development environment unsets bad AWS_ envs before attempting to run tests on your python code. :)

Steps to Reproduce (for bugs)

AWS_PROFILE= python -c 'import aws_lambda_powertools'

Environment

Powertools version used: 1.10.5
Packaging format (Layers, PyPi): PyPi
AWS Lambda function runtime:: n/a
Debugging logs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/__init__.py", line 9, in <module>
    from .tracing import Tracer  # noqa: F401
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/tracing/__init__.py", line 6, in <module>
    from .tracer import Tracer
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/tracing/tracer.py", line 10, in <module>
    import aws_xray_sdk.core
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/__init__.py", line 10, in <module>
    xray_recorder = AsyncAWSXRayRecorder()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 66, in __init__
    self._sampler = DefaultSampler()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/sampler.py", line 25, in __init__
    self._connector = ServiceConnector()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 23, in __init__
    self._xray_client = self._create_xray_client()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 164, in _create_xray_client
    aws_access_key_id='', aws_secret_access_key=''
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 802, in create_client
    verify = self.get_config_variable('ca_bundle')
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 241, in get_config_variable
    logical_name)
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 340, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile () could not be found

@lorengordon

@lorengordon this has to do with importing the xray core, which is then picking up the AWS_PROFILE being set.

I would either unset AWS_PROFILE and have dummy values for the rest

unset AWS_PROFILE 
export AWS_ACCESS_KEY_ID=fake
export AWS_SECRET_ACCESS_KEY=fake
export AWS_DEFAULT_REGION=us-east-1

Or unset it before importing powertools in the tests

import os

if "AWS_PROFILE" in os.environ:
    del os.environ["AWS_PROFILE"]

@michaelbrewer

@michaelbrewer Of course, that's my workaround. When I first attempted to reproduce, I imported the xray core directly and it worked fine, so I thought it was something in this library. But I must have messed up the env, because now I am reproducing easily on that import. I'll open an issue on the xray sdk. Thanks!

$ env |grep AWS_
AWS_PROFILE=

$ python -c 'import aws_xray_sdk.core'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/__init__.py", line 10, in <module>
    xray_recorder = AsyncAWSXRayRecorder()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 66, in __init__
    self._sampler = DefaultSampler()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/sampler.py", line 25, in __init__
    self._connector = ServiceConnector()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 23, in __init__
    self._xray_client = self._create_xray_client()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 164, in _create_xray_client
    aws_access_key_id='', aws_secret_access_key=''
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 802, in create_client
    verify = self.get_config_variable('ca_bundle')
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 241, in get_config_variable
    logical_name)
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 340, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile () could not be found

I suppose it could be a feature request here, in the sense that I don't need or want xray right now. I am only using the logger. So it might be nice to gate the import to an explicit call to a powertools class/method...

@heitorlessa

@heitorlessa is looking at lazy loading libraries

@michaelbrewer

100% and thanks @michaelbrewer for helping out.

I'm not sure how I missed this issue in my notifications so apologies @lorengordon for not replying earlier.

I've done some research on backwards compatible solutions that work for 3.6+ (what we support), and am scheduling next Friday to experiment and will keep you posted.

Thanks for reporting it!

@heitorlessa

@heitorlessa All good, the workarounds are viable for now. 3.6+ works for us. Feel free to close this if you have another tracking issue for lazy loading libraries... I opened another issue on the xray sdk to better understand why it needs to setup the session as a global, and impact imports like this...

Awesome - I'll subscribe to that.

I'll keep this open, and I'll try this method first as it seems clever enough to also prevent PyCharm/VSCode debugger to crash due to increased ref counting in getattr

https://wil.yegelwel.com/lazily-importing-python-modules/

Nice, and it's very simple! I've got use cases for that in my own projects! 😂

As a pure hack, i have disabled the one i don't need during my unit testing

import sys
from unittest.mock import Mock

mock = Mock()
sys.modules['aws_lambda_powertools.tracing']=mock
sys.modules['aws_lambda_powertools.metrics']=mock
sys.modules['aws_lambda_powertools.loggin']=mock

@lorengordon

hey @lorengordon - This will now be available in the next release. If you're not using Tracer, you should see a ~400ms+ perf improvement in cold start depending on your memory setting.

Here's the aftermath using 128M memory: from ~620ms to ~50ms.

Before

After

@heitorlessa

Wow, that's fantastic, thanks @heitorlessa!

@heitorlessa

@heitorlessa @lorengordon and this also works without any errors ;)

AWS_PROFILE= python -c 'import aws_lambda_powertools'

This is now available in 1.11.0 - Literally just launched ;) 🎉

Navigation Menu

Repository navigation

Exception on import when AWS_PROFILE is invalid/null #294

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Environment

Before

After

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Exception on import when AWS_PROFILE is invalid/null #294

Description

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Environment

Activity

michaelbrewer commented on Feb 26, 2021

lorengordon commented on Feb 26, 2021

lorengordon commented on Feb 26, 2021

michaelbrewer commented on Feb 26, 2021

heitorlessa commented on Feb 26, 2021

lorengordon commented on Feb 26, 2021

heitorlessa commented on Feb 26, 2021

lorengordon commented on Feb 26, 2021

michaelbrewer commented on Feb 27, 2021

heitorlessa commented on Mar 3, 2021

Before

After

lorengordon commented on Mar 3, 2021

michaelbrewer commented on Mar 3, 2021

heitorlessa commented on Mar 5, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions