RFC: Powertools for AWS Lambda (Python) v3

Is this related to an existing feature request or issue?

Which Powertools for AWS Lambda (Python) utility does this relate to?

Other

Summary

We are excited to announce that we are starting a new major version (v3) for Powertools for AWS Lambda (Python). We are expediting a new major version to align with the scheduled Pydantic v1 end-of-life on June 30th, signalling the switch to much anticipated Pydantic v2. Pydantic plays a vital role and we care deeply about supply chain security. This release highlights our dedication to providing a secure and stable library that our customers trust.

Today, we already support both Pydantic v1 and v2. In this new major version, we will make Pydantic v2 the default, removing Pydantic v1. We will take this opportunity to introduce minor breaking changes to address common pain points. As always, we will provide an extensive upgrade guide to make this transition as smooth as possible.

Community

We invite the entire Powertools community, including users and contributors, to actively participate in this RFC. We greatly value the community's contribution, ideas, and willingness to solve problems or validate proposed changes.

Use case

The primary use case is to offer Pydantic v2. That includes a new Lambda Layer for each Python version since Pydantic v2 contains non-Python code (Rust). It future-proof Layers to contain any additional compiled dependency (like cryptography) without a new major version.

Additional key items on v3 backlog

Distribute Lambda Layers in the AWS GovCloud regions
Return sensible defaults over Optional in Event Source Data Classes to further reduce customers code
Remove deprecated code, deprecate legacy code to be removed in v4, and reduce API surface

Proposal

The action plan for this release with scope and tasks is as follows:

Update our documentation to comply with our Versioning Policy and inform customers that we have started work on a new major version
Create a branch named v3
Make adjustments to our GitHub actions to run checks on this new branch
Update Powertools v3 milestone to reflect current situation
Address and resolve the issues mapped out for the execution of this launch plan.
If possible, release early alpha versions for testing

Quick summary (Work in progress - Items can be added or removed):

✅ = done
🚧 = Work in progress
🚫 = Not considered in v3

Item	Issue/PR	Status	Code change required	Utility	Notes	Cross-Language
~~refactor the `SSMProvider` class to implement the `get_multiple` method~~	#3252	✅	Yes	Parameters
~~Return empty `Dict` or `List` in Event Source Data Classes instead of `None`~~	#2605	✅	Yes	Event Source Data Class
~~`event_parser` envelopes should handle unions of BaseModels~~	#2734	✅	No	Parser
~~Pydantic v2 managed layer~~	#2939	✅	No	Layer
Deprecate `tracing.base.BaseProvider`	#4105	🚧	No	Tracer
~~`resolved_headers_field` is not Optional~~	#4147	✅	No	Event Source Data Class
~~Create a Powertools Lambda layer for each Python version~~	#3859	✅	No	Layer
Publish layer to Gov Cloud regions	#1166	🚧	No	Layer		✅
~~Remove any specific code for Pydantic v1~~	#4191	✅	Yes	Codebase
~~Update Powertools documentation to add banner about v3~~	#4198	✅	No	Documentation
~~Add the `aws-encryption-sdk` dependency in the Powertools layer~~	#4192	✅	No	Layer	Depends on #3859
~~Review the method `extract_data_from_envelope` in JMESPath Functions~~	#4218	✅	No	JMESPath Functions
~~Split the code from the `api_gateway.py` file into several other files to ease maintenance~~	#4194	🚫	Yes	Event Handler
~~Add support for v3 branch in the github actions~~	#4220	✅	No	CI
Release the new version of Tracer utility with support for external providers	#2342	🚧	Yes	Tracer
~~Unmarshal DynamoDB fields when using `DynamoDBStreamModel`~~	#4219	✅	Yes	Parser	Reference: #1619	✅
~~Remove legacy code in the Metrics utility?~~	-	🚫	Yes	Metrics	Needs discussion
Keep the compatibility with Pydantic v2 and Bedrock agent resolver	#4196	🚧	No	Event Handler
~~Change the default TTL (cache) to 5 minutes~~	#4193	✅	Yes	Parameters		✅
~~Update boto3 version to 1.34.32~~	#4195	✅	No	Dependencies
Change behavior to fail batch on first error when using DynamoDB and Kinesis in batch processing	NEED ISSUE	🚧	Yes	Batch Processing	Reference: https://tinyurl.com/yc335m3s	✅
~~Remove the flag `log_uncaught_exceptions` in the Logger utility~~	#4199	🚫	Yes	Logger	See: #1798 (comment)
Support for retrieving batch of secrets	#4200	🚧	No	Parameters

Deadlines

We plan to release the first version of v3 by the end of June/beginning of July.

Out of scope

We will not include in v3:

Modularization to further reduce sizes
Strict typing
New utilities

Potential challenges

Adding Pydantic v2 in the Powertools layer requires layers per Python version, so the effort to implement this is not yet clear.

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

This feature request meets Powertools for AWS Lambda (Python) Tenets
Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

There's a lot of overlap between the Event Handler, Event Source Data Classes, Parser, and Validation

Is there a nice way to unify it all somehow?

There's a lot of overlap between the Event Handler, Event Source Data Classes, Parser, and Validation

Is there a nice way to unify it all somehow?

I agree. Since v3 will use pydantic by default, is there a need to maintain event source data classes? why not move to the parser's pydantic schemas exclusively? I suppose there are some gaps in coverage and missing schemas, but that can be fixed :)
Event handler/validation already support pydantic/parser schemas.

@leandrodamascena

This is exciting! Thanks for the update @leandrodamascena

Hello everyone! We are making good progress with V3 and hope to have an alpha release in the next few weeks and then release the official version this month/early September. Thanks for everyone's patience.

Awesome, thank you!

Hello everyone! We have a first draft of the upgrade guide. This should give you an idea of the changes you will need to make to migrate to v3. We will continue to make changes to this upgrade guide until we launch it.

#5028

Hello everyone! While we are making improvements to the upgrade guide, we already have a development version for local testing or even for your development environment.

We would love to hear your feedback on this new version.

v3.0.0.dev10 - https://pypi.org/project/aws-lambda-powertools/3.0.0.dev10/

Thanks

Thank you for advancing this product!
I hope this will be the start of improved OpenAPI autogeneration, making it as easy as in FastAPI 😄

@Hatter1337

Hi @Hatter1337! Thanks for using Powertools and giving us some ideas on what you want to see we investing time into it. But I'm curious about this:

I hope this will be the start of improved OpenAPI autogeneration, making it as easy as in FastAPI 😄

We already support OpenAPI generation in V2. The experience is simple and you just need to write down your routes and access swagger or generate OpenAPI JSON Schema.

https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#openapi

Are you talking about something different? Any improvement? I'm really curious to hear from you.

@Hatter1337

Hi @Hatter1337! Thanks for using Powertools and giving us some ideas on what you want to see we investing time into it. But I'm curious about this:

I hope this will be the start of improved OpenAPI autogeneration, making it as easy as in FastAPI 😄

We already support OpenAPI generation in V2. The experience is simple and you just need to write down your routes and access swagger or generate OpenAPI JSON Schema.

https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#openapi

Are you talking about something different? Any improvement? I'm really curious to hear from you.

This might not be the best place for this discussion, but I’ll respond here...

I've been using Powertools for quite long time, and to be honest, I often end up writing the Swagger file manually, primarily because OpenAPI generation requires almost full descriptions:

@router.get(
    "/user/<id>",
    summary="Get user data",
    description="Get user data from ...",
    response_description="User data",
    responses={
        200: {
            "description": "Success",
            "content": {"application/json": {"model": UserDataResponse}},
        },
        400: {
            "description": "Bad request",
            "content": {"application/json": {"model": BadRequestResponse}},
        },
        404: {
            "description": "Not found",
            "content": {"application/json": {"model": NotFoundResponse}},
        },
    },
    tags=["User"],
)

All of this is in the same file as the business logic, making it quite bulky and harder to manage. I'm not sure how it's implemented in FastAPI, but there it seems much simpler 🙊

Occasionally, I run into certain issues, one of which I've described here. Another example is that I haven’t been able to find information on how to add something similar to this when using app.enable_swagger:

@app.get("/swagger", include_in_schema=False)
def openapi():
    return app.get_openapi_json_schema(
        title="Awesome API",
        servers=[servers],
        security_schemes={
            "apikey": APIKey(
                name="X-API-KEY",
                description="API KeY",
                in_=APIKeyIn.header,
            ),
        },
    )

Also, I prefer to split API endpoints across different Lambda functions - for example, having User CRUD in one Lambda function and authentication in another, rather than putting the entire API in a single Lambda function. However, with this approach, it's almost impossible to generate a unified OpenAPI/Swagger file for the whole API.

That said, I still believe Powertools is an amazing framework, and I'm grateful to everyone who contributed to its development. I hope I can also be helpful to the community in the future.

Where can we find the new doc for V3?

This is now released under 3.0.0 version!

Navigation Menu

Repository navigation

RFC: Powertools for AWS Lambda (Python) v3 #4189

Is this related to an existing feature request or issue?

Which Powertools for AWS Lambda (Python) utility does this relate to?

Summary

Community

Use case

Proposal

Quick summary (Work in progress - Items can be added or removed):

Deadlines

Out of scope

Potential challenges

Dependencies and Integrations

Alternative solutions

Acknowledgment

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC: Powertools for AWS Lambda (Python) v3 #4189

Description

Is this related to an existing feature request or issue?

Which Powertools for AWS Lambda (Python) utility does this relate to?

Summary

Community

Use case

Proposal

Quick summary (Work in progress - Items can be added or removed):

Deadlines

Out of scope

Potential challenges

Dependencies and Integrations

Alternative solutions

Acknowledgment

Activity

jplock commented on Apr 24, 2024

ran-isenberg commented on Apr 26, 2024

8 remaining items

jesseadams commented on Aug 12, 2024

zerubeus commented on Aug 19, 2024

leandrodamascena commented on Aug 21, 2024

leandrodamascena commented on Aug 28, 2024

Hatter1337 commented on Sep 10, 2024

leandrodamascena commented on Sep 10, 2024

Hatter1337 commented on Sep 10, 2024

zerubeus commented on Sep 19, 2024

github-actions commented on Sep 23, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions