Presto hook is broken in the latest provider release (2.2.0)

[pingzh]

Apache Airflow version

2.2.5 (latest released)

What happened

The latest presto provider release https://pypi.org/project/apache-airflow-providers-presto/ is broken due to:

  File "/usr/local/lib/python3.8/site-packages/airflow/providers/presto/hooks/presto.py", line 117, in get_conn
    http_headers = {"X-Presto-Client-Info": generate_presto_client_info()}
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/presto/hooks/presto.py", line 56, in generate_presto_client_info
    'try_number': context_var['try_number'],
KeyError: 'try_number'

What you think should happen instead

This is due to the latest airflow release 2.2.5 does not include this PR:

#22297

the presto hook changes were introduced in this pr #22416

How to reproduce

No response

Operating System

Mac

Versions of Apache Airflow Providers

https://pypi.org/project/apache-airflow-providers-presto/

version: 2.2.0

Deployment

Other

Deployment details

local

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

cc @levyitay

[pingzh]

cc @ephraimbuddy

[eladkal]

Providers must be compatible with Airflow>=2.1 so regardless if #22297 was released in 2.2.5 or not... that is not enough.

We need to make the provider compatible with 2.1

[pingzh]

@eladkal good point. I remember there is a CI check for the compatibility. not sure why it did not catch it

@eladkal could you please take a look? #23061

[potiuk]

@eladkal good point. I remember there is a CI check for the compatibility. not sure why it did not catch it

@eladkal could you please take a look? #23061

The check is not 100% - it only checks that imports are not broken. We are adding more "back-compatibility" checks as pre-commits but I think we are just learning on the kinds of things that should be taken into account.

Here is the currenet check and we should add try_number check there too: https://github.com/apache/airflow/blob/main/scripts/ci/pre_commit/pre_commit_check_2_1_compatibility.py

In short what needs to happen - we need to formalize the provider's interface a bit more than we did so far in the future and make sure that any changes to "contract" in Airflow are "flagged" and any changes are protected against those changes. For now it is more retro-active but before we split off providers from the core repositry (I think this is something we are going to discuss during the next months) we will have to take the learnings from such cases and formalize it.

The May 21 bumping min-compatibility to 2.2. will be a good opportunity to do it.

Those cases are a good learning grounds for the future more "robust" solution.