Unable to create polaris catalog due to missing headers

[rinzool]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

Description

Since version 0.10.0 it is not possible to create a Polaris catalog with credential parameter.

For example:

catalog_parameters = {
    "type": "rest",
    # ...
    "scope": "PRINCIPAL_ROLE:ALL",
    "header.Polaris-Realm": "my-realm",
    "credential": f"{client_id}:{client_secret}"
}

catalog = load_catalog("default", **catalog_parameters)

This code fails with error RESTError: MissingOrInvalidRealm: Missing or invalid realm.

The exact same code works with pyiceberg 0.9.1.

Cause

Polaris needs header Polaris-Realm.

We can see in the full stacktrace (see below) that the error occurs when we create the AuthManager line 266: https://github.com/apache/iceberg-python/blob/pyiceberg-0.10.0/pyiceberg/catalog/rest/__init__.py#L266.

But headers are injected after initializing AuthManager at line 269:
https://github.com/apache/iceberg-python/blob/pyiceberg-0.10.0/pyiceberg/catalog/rest/__init__.py#L269.

Full stacktrace

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/__init__.py:128](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/__init__.py#line=127), in load_rest(name, conf)
    125 def load_rest(name: str, conf: Properties) -> Catalog:
    126     from pyiceberg.catalog.rest import RestCatalog
--> 128     return RestCatalog(name, **conf)

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py:234](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py#line=233), in RestCatalog.__init__(self, name, **properties)
    232 super().__init__(name, **properties)
    233 self.uri = properties[URI]
--> 234 self._fetch_config()
    235 self._session = self._create_session()

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py:364](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py#line=363), in RestCatalog._fetch_config(self)
    361 if warehouse_location := self.properties.get(WAREHOUSE_LOCATION):
    362     params[WAREHOUSE_LOCATION] = warehouse_location
--> 364 with self._create_session() as session:
    365     response = session.get(self.url(Endpoints.get_config, prefixed=False), params=params)
    366 try:

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py:266](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py#line=265), in RestCatalog._create_session(self)
    264     session.auth = AuthManagerAdapter(AuthManagerFactory.create(auth_impl or auth_type, auth_type_config))
    265 else:
--> 266     session.auth = AuthManagerAdapter(self._create_legacy_oauth2_auth_manager(session))
    268 # Set HTTP headers
    269 self._config_headers(session)

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py:295](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/__init__.py#line=294), in RestCatalog._create_legacy_oauth2_auth_manager(self, session)
    285 auth_url = self.auth_url if client_credentials is not None else None
    287 auth_config = {
    288     "session": session,
    289     "auth_url": auth_url,
   (...)
    292     "optional_oauth_params": self._extract_optional_oauth_params(),
    293 }
--> 295 return AuthManagerFactory.create("legacyoauth2", auth_config)

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py:321](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py#line=320), in AuthManagerFactory.create(cls, class_or_name, config)
    318     except Exception as err:
    319         raise ValueError(f"Could not load AuthManager class for '{class_or_name}'") from err
--> 321 return manager_cls(**config)

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py:94](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py#line=93), in LegacyOAuth2AuthManager.__init__(self, session, auth_url, credential, initial_token, optional_oauth_params)
     92 self._credential = credential
     93 self._optional_oauth_params = optional_oauth_params
---> 94 self._refresh_token()

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py:122](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py#line=121), in LegacyOAuth2AuthManager._refresh_token(self)
    120 def _refresh_token(self) -> None:
    121     if self._credential is not None:
--> 122         self._token = self._fetch_access_token(self._credential)

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py:116](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/auth.py#line=115), in LegacyOAuth2AuthManager._fetch_access_token(self, credential)
    114     response.raise_for_status()
    115 except HTTPError as exc:
--> 116     _handle_non_200_response(exc, {400: OAuthError, 401: OAuthError})
    118 return TokenResponse.model_validate_json(response.text).access_token

File [/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/response.py:111](https://poc-dwh-qfn.forepaas.io/opt/conda/lib/python3.11/site-packages/pyiceberg/catalog/rest/response.py#line=110), in _handle_non_200_response(exc, error_handler)
    108     errs = ", ".join(err["msg"] for err in e.errors())
    109     response = f"RESTError {exc.response.status_code}: Received unexpected JSON Payload: {exc.response.text}, errors: {errs}"
--> 111 raise exception(response) from exc

RESTError: MissingOrInvalidRealm: Missing or invalid realm

Solution

Move self._config_headers(session) instruction in the beginning of _create_session method.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time