pyarrow 18 regression: ValueError: type(schema)=<class 'pyarrow.lib.Schema'>

[grihabor]

Apache Iceberg version

0.7.1 (latest release)

Please describe the bug 🐞

After updating pyarrow from 17.0.0 to 18.0.0, I've got this error:

Traceback (most recent call last):
  File "/home/grihabor/projects/playground/pyiceberg-pyarrow-schema/./run.py", line 38, in <module>
    tbl = catalog.create_table(
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/grihabor/projects/playground/pyiceberg-pyarrow-schema/.venv/pyarrow-18/lib/python3.12/site-packages/pyiceberg/catalog/sql.py", line 193, in create_table
    schema: Schema = self._convert_schema_if_needed(schema)  # type: ignore
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grihabor/projects/playground/pyiceberg-pyarrow-schema/.venv/pyarrow-18/lib/python3.12/site-packages/pyiceberg/catalog/__init__.py", line 732, in _convert_schema_if_needed
    raise ValueError(f"{type(schema)=}, but it must be pyiceberg.schema.Schema or pyarrow.Schema")
ValueError: type(schema)=<class 'pyarrow.lib.Schema'>, but it must be pyiceberg.schema.Schema or pyarrow.Schema

Steps to reproduce

run.py script:

import shutil
from pyiceberg.catalog.sql import SqlCatalog
from pathlib import Path
import pyarrow as pa

warehouse_path = Path("/tmp/warehouse")
if warehouse_path.exists():
    shutil.rmtree(warehouse_path)

Path(warehouse_path).mkdir()
catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)


df = pa.Table.from_pylist(
    mapping=[
        {"city": "Amsterdam", "lat": 52.371807, "long": 4.896029},
        {"city": "San Francisco", "lat": 37.773972, "long": -122.431297},
        {"city": "Drachten", "lat": 53.11254, "long": 6.0989},
        {"city": "Paris", "lat": 48.864716, "long": 2.349014},
    ],
    schema=pa.schema(
        [
            ("city", pa.large_string()),
            ("lat", pa.float64()),
            ("long", pa.float64()),
        ]
    ),
)


catalog.create_namespace("db")
tbl = catalog.create_table(
    identifier="db.cities",
    schema=df.schema,
)
tbl.overwrite(df)
print(tbl.scan().to_arrow())

This works:

uv venv --python 3.12 .venv/pyarrow-17
source .venv/pyarrow-17/bin/activate
uv pip install pyiceberg pyarrow==17.0.0 sqlalchemy
python ./run.py

This throws the exception:

uv venv --python 3.12 .venv/pyarrow-18
source .venv/pyarrow-18/bin/activate
uv pip install pyiceberg pyarrow==18.0.0 sqlalchemy
python ./run.py

[kevinjqliu]

i was not able to reproduce on my local setup.
On main branch,

poetry add pyarrow==18.0.0
poetry run python run.py

Also verified that its using arrow 18

➜  iceberg-python git:(main) ✗ poetry run python                        
import pPython 3.11.0 (main, Sep  2 2023, 10:31:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
>>> pyarrow.__version__
'18.0.0'

I noticed that in your example, you've used python to run ./run.py, perhaps you need to use

uv run python ./run.py

[grihabor]

Thanks for verifying, I've caught this in CI on dependabot update and reproduced it locally, I'll try to reproduce the problem in a docker container