Import statement unexpectedly executes functions in __init__.py

[ndrluis]

Apache Iceberg version

None

Please describe the bug 🐞

When we import any functions that reside in __init__.py, it triggers the execution of the code instead of merely importing it. We must relocate the functions to a different file and manage the imports in __init__.py using the __all__ variable. This approach enables both star imports and specific imports without running the code upon import, thereby providing isolation between the internal and public API.

Examples:

AWS SDK for Pandas
Pandas Arrays
Scikit-Learn Datasets

More info:

Python.org: Packages
Python.org: Importing from a Package

This is causing me problems with the CLI I'm building as an internal tool. Can I be responsible for fixing this behavior?

[ndrluis]

After reading the __init__.py file in each namespace, I believe that this problem exists only in the table, io and catalog namespaces.

Does anyone have suggestions on how to organize these functions?

[ndrluis]

I was thinking it would be easier, but I found a test that makes a table import through the catalog module, so I think that we might encounter more cases like this.

Considering that, and to make it easier to review, I will create one PR for each namespace. I will need some help to determine if something should be public or private.

Starting with the catalog, I believe that only Catalog and load_catalog should be public. Does anyone disagree with this? Another point is that I believe delete_files and delete_data_files should be private methods of Catalog, but I think we can address this in another PR.

[ndrluis]

I was wrong about my hypothesis, but I believe that it is something occurring during the import process because the stacktrace error shows that the import is executing certain operations.

This is the stacktrace (I'm using my branch with the public API isolation, but it's the same error):
https://gist.github.com/ndrluis/e72346240cd7c22199ccea71b9d72b3f

[ndrluis]

I found the problem, it's related to Python 3.9.7. I updated to 3.9.12 and fixed the error.

From the project's perspective, I believe that the isolation of the public API and the correct use of imports are important tasks to address, so I will close this issue as a bug and open another as an improvement.