(Left Over?) pyarrow does not compile

[alamb]

Describe the bug
I am not sure this is a bug but I was surprised that datafusion still had an optional py03 dependency: #1627

There are some python bindings in datafusion: https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/pyarrow.rs#L28

The code came in via #873 from @kszucs

Given the datafusion python bindings live in https://github.com/datafusion-contrib/datafusion-python now, I am not sure what this code is used for. It also no longer compiles.

To Reproduce

cargo test -p datafusion --features=pyarrow

error[E0053]: method `from_pyarrow` has an incompatible type for trait
  --> datafusion/src/pyarrow.rs:35:28
   |
35 |     fn from_pyarrow(value: &PyAny) -> PyResult<Self> {
   |                            ^^^^^^
   |                            |
   |                            expected struct `pyo3::types::any::PyAny`, found struct `pyo3::PyAny`
   |                            help: change the parameter type to match the trait: `&pyo3::types::any::PyAny`
   |
   = note: expected fn pointer `fn(&pyo3::types::any::PyAny) -> std::result::Result<_, pyo3::err::PyErr>`
              found fn pointer `fn(&pyo3::PyAny) -> std::result::Result<_, pyo3::PyErr>`

error[E0053]: method `to_pyarrow` has an incompatible type for trait
  --> datafusion/src/pyarrow.rs:52:31
   |
52 |     fn to_pyarrow(&self, _py: Python) -> PyResult<PyObject> {
   |                               ^^^^^^
   |                               |
   |                               expected struct `pyo3::python::Python`, found struct `pyo3::Python`
   |                               help: change the parameter type to match the trait: `pyo3::python::Python<'_>`
   |
   = note: expected fn pointer `fn(&ScalarValue, pyo3::python::Python<'_>) -> std::result::Result<pyo3::instance::Py<pyo3::types::any::PyAny>, pyo3::err::PyErr>`
              found fn pointer `fn(&ScalarValue, pyo3::Python<'_>) -> std::result::Result<Py<pyo3::PyAny>, pyo3::PyErr>`

error[E0053]: method `to_pyarrow` has an incompatible type for trait
  --> datafusion/src/pyarrow.rs:52:31
   |
52 |     fn to_pyarrow(&self, _py: Python) -> PyResult<PyObject> {
   |                               ^^^^^^
   |                               |
   |                               expected struct `pyo3::python::Python`, found struct `pyo3::Python`
   |                               help: change the parameter type to match the trait: `pyo3::python::Python<'_>`
   |
   = note: expected fn pointer `fn(&scalar::ScalarValue, pyo3::python::Python<'_>) -> std::result::Result<pyo3::instance::Py<pyo3::types::any::PyAny>, pyo3::err::PyErr>`
              found fn pointer `fn(&scalar::ScalarValue, pyo3::Python<'_>) -> std::result::Result<Py<pyo3::PyAny>, pyo3::PyErr>`

error[E0277]: the trait bound `arrow::array::ArrayData: PyClass` is not satisfied
   --> datafusion/src/pyarrow.rs:46:27
    |
46  |         let array = array.extract::<ArrayData>()?;
    |                           ^^^^^^^ the trait `PyClass` is not implemented for `arrow::array::ArrayData`
    |
    = note: required because of the requirements on the impl of `pyo3::FromPyObject<'_>` for `arrow::array::ArrayData`
note: required by a bound in `pyo3::PyAny::extract`
   --> /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.14.5/src/types/any.rs:614:12
    |
614 |         D: FromPyObject<'a>,
    |            ^^^^^^^^^^^^^^^^ required by this bound in `pyo3::PyAny::extract`

error[E0308]: mismatched types
  --> datafusion/src/pyarrow.rs:59:28
   |
59 |         Self::from_pyarrow(value)
   |                            ^^^^^ expected struct `pyo3::types::any::PyAny`, found struct `pyo3::PyAny`
   |
   = note: expected reference `&pyo3::types::any::PyAny`
              found reference `&'source pyo3::PyAny`
   = note: perhaps two different versions of crate `pyo3` are being used?

error[E0308]: mismatched types
  --> datafusion/src/pyarrow.rs:59:9
   |
58 |     fn extract(value: &'source PyAny) -> PyResult<Self> {
   |                                          -------------- expected `std::result::Result<ScalarValue, pyo3::PyErr>` because of return type
59 |         Self::from_pyarrow(value)
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `pyo3::PyErr`, found struct `pyo3::err::PyErr`
   |
   = note: expected enum `std::result::Result<_, pyo3::PyErr>`
              found enum `std::result::Result<_, pyo3::err::PyErr>`
   = note: perhaps two different versions of crate `pyo3` are being used?

error[E0308]: mismatched types
  --> datafusion/src/pyarrow.rs:65:25
   |
65 |         self.to_pyarrow(py).unwrap()
   |                         ^^ expected struct `pyo3::python::Python`, found struct `pyo3::Python`
   |
   = note: perhaps two different versions of crate `pyo3` are being used?

error[E0308]: mismatched types
  --> datafusion/src/pyarrow.rs:65:9
   |
64 |     fn into_py(self, py: Python) -> PyObject {
   |                                     -------- expected `Py<pyo3::PyAny>` because of return type
65 |         self.to_pyarrow(py).unwrap()
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `Py`, found struct `pyo3::instance::Py`
   |
   = note: expected struct `Py<pyo3::PyAny>`
              found struct `pyo3::instance::Py<pyo3::types::any::PyAny>`
   = note: perhaps two different versions of crate `pyo3` are being used?

Some errors have detailed explanations: E0053, E0277, E0308.
For more information about an error, try `rustc --explain E0053`.
error: could not compile `datafusion` due to 7 previous errors
warning: build failed, waiting for other jobs to finish...
error[E0308]: mismatched types
  --> datafusion/src/pyarrow.rs:59:9
   |
58 |     fn extract(value: &'source PyAny) -> PyResult<Self> {
   |                                          -------------- expected `std::result::Result<scalar::ScalarValue, pyo3::PyErr>` because of return type
59 |         Self::from_pyarrow(value)
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `pyo3::PyErr`, found struct `pyo3::err::PyErr`
   |
   = note: expected enum `std::result::Result<_, pyo3::PyErr>`
              found enum `std::result::Result<_, pyo3::err::PyErr>`
   = note: perhaps two different versions of crate `pyo3` are being used?

error: build failed

Compilation exited abnormally with code 101 at Fri Jan 21 09:55:03

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[kszucs]

@alamb it is actually required for the python bindings. It enables convenient pyo3 conversions.

We cannot implement this in the python repository since both the trait and type would be foreign then. Sadly by moving the python bindings to another repository this module hasn't been tested anymore.

[alamb]

We can at least add a test for the module in CI to ensure it doesn't get broken. I'll do that

[alamb]

Filed #1635 to track adding a test to CI