Arrow Support

[aersam]

Is your feature request related to a problem? Please describe.

Python lists / arrays are pretty slow and memory-intense, therefore many libraries allow reading things as Arrow Data. Examples are fastexcel or arrow-odbc. Many Data libraries support Arrow out of the box, like DuckDB/Polars/Spark/Pandas/DeltaLake. Would also be a great source for Bulk Import.

Describe the solution you'd like

Best would be an API similar to duckdb which has methods for Arrow Tables and RecordBatchReader: https://duckdb.org/docs/stable/guides/python/export_arrow.html#export-as-a-recordbatchreader This can also be done without a Dependency on PyArrow, since arrow supports PyCapsules (eg, see arro3 )

Describe alternatives you've considered

Well, I can also live with not-so fast queries :)

[sumitmsft]

Hi Adrian Ehrsam (@aersam) Thanks for raising this feature request.

As of now there is no roadmap of adding support for Arrow Data. Currently, we are building other features which are extremely critical for the correct functioning of the driver. However, we will triage Arrow Data and track it internally on how best we can support this. I'll keep this issue open until next update.

Having said that, it also gives an opportunity to you and other community members to help us by contributing to this project. Support for Arrow Data is good feature to start with. You can write the feature and raise a PR for us to review. Please let us know if you need any assistance from our side.

Again, thank you so much for using this mssql-python driver, and sharing your thoughts and suggestions with us! Appreciate it.

[dhirschfeld]

I'm excited to see work ongoing for arrow read support! 🚀

When the time comes to implement bulk-import it would also be great to have a pyarrow.Table as a supported data source.

[dlevy-msft-sql]

I'm excited to see work ongoing for arrow read support! 🚀

When the time comes to implement bulk-import it would also be great to have a pyarrow.Table as a supported data source.

I've got good news - BCP takes an iterator. It should already just work.

[gargsaumya]

Closing this issue as Apache Arrow support has already been implemented. Support for Apache Arrow in Bulk Copy is being tracked separately under #570
Thanks for the feedback and discussion.