FIX: Request SQL_CHAR as SQL_C_WCHAR in arrow fetch path

[ffelixg]

Work Item / Issue Reference

AB#44922

GitHub Issue: #553

---

Summary

Due to #495, we can now request SQL_CHAR data as SQL_C_WCHAR, i.e. utf16le strings. Doing this for the arrow path ensures that arrow methods always return correct data no matter the encoding settings / locale / operating system. There does not seem to be any significant negative performance impact.

[Copilot]

Pull request overview

Updates the Arrow fetch path in the C++ pybind layer to always request SQL_CHAR/SQL_VARCHAR data as SQL_C_WCHAR (UTF-16) so Arrow results are correct regardless of server/client codepage, locale, or platform—addressing the VARCHAR non-ASCII decoding issues reported in #553.

Changes:

• Switch Arrow batch binding/fetching for SQL_CHAR/SQL_VARCHAR from SQL_C_CHAR to SQL_C_WCHAR to avoid codepage-dependent decoding.
• Remove the narrow-char copy path for SQL_CHAR/SQL_VARCHAR in Arrow batch production and route through the existing wide-char → UTF-8 conversion logic.
• Add an Arrow regression test covering Unicode round-tripping through a UTF-8-collated VARCHAR column.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
mssql_python/pybind/ddbc_bindings.cpp	Changes Arrow batch binding and fetch handling so VARCHAR is requested as SQL_C_WCHAR, ensuring consistent Unicode correctness.
tests/test_004_cursor_arrow.py	Adds a regression test to validate Arrow output for Unicode data stored in VARCHAR with UTF-8 collation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 90%

🎯 Overall Coverage 80%

📈 Total Lines Covered: 6671 out of 8245 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/cursor.py (100%)
• mssql_python/pybind/ddbc_bindings.cpp (88.9%): Missing lines 4633-4635

Summary

• Total: 30 lines
• Missing: 3 lines
• Coverage: 90%

mssql_python/pybind/ddbc_bindings.cpp

Lines 4629-4639

  4629 
  4630 SQLRETURN FetchArrowBatch_wrap(SqlHandlePtr StatementHandle, py::list& capsules,
  4631                                int arrowBatchSize,
  4632                                int charCtype) {
! 4633     // Fetch narrow char data as SQL_C_CHAR if on Linux/macOS and configured by the user
! 4634     charCtype = EffectiveCharCtypeForFetch(charCtype, "utf-8");
! 4635 
  4636     // An overly large fetch size doesn't seem to help performance
  4637     int fetchSize = 64;
  4638 
  4639     SQLRETURN ret;

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.pybind.ddbc_bindings.cpp: 76.4%
mssql_python.row.py: 76.9%
mssql_python.__init__.py: 77.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.logging.py: 85.5%
mssql_python.connection.py: 85.6%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report