Skip to content

Conversation

@gargsaumya
Copy link
Contributor

@gargsaumya gargsaumya commented Sep 3, 2025

Work Item / Issue Reference

AB#34162
AB#38110

GitHub Issue: #<ISSUE_NUMBER>


Summary

This pull request improves the handling of large object (LOB) columns when fetching data from SQL Server in the mssql_python/pybind/ddbc_bindings.cpp file. The main change is to detect LOB columns (such as large strings or binary data) and switch to a per-row fetching strategy using SQLGetData_wrap for those columns, ensuring correct streaming and memory usage. For non-LOB columns, the batch fetching logic remains unchanged, but now includes logic to handle LOBs appropriately during batch fetches.

LOB detection and fetch strategy:

  • Added logic in both FetchMany_wrap and FetchAll_wrap to detect LOB columns by checking column data types and sizes, and to fall back to per-row fetch using SQLGetData_wrap when LOBs are present. This avoids buffer overflows and streams LOBs correctly. [1] [2]

  • Updated calls to FetchBatchData in both wrappers to pass the list of detected LOB columns, so batch fetches can handle them appropriately. [1] [2]

Batch fetch improvements:

  • Modified the signature of FetchBatchData to accept the LOB columns list and updated its logic to handle LOB columns differently: instead of throwing exceptions when buffer sizes are insufficient, it now streams LOB data using FetchLobColumnData. [1] [2] [3] [4] [5]

@github-actions github-actions bot added the pr-size: large Substantial code update label Sep 3, 2025
Copy link
Contributor

@sumitmsft sumitmsft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few comments

Copy link
Collaborator

@bewithgaurav bewithgaurav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lightly addressed some existing comments, requested one change

@gargsaumya gargsaumya force-pushed the saumya/fetchone-nvarcharmax branch from 63834f3 to d0ccd44 Compare September 11, 2025 06:20
@gargsaumya gargsaumya force-pushed the saumya/streaming-fetchall branch from 2e0ba04 to 1b45c78 Compare September 11, 2025 07:26
@github-actions github-actions bot added pr-size: large Substantial code update and removed pr-size: large Substantial code update labels Sep 11, 2025
@github-actions github-actions bot added pr-size: large Substantial code update and removed pr-size: large Substantial code update labels Sep 11, 2025
@github-actions github-actions bot added pr-size: large Substantial code update and removed pr-size: large Substantial code update labels Sep 11, 2025
@gargsaumya gargsaumya merged commit b3b0953 into saumya/fetchone-nvarcharmax Sep 15, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: large Substantial code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants