-
Notifications
You must be signed in to change notification settings - Fork 31
FEAT: LOB detection and fetch strategy for FetchAll and FetchMany #221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
gargsaumya
merged 6 commits into
saumya/fetchone-nvarcharmax
from
saumya/streaming-fetchall
Sep 15, 2025
Merged
FEAT: LOB detection and fetch strategy for FetchAll and FetchMany #221
gargsaumya
merged 6 commits into
saumya/fetchone-nvarcharmax
from
saumya/streaming-fetchall
Sep 15, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sumitmsft
requested changes
Sep 9, 2025
Contributor
sumitmsft
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left few comments
bewithgaurav
requested changes
Sep 11, 2025
Collaborator
bewithgaurav
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lightly addressed some existing comments, requested one change
63834f3 to
d0ccd44
Compare
2e0ba04 to
1b45c78
Compare
sumitmsft
approved these changes
Sep 12, 2025
bewithgaurav
approved these changes
Sep 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
This pull request improves the handling of large object (LOB) columns when fetching data from SQL Server in the
mssql_python/pybind/ddbc_bindings.cppfile. The main change is to detect LOB columns (such as large strings or binary data) and switch to a per-row fetching strategy usingSQLGetData_wrapfor those columns, ensuring correct streaming and memory usage. For non-LOB columns, the batch fetching logic remains unchanged, but now includes logic to handle LOBs appropriately during batch fetches.LOB detection and fetch strategy:
Added logic in both
FetchMany_wrapandFetchAll_wrapto detect LOB columns by checking column data types and sizes, and to fall back to per-row fetch usingSQLGetData_wrapwhen LOBs are present. This avoids buffer overflows and streams LOBs correctly. [1] [2]Updated calls to
FetchBatchDatain both wrappers to pass the list of detected LOB columns, so batch fetches can handle them appropriately. [1] [2]Batch fetch improvements:
FetchBatchDatato accept the LOB columns list and updated its logic to handle LOB columns differently: instead of throwing exceptions when buffer sizes are insufficient, it now streams LOB data usingFetchLobColumnData. [1] [2] [3] [4] [5]